Senior Data Annotation Jobs in Montreal
<strong>Overview<br><br></strong>Senior data annotation jobs in Montreal focus on creating and evaluating high-quality training data for AI/ML systems. At Rex.zone, you will support LLM training pipelines through data labeling, RLHF evaluation, prompt evaluation, and QA review to improve model performance and safety in a remote, full-time role.<br><br><strong>Key Responsibilities<br><br></strong><ul><li>Deliver high-accuracy data labeling and review across NLP, LLM evaluation, computer vision annotation, and content safety labeling</li><li>Design, maintain, and enforce annotation guidelines compliance and taxonomy standards</li><li>Lead QA evaluation workflows including sampling plans, inter-annotator agreement checks, and adjudication</li><li>Execute RLHF tasks such as pairwise ranking, preference labeling, and consistency audits</li><li>Conduct prompt evaluation and rubric-based scoring aligned to model requirements</li><li>Perform dataset audits, ambiguity logging, and label error analysis to improve training data quality</li><li>Partner with engineering/ML stakeholders to define acceptance criteria and feedback loops into LLM training pipelines</li><li>Document edge cases and escalation paths to reduce guideline drift</li><li>Support onboarding and calibration using examples, gold sets, and retraining plans</li><li>Track quality metrics and recommend process improvements that improve throughput without sacrificing accuracy<br><br></li></ul><strong>Required Qualifications<br><br></strong><ul><li>Senior data annotation or data labeling operations experience with QA evaluation ownership</li><li>Strong understanding of taxonomy design, guideline adherence, and ambiguity resolution</li><li>Hands-on experience with LLM evaluation, RLHF, or prompt evaluation workflows</li><li>Familiarity with NLP tasks such as named entity recognition and text classification</li><li>Comfortable working remotely with cross-functional stakeholders<br><br></li></ul><strong>Nice to Have<br><br></strong><ul><li>Content safety labeling or policy-driven evaluation experience</li><li>Inter-annotator agreement metrics and quality sampling methodologies</li><li>Computer vision annotation exposure (bounding boxes, polygons, segmentation)</li><li>Multilingual evaluation or locale-specific guidelines<br><br></li></ul><strong>Work Model<br><br></strong>Remote, Full-time.<br><br>Apply with relevant annotation experience and examples of QA evaluation or guideline development work.
