Senior AI Data Annotation Jobs in Toronto (Remote)
Rex.zone is hiring senior contributors to lead AI data annotation and QA evaluation workstreams that improve training data quality for modern AI/ML systems. This is a remote , full-time role aligned to real-world LLM training pipelines across NLP, computer vision, and content safety domains.
About The Role You will translate product and research goals into labeling instructions, refine guidelines for edge cases, adjudicate ambiguity, and drive measurable quality improvements through audits, calibration, and error analysis. Work includes RLHF preference ranking, prompt evaluation, named entity recognition, computer vision annotation (bounding boxes/segmentation), and content safety labeling.
What You Will Do
- Create, iterate, and maintain annotation guidelines and SOPs for complex datasets
- Perform and audit data labeling with measurable annotation guidelines compliance
- Execute RLHF comparison tasks and curate preference data with consistent rubrics
- Run QA evaluation programs: sampling plans, golden sets, inter-annotator agreement, defect taxonomy tracking
- Connect annotation outcomes to model performance improvement and large language model evaluation outcomes
- Document dataset lineage, versioning, and evaluation criteria
- Senior-level experience in AI data annotation / data labeling and rubric-based review
- Ability to interpret ambiguous examples, adjudicate edge cases, and write clear instructions
- Understanding of how labeled datasets impact RLHF outcomes and downstream model behavior
- Strong communication and cross-functional collaboration (Engineering, Product, Research)
- Hands-on RLHF workflows and prompt evaluation for conversational AI
- Content safety labeling and policy-driven evaluation experience
- Multimodal computer vision annotation exposure (detection/segmentation)
- Experience scaling annotation programs with vendors or distributed teams
