Quality Systems Lead

Encord
London
Workplace: OnsiteFull timeFunction: Quality & Regulatory (Non-Software)Experience: 4+ yearsSkills: ["Leadership","Writing","Statistical reasoning","Calibration","Systems thinking"]

Own end-to-end data quality measurement for AI training and deployment: define rubrics and acceptance criteria, build automated evaluation (model-assisted and LLM-based screening, agreement analysis, and drift/anomaly detection), and convert audit judgments into labeled ground-truth. Lead and calibrate an audit team (~10 specialists in India), set certification pass thresholds, report quality KPIs, and partner with product and engineering to make quality measurement native in the platform.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Encord
Encord
1 week ago

Quality Systems Lead

âś“ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 9 hours agoStatus: Live

Job Summary

Own end-to-end data quality measurement for AI training and deployment: define rubrics and acceptance criteria, build automated evaluation (model-assisted and LLM-based screening, agreement analysis, and drift/anomaly detection), and convert audit judgments into labeled ground-truth. Lead and calibrate an audit team (~10 specialists in India), set certification pass thresholds, report quality KPIs, and partner with product and engineering to make quality measurement native in the platform.
Location: London
Workplace: Onsite
Employment Type: Full time
Job Function: Quality & Regulatory (Non-Software)
Seniority: Manager level

Key Responsibilities

  • •Build automated dataset quality evaluation and root-cause detection using model-assisted and LLM-based screening, agreement analysis, and anomaly/drift detection.
  • •Hire, train, calibrate, and manage a dedicated audit team (~10 specialists in India) with standards based on inter-rater agreement and catch rate.
  • •Turn audit output into labeled ground truth that trains and validates the automated layer, managing automated vs. manual coverage over time.
  • •Own the quality standard for each data type: create rubrics (including edge cases), golden sets, and acceptance criteria with customers and the Special Projects lead.
  • •Develop scoring systems for annotator/reviewer performance, set certification pass thresholds, report quality KPIs, and work with Project Management on remediation.
Travel: Low travel

Pay and Benefits

Equity and Bonus:Equity
Perks:Annual LeaveLearning BudgetCommissionEquity

Key Requirements

  • •4+ years owning both technical and operational outcomes in a data/AI/service delivery environment where quality was measured rather than asserted.
  • •Hands-on Python and SQL to build analysis and tooling for evaluation.
  • •Practical experience applying models to a quality/evaluation problem (e.g., LLM-as-judge, model-assisted QA, automated evaluation, anomaly detection, or classifier-based screening).
  • •Sampling methodology and agreement statistics (e.g., Cohen’s and Fleiss’ kappa, F1 against ground truth) applied to real production data.
  • •Experience hiring, training, and managing a team (ideally audit/review/QA), ideally distributed.
Experience:4+ yearsDataAIService deliveryAnnotationEvaluationModel training
Skills:LeadershipWritingStatistical reasoningCalibrationSystems thinking
Tech Stack:PythonSQLLLM

Company Brief

Encord
Provides a computer vision data platform for labeling, dataset management, and model evaluation, enabling teams to build, validate, and monitor ML models with collaboration and tooling for high-quality annotated data.
Industry: Data Infrastructure
Company Size: Medium (51 to 250 employees)
Growth: Growth Stage Startup
Funding: Series A
Headquarters: London, United Kingdom
Founded: 2019
WebsiteLinkedIn