Human Data Quality Engineer (Founding Team)

Prolific
Mexico, United States
Workplace: RemoteFull timeFunction: QA, Test & Release EngineeringExperience: 5+ yearsSkills: ["Stakeholder management","Analytical thinking","Statistical thinking","Communication","Builder mindset"]

Build quality frameworks and evaluation rubrics for complex human-data programs that power frontier AI models. Partner with clients and AI labs to translate model needs into robust annotation and study design requirements, then engineer quality upstream across recruitment, screening, training, and launch readiness. Create scalable quality systems, automated checks, dashboards, and monitoring using Python and SQL, investigating data integrity issues and turning insights into durable improvements.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Prolific
Prolific
7 months ago

Human Data Quality Engineer (Founding Team)

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 10 hours agoStatus: Live

Job Summary

Build quality frameworks and evaluation rubrics for complex human-data programs that power frontier AI models. Partner with clients and AI labs to translate model needs into robust annotation and study design requirements, then engineer quality upstream across recruitment, screening, training, and launch readiness. Create scalable quality systems, automated checks, dashboards, and monitoring using Python and SQL, investigating data integrity issues and turning insights into durable improvements.
Location: Mexico, United States
Workplace: Remote
Employment Type: Full time
Job Function: QA, Test & Release Engineering
Seniority: Mid level

Key Responsibilities

  • •Design quality frameworks for human data programs, including evaluation rubrics and launch readiness.
  • •Partner with clients as a strategic thought partner to challenge annotation schemas and data requirements.
  • •Engineer quality upstream across the operational workflow (recruitment, screening, training, guidelines, calibration).
  • •Build scalable quality systems, measurement frameworks, and automated checks using Python and SQL.
  • •Investigate data quality/integrity issues and build monitoring/reporting dashboards; translate findings into scalable improvements.

Pay and Benefits

Equity and Bonus:Equity
Perks:Remote WorkEquity

Key Requirements

  • •5+ years of experience building quality, evaluation, or annotation systems in AI/ML/LLMs or human data environments.
  • •Strong Python and SQL skills, with a data-driven approach to solving complex quality problems.
  • •Understanding of machine learning pipelines and how human data impacts model performance.
  • •Strong analytical and statistical thinking, including experience designing scalable quality frameworks.
  • •Credibility to collaborate with stakeholders at frontier labs and guide partners across client and internal teams.
Experience:5+ yearsAIMachine learningLLMsHuman dataAnnotationHuman evaluation
Skills:Stakeholder managementAnalytical thinkingStatistical thinkingCommunicationBuilder mindset
Languages:English
Tech Stack:PythonSQLMachine learningLLMsRLHFDashboardsMonitoringReporting

Company Brief

Prolific
Operates an online participant recruitment platform connecting researchers and companies with vetted participants for academic and market research studies, offering survey hosting, participant management, and data quality tools.
Industry: SaaS
Company Size: Medium (51 to 250 employees)
Growth: Growth Stage Startup
Funding: Series A
Headquarters: Oxford, United Kingdom
Founded: 2014
WebsiteLinkedIn