AI Evaluation Engineer (QA)

Appnovation
São Paulo, Toronto
Workplace: OnsiteContractFunction: Data Science & Machine LearningExperience: 4+ yearsEducation: bachelorsSkills: ["Analytical","Problem solving","Decision-making","Detail-oriented","Communication"]

Own AI answer-quality evaluations at scale, from small human UAT batches to millions of automated runs. Use statistical methods to measure factual grounding and accuracy lift, and build a metrics framework that proves quality improvements over time. Design load and quality tests, define quality gates, automate regression/eval suites, and integrate them into CI/CD. Partner with engineering to reproduce, triage, and verify fixes while communicating results to technical and non-technical stakeholders.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Appnovation
Appnovation
1 day ago

AI Evaluation Engineer (QA)

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 12 hours agoStatus: Live

Job Summary

Own AI answer-quality evaluations at scale, from small human UAT batches to millions of automated runs. Use statistical methods to measure factual grounding and accuracy lift, and build a metrics framework that proves quality improvements over time. Design load and quality tests, define quality gates, automate regression/eval suites, and integrate them into CI/CD. Partner with engineering to reproduce, triage, and verify fixes while communicating results to technical and non-technical stakeholders.
Location: São Paulo, Toronto
Workplace: Onsite
Employment Type: Contract
Job Function: Data Science & Machine Learning
Seniority: Mid level

Key Responsibilities

  • •Run evaluations at scale across large question sets, from human UAT batches to automated evaluations at very large volume.
  • •Statistically measure factual grounding and accuracy lift (before/after), beyond human A/B testing.
  • •Build a metrics framework that quantifies quality improvement using source-supported grounding and better answer outcomes.
  • •Design load and quality tests as the corpus scales, defining test plans, test cases, and quality gates.
  • •Automate regression and evaluation suites and integrate them into CI/CD; report metrics and collaborate with engineering to verify fixes.

Key Requirements

  • •Bachelor’s degree in a technical field or equivalent experience.
  • •4+ years in QA/test engineering with exposure to data/ML systems.
  • •Strong Python and data-science techniques for measuring factual grounding and answer quality.
  • •Experience with LLM evaluation frameworks and statistical analysis.
  • •Ability to build automated, large-scale evaluation harnesses and run lighter human-in-the-loop A/B tests.
Experience:4+ yearsMLData/MLLLMLife Sciences
Education:Bachelor's
Skills:AnalyticalProblem solvingDecision-makingDetail-orientedCommunication
Languages:English
Tech Stack:PythonLLM evaluation frameworksStatistical analysisCI/CDA/B testingTest automation frameworks

Company Brief

Appnovation
Provides digital transformation, design, and engineering services, helping enterprises build customer experiences, digital platforms, and integrations using open-source and cloud technologies across industries.
Industry: Consulting
Company Size: Enterprise (1,001+ employees)
Growth: Established Company
Funding: Bootstrapped
Headquarters: Vancouver, Canada
Founded: 2007
WebsiteLinkedIn