AI Evaluation Engineer (QA)

Appnovation
Montreal
Workplace: OnsiteContractFunction: Data Science & Machine LearningExperience: 4+ yearsEducation: bachelorsSkills: ["Analytical skills","Problem-solving","Decision-making","Communication","Attention to detail"]

Build and run AI/LLM quality evaluations at scale, from small human UAT batches to hundreds of thousands or millions of automated tests. You’ll measure factual grounding and accuracy lift with statistical rigor, design load and quality tests as the evaluation corpus grows, and maintain test plans and quality gates. Automate regression and evaluation suites and integrate them into CI/CD, reporting quality metrics clearly and partnering with engineering to validate fixes.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Appnovation
Appnovation
1 day ago

AI Evaluation Engineer (QA)

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 12 hours agoStatus: Live

Job Summary

Build and run AI/LLM quality evaluations at scale, from small human UAT batches to hundreds of thousands or millions of automated tests. You’ll measure factual grounding and accuracy lift with statistical rigor, design load and quality tests as the evaluation corpus grows, and maintain test plans and quality gates. Automate regression and evaluation suites and integrate them into CI/CD, reporting quality metrics clearly and partnering with engineering to validate fixes.
Location: Montreal
Workplace: Onsite
Employment Type: Contract
Job Function: Data Science & Machine Learning
Seniority: Mid level

Key Responsibilities

  • •Run large-scale evaluations across big question sets, from human UAT batches to automated evals.
  • •Statistically measure factual grounding and accuracy lift (before/after), beyond human A/B testing.
  • •Build a metrics framework to demonstrate quality improvement over time.
  • •Design load and quality tests as the corpus evolves, and define test plans and quality gates.
  • •Automate regression and evaluation suites and integrate them into CI/CD; report metrics and collaborate with engineering to verify fixes.

Key Requirements

  • •Bachelor’s degree in a technical field or equivalent experience.
  • •4+ years in QA / test engineering, with exposure to data/ML systems.
  • •Strong Python skills and data-science techniques to measure factual grounding and answer quality.
  • •Experience with LLM evaluation frameworks and statistical analysis.
  • •Ability to build large-scale automated evaluation harnesses and lighter human-in-the-loop A/B tests.
Experience:4+ yearsData/MLLLMQA automation
Education:Bachelor's
Skills:Analytical skillsProblem-solvingDecision-makingCommunicationAttention to detail
Tech Stack:PythonLLMA/B testingCI/CDUATLoad testingTest automationRegression testingStatistical analysis

Company Brief

Appnovation
Provides digital transformation, design, and engineering services, helping enterprises build customer experiences, digital platforms, and integrations using open-source and cloud technologies across industries.
Industry: Consulting
Company Size: Enterprise (1,001+ employees)
Growth: Established Company
Funding: Bootstrapped
Headquarters: Vancouver, Canada
Founded: 2007
WebsiteLinkedIn