AI Evaluation Engineer
Dialpad
Vancouver, Kitchener
Workplace: OnsiteFull timeCAD 104,000 - 119,750 annuallyFunction: Data Science & Machine LearningExperience: 3+ yearsSkills: ["Analytical","Communication","Collaboration","Documentation","Reporting"]Own evaluation coverage for Dialpad’s agentic AI voice and chat systems. Design validation strategies across staging, beta, and release candidates; build regression evaluations, A/B comparisons, and red-teaming analyses; and co-develop LLM-judge metrics with calibration and prompt refinement. Curate scenario/benchmark datasets, set up annotation job pipelines, and maintain QA tooling to make recurring evaluations scalable. Triage failures and partner with applied science, engineering, and Product QA for release readiness decisions.
