AI Engineer, Evaluation

Distyl
San Francisco
Workplace: HybridFull time150,000 - 250,000 annuallyFunction: Data Science & Machine LearningExperience: 2+ yearsSkills: ["Python","Testing","Regression","Pipelines","LLMs","Evaluation pipelines","Git","Code quality"]

Design and implement evaluation frameworks for AI systems in production, building pipelines and graders that translate human judgments into measurable test cases. Collaborate across engineers and domain experts to ensure evaluation drives system iteration, prompts, model selection, and release readiness in a hybrid, production-focused environment.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Distyl
Distyl
2 months ago

AI Engineer, Evaluation

âś“ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 6 hours agoStatus: Live

Job Summary

Design and implement evaluation frameworks for AI systems in production, building pipelines and graders that translate human judgments into measurable test cases. Collaborate across engineers and domain experts to ensure evaluation drives system iteration, prompts, model selection, and release readiness in a hybrid, production-focused environment.
Location: San Francisco
Workplace: Hybrid
Employment Type: Full time
Job Function: Data Science & Machine Learning

Key Responsibilities

  • •Design and implement evaluation frameworks that enable Evaluation-Driven Development for AI systems deployed in customer environments
  • •Define how system quality is measured in each domain, ensuring evaluation signals reflect real user needs, domain constraints, and business objectives
  • •Build and maintain golden test cases and regression suites in Python, using human-authored and AI-assisted test generation
  • •Develop and maintain evaluation pipelines—offline and online—that integrate into system iteration loops, informing prompt design and model selection
  • •Define, calibrate, and operate LLM-based graders, aligning automated judgments with expert human assessments and refining signals as systems evolve
Travel: Medium travel

Pay and Benefits

Salary: 150,000 - 250,000 annually
Equity and Bonus:Equity
Perks:Health InsuranceDentalVision401kEquityCommuter BenefitsMeal Allowance

Key Requirements

  • •2+ years of software engineering experience
  • •Strong Python engineering skills and ability to build production-grade evaluation pipelines
  • •Experience with Evaluation-Driven or Experiment-Driven Development and understanding of metric pitfalls
  • •Ability to translate human judgment into code and encode judgments into test cases, scoring functions, and graders
  • •Systems-oriented mindset to design evaluation systems that support fast iteration and production trust
Experience:2+ yearsAIEnterprise software
Skills:PythonTestingRegressionPipelinesLLMsEvaluation pipelinesGitCode quality
Languages:English
Tech Stack:PythonLLMsTestingRegressionPipelinesProductionGit

Company Brief

Distyl
Builds enterprise-grade AI systems and integration services (Distillery) to help Fortune 500 companies become AI-native, delivering measurable operational impact across healthcare, telecom, manufacturing, and finance.
Industry: AI & Machine Learning
Company Size: Medium (51 to 250 employees)
Growth: Scaleup
Valuation: Unicorn (USD 1B+)
Funding: Series B
Headquarters: San Francisco, United States
Founded: 2022
WebsiteLinkedInGlassdoor