Machine Learning Engineer II, GenAI Evaluation Framework

AutoTrader.com
Atlanta
Workplace: HybridFull timeUSD 92,300 - 153,900 annuallyFunction: Data Science & Machine LearningExperience: 2+ yearsEducation: bachelorsSkills: ["Roadmap ownership","Prioritization","Tradeoff communication","Written communication","Documentation"]

Own and build Cox’s GenAI Evaluation Framework (GEF), an internal product that measures production AI agent quality against standards. You’ll roadmap and sequence onboarding for adopting teams, implement LLM-as-judge, rule-based and statistical evaluators, and operate evaluation pipelines that publish results to dashboards. Create and maintain golden datasets, integrate evaluation checks into CI/CD, and provide monthly quality and cost-per-quality reporting for governance reviews.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
AutoTrader.com
AutoTrader.com
12 hours ago

Machine Learning Engineer II, GenAI Evaluation Framework

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 12 hours agoStatus: Live

Job Summary

Own and build Cox’s GenAI Evaluation Framework (GEF), an internal product that measures production AI agent quality against standards. You’ll roadmap and sequence onboarding for adopting teams, implement LLM-as-judge, rule-based and statistical evaluators, and operate evaluation pipelines that publish results to dashboards. Create and maintain golden datasets, integrate evaluation checks into CI/CD, and provide monthly quality and cost-per-quality reporting for governance reviews.
Location: Atlanta
Workplace: Hybrid
Employment Type: Full time
Job Function: Data Science & Machine Learning
Seniority: Mid level

Key Responsibilities

  • •Own and evolve the GEF product, including the roadmap, what ships, what gets retired, and onboarding sequencing for adopting teams.
  • •Implement and maintain LLM-as-judge, rule-based, and statistical evaluators, calibrated against human-labeled baselines.
  • •Build and operate evaluation pipelines that score agent traffic on a schedule and publish results to dashboards and the Agent Registry.
  • •Assemble, version, and maintain golden datasets with product teams and subject matter experts.
  • •Integrate evaluation checks into CI/deployment pipelines and provide monthly quality health, adoption, cost-per-quality reporting, and governance audit evidence.
Travel: Low travel

Pay and Benefits

Salary: USD 92,300 - 153,900 annually
Perks:Health InsuranceDentalVision401kPaid Leave

Key Requirements

  • •2+ years of professional machine learning engineering experience with a Bachelor’s degree, or up to 2 years combined with a Master’s in a quantitative field.
  • •Must currently be authorized to work in the United States without current or future sponsorship.
  • •Must currently live within a commutable distance to Atlanta.
  • •Experience owning the direction of a technical product, internal tool, library, or platform used by other teams.
  • •Strong Python experience with testing, version control, code review, and continuous integration; hands-on experience building with large language models.
Experience:2+ yearsMachine learningGenerative AILLM evaluationDeveloper toolsAI governance
Education:Bachelor's
Skills:Roadmap ownershipPrioritizationTradeoff communicationWritten communicationDocumentation
Tech Stack:PythonLLMRetrievalAPI integrationAgent frameworksAWSAWS BedrockDeepEvalLangfuseRAGASOpenTelemetryDatadogCICI/CD

Eligibility

Work Authorization:Authorization required. Sponsorship not provided.

Company Brief

AutoTrader.com
Operates a leading online marketplace for buying and selling new and used vehicles, connecting consumers, dealers, and automotive service providers with vehicle listings, pricing tools, reviews, and financing resources.
Industry: Online Marketplaces
Company Size: Enterprise (1,001+ employees)
Growth: Established Company
Headquarters: Atlanta, United States
Founded: 1997
WebsiteLinkedIn