Senior Software Engineer, AI Evals

Sentry
San Francisco
Workplace: HybridFull timeUSD 240,000 - 280,000 annuallyFunction: Software EngineeringExperience: 5+ yearsEducation: bachelorsSkills: ["Communication","Collaboration","Problem-solving","Teamwork","Critical thinking"]

Senior Software Engineer on Sentry’s AI/ML team responsible for building evaluation infrastructure that measures accuracy, reliability, and real-world performance of AI systems. You’ll design datasets, benchmarks, and test harnesses, build metrics pipelines, and collaborate with applied AI engineers and product leaders to translate goals into measurable criteria for AI initiatives from experimentation to production monitoring.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Sentry
Sentry
7 months ago

Senior Software Engineer, AI Evals

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 10 hours agoStatus: Live

Job Summary

Senior Software Engineer on Sentry’s AI/ML team responsible for building evaluation infrastructure that measures accuracy, reliability, and real-world performance of AI systems. You’ll design datasets, benchmarks, and test harnesses, build metrics pipelines, and collaborate with applied AI engineers and product leaders to translate goals into measurable criteria for AI initiatives from experimentation to production monitoring.
Location: San Francisco
Workplace: Hybrid
Employment Type: Full time
Job Function: Software Engineering

Key Responsibilities

  • •Design and build robust evaluation frameworks to measure accuracy, reliability, regressions, and edge cases in AI systems
  • •Create and curate high-quality datasets, golden test cases, and benchmarks grounded in real production data
  • •Build automated test harnesses and metrics pipelines to continuously evaluate models, prompts, and agentic workflows
  • •Partner closely with applied AI engineers and product leaders to define what “good” looks like and translate it into measurable criteria
  • •Own the evaluation lifecycle for major AI initiatives, from early experimentation through production monitoring

Pay and Benefits

Salary: USD 240,000 - 280,000 annually
Equity and Bonus:Equity

Key Requirements

  • •Minimum 5+ years of professional experience with a Bachelor’s degree in computer science, machine learning, or a related field
  • •Experience building testing, evaluation, or data infrastructure for complex systems (AI/ML experience strongly preferred)
  • •Comfort writing production-quality code (we use Python and TypeScript)
  • •Experience working with structured and unstructured datasets, labeling workflows, or data quality pipelines
  • •Familiarity with modern ML systems and evaluation techniques (e.g., offline metrics, online evaluation, regression testing for models or prompts)
Experience:5+ yearsAI/MLSoftware engineering
Education:Bachelor's
Skills:CommunicationCollaborationProblem-solvingTeamworkCritical thinking
Languages:English
Tech Stack:PythonTypeScriptAI/MLDatasetsEvaluation frameworks

Company Brief

Sentry
Provides an error monitoring and performance observability platform for developers, helping teams identify, triage, and resolve application issues across frontend and backend systems with real-time alerts and diagnostic context.
Industry: Developer Tools
Company Size: Large (251 to 1,000 employees)
Growth: Scaleup
Headquarters: San Francisco, United States
WebsiteLinkedIn