Anthropic

3 months ago

Research Engineer, Model Evaluations

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI

Last checked: 8 hours agoStatus: Live

Job Summary

Design and run evaluations of Claude’s capabilities across reasoning, agentic behavior, knowledge, and safety; build scalable, reliable evaluation infrastructure; own dashboards to monitor model health; debug anomalous results during training, and collaborate with researchers to translate insights into measurable, defensible metrics that advance AI safety and performance.

Location: San Francisco, New York

Workplace: Hybrid

Employment Type: Full time

Job Function: Research & Scientific (R&D)

Key Responsibilities

•Design and run new evaluations of Claude's capabilities — reasoning, agentic behavior, knowledge, safety properties — and produce visualizations that make the results legible to researchers and decision-makers
•Build and harden the distributed eval execution platform so hundreds of evals run reliably against checkpoints throughout production RL training runs
•Own the dashboards researchers and leadership use to monitor model health during training, improving signal-to-noise, reducing latency, and making regressions impossible to miss
•Debug anomalous eval results mid-training-run, determine whether the cause is a model change or an infrastructure issue, and communicate the answer clearly under time pressure
•Improve the tooling, libraries, and workflows researchers use to implement and iterate on evaluations

Pay and Benefits

Salary: USD 320,000 - 485,000 annually

Key Requirements

•Strong Python programming skills, including production or research infrastructure
•Experience building or operating distributed systems, data pipelines, or other infrastructure that needs to be reliable at scale
•Clear written and verbal communication, especially when explaining technical results to non-specialists
•Comfort operating in an on-call or production-support capacity when training runs are live
•Care about the societal impacts of your work and an interest in steering powerful AI to be safe and beneficial

Experience:Artificial intelligenceMachine learningResearch

Education:Bachelor's

Skills:CommunicationCollaborationPresentation

Languages:English

Tech Stack:PythonDistributed systemsData pipelinesObservabilityDashboardingRL training

Company Brief

Anthropic

Develops large-scale AI systems and safety research to create reliable, steerable, and interpretable AI assistants and models for commercial and research applications.

Industry: AI & Machine Learning

Company Size: Enterprise (1,001+ employees)

Growth: Scaleup

Valuation: Unicorn (USD 1B+)

Funding: Series C

Headquarters: San Francisco, United States

Founded: 2021

Website LinkedIn

Loading job details...

Preparing the role view and application actions.

Anthropic

3 months ago

Research Engineer, Model Evaluations

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI

Last checked: 8 hours agoStatus: Live

Job Summary

Location: San Francisco, New York

Workplace: Hybrid

Employment Type: Full time

Job Function: Research & Scientific (R&D)

Key Responsibilities

•Design and run new evaluations of Claude's capabilities — reasoning, agentic behavior, knowledge, safety properties — and produce visualizations that make the results legible to researchers and decision-makers
•Build and harden the distributed eval execution platform so hundreds of evals run reliably against checkpoints throughout production RL training runs
•Own the dashboards researchers and leadership use to monitor model health during training, improving signal-to-noise, reducing latency, and making regressions impossible to miss
•Debug anomalous eval results mid-training-run, determine whether the cause is a model change or an infrastructure issue, and communicate the answer clearly under time pressure
•Improve the tooling, libraries, and workflows researchers use to implement and iterate on evaluations

Pay and Benefits

Salary: USD 320,000 - 485,000 annually

Key Requirements

•Strong Python programming skills, including production or research infrastructure
•Experience building or operating distributed systems, data pipelines, or other infrastructure that needs to be reliable at scale
•Clear written and verbal communication, especially when explaining technical results to non-specialists
•Comfort operating in an on-call or production-support capacity when training runs are live
•Care about the societal impacts of your work and an interest in steering powerful AI to be safe and beneficial

Experience:Artificial intelligenceMachine learningResearch

Education:Bachelor's

Skills:CommunicationCollaborationPresentation

Languages:English

Tech Stack:PythonDistributed systemsData pipelinesObservabilityDashboardingRL training

Company Brief

Anthropic

Develops large-scale AI systems and safety research to create reliable, steerable, and interpretable AI assistants and models for commercial and research applications.

Industry: AI & Machine Learning

Company Size: Enterprise (1,001+ employees)

Growth: Scaleup

Valuation: Unicorn (USD 1B+)

Funding: Series C

Headquarters: San Francisco, United States

Founded: 2021

Website LinkedIn