Netflix

1 month ago

Data Scientist 5 - AI Evals

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI

Last checked: 22 hours agoStatus: Live

Job Summary

Architect and operate AI evaluation systems for GenAI in Netflix Games, partnering with research and shipping robust eval pipelines from R&D to production. Build golden datasets, adversarial challenge sets, and synthetic testbeds to measure model quality for player-facing experiences and internal agentic tools. Design experiments for trade-offs between technical attributes and end-user experience, and lead safety red-teaming to mitigate toxicity, hallucinations, jailbreaks, and off-character behavior.

Location: Panama

Workplace: Remote

Employment Type: Full time

Job Function: Data Science & Machine Learning

Seniority: Mid level

Key Responsibilities

•Partner with GenAI research to transition GenAI product work from R&D into production at scale and in live operations.
•Build and operate evaluation pipelines for production-stage GenAI experiences using automated metrics, LLM-as-a-Judge, human-in-the-loop grading, and simulation-based testing.
•Curate golden datasets, test suites, adversarial challenge sets, and synthetic testbeds to establish ground-truth performance across generative tasks.
•Design experiments to evaluate trade-offs between technical attributes and end-user experience quality in real-time game environments.
•Develop red-teaming protocols and safety evaluators to detect and mitigate toxicity, hallucinations, jailbreaks, and out-of-character behavior.

Pay and Benefits

Salary: USD 372,000 - 600,000 annually

Perks:Health Insurance401kEquityPaid Leave

Key Requirements

•Ph.D. in Data Science, Computer Science, Statistics, Cognitive Science, or a related quantitative field.
•4+ years of industry experience in Data Science, ML, or AI, with strong experimental design, causal inference, A/B testing, and uncertainty quantification.
•Experience with modern AI Evals and observability frameworks, including OpenAI/Anthropic evaluation suites.
•Proven track record evaluating LLM and agentic systems, including prompt engineering, RAG Evals, and agentic Evals.
•Ability to collaborate with game teams to translate creative objectives into data specifications.

Experience:4+ years

Education:PhD / Doctorate

Skills:CollaborationExperimental designUncertainty quantificationCross-functional translation

Tech Stack:GenAILLM-as-a-JudgeOpenAI evaluation suitesAnthropic evaluation suitesA/B testingCausal inferenceRAG EvalsAgentic EvalsPrompt engineeringMLOpsObservability frameworksGolden datasetsAdversarial challenge setsSynthetic testbedsRed-teaming

Company Brief

Netflix

Global subscription streaming service that produces and distributes movies, TV series, and games across multiple devices, operating worldwide with extensive original content and localized offerings.

Industry: Streaming Platforms

Company Size: Enterprise (1,001+ employees)

Revenue: USD 1B+

Growth: Public Company

Valuation: Public Company (Market Cap in USD)

Funding: IPO / Publicly Listed

Headquarters: Los Gatos, United States

Founded: 1997

Glassdoor: 4.1

Website LinkedIn Glassdoor

Loading job details...

Preparing the role view and application actions.

Netflix

1 month ago

Data Scientist 5 - AI Evals

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI

Last checked: 22 hours agoStatus: Live

Job Summary

Location: Panama

Workplace: Remote

Employment Type: Full time

Job Function: Data Science & Machine Learning

Seniority: Mid level

Key Responsibilities

•Partner with GenAI research to transition GenAI product work from R&D into production at scale and in live operations.
•Build and operate evaluation pipelines for production-stage GenAI experiences using automated metrics, LLM-as-a-Judge, human-in-the-loop grading, and simulation-based testing.
•Curate golden datasets, test suites, adversarial challenge sets, and synthetic testbeds to establish ground-truth performance across generative tasks.
•Design experiments to evaluate trade-offs between technical attributes and end-user experience quality in real-time game environments.
•Develop red-teaming protocols and safety evaluators to detect and mitigate toxicity, hallucinations, jailbreaks, and out-of-character behavior.

Pay and Benefits

Salary: USD 372,000 - 600,000 annually

Perks:Health Insurance401kEquityPaid Leave

Key Requirements

•Ph.D. in Data Science, Computer Science, Statistics, Cognitive Science, or a related quantitative field.
•4+ years of industry experience in Data Science, ML, or AI, with strong experimental design, causal inference, A/B testing, and uncertainty quantification.
•Experience with modern AI Evals and observability frameworks, including OpenAI/Anthropic evaluation suites.
•Proven track record evaluating LLM and agentic systems, including prompt engineering, RAG Evals, and agentic Evals.
•Ability to collaborate with game teams to translate creative objectives into data specifications.

Experience:4+ years

Education:PhD / Doctorate

Skills:CollaborationExperimental designUncertainty quantificationCross-functional translation

Company Brief

Netflix

Global subscription streaming service that produces and distributes movies, TV series, and games across multiple devices, operating worldwide with extensive original content and localized offerings.

Industry: Streaming Platforms

Company Size: Enterprise (1,001+ employees)

Revenue: USD 1B+

Growth: Public Company

Valuation: Public Company (Market Cap in USD)

Funding: IPO / Publicly Listed

Headquarters: Los Gatos, United States

Founded: 1997

Glassdoor: 4.1

Website LinkedIn Glassdoor