Software Engineer L5 - AI Observability & Agent Evaluation

Netflix
United States
Workplace: OnsiteFull timeUSD 388,000 - 619,000 annuallyFunction: Software EngineeringSkills: ["Cross-functional collaboration","Clear communication","Hands-on execution"]

Build Netflix’s AI Observability platform, providing end-to-end observability for ML and GenAI systems across online inference, batch scoring, feature pipelines, and agent orchestration. Develop reusable frameworks for monitoring model performance, data quality, drift, and degradation, and extend LLM/agent evaluation capabilities. Lead build-vs-buy efforts for observability and evaluation tooling, integrating vendor SDKs, APIs, and dashboards so teams can ship AI-powered products with confidence.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Netflix
Netflix
4 days ago

Software Engineer L5 - AI Observability & Agent Evaluation

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 23 hours agoStatus: Live

Job Summary

Build Netflix’s AI Observability platform, providing end-to-end observability for ML and GenAI systems across online inference, batch scoring, feature pipelines, and agent orchestration. Develop reusable frameworks for monitoring model performance, data quality, drift, and degradation, and extend LLM/agent evaluation capabilities. Lead build-vs-buy efforts for observability and evaluation tooling, integrating vendor SDKs, APIs, and dashboards so teams can ship AI-powered products with confidence.
Location: United States
Workplace: Onsite
Employment Type: Full time
Job Function: Software Engineering
Seniority: Mid level

Key Responsibilities

  • •Build observability framework and platform capabilities for metrics, logs, and distributed traces across online inference, batch scoring, feature pipelines, and agent orchestration.
  • •Create primitives for teams to monitor model performance, data quality, drift, and degradation on their own systems.
  • •Build and extend evaluation frameworks for LLM and agentic systems, including grounding/hallucination, task success, tool use, trajectory correctness, and LLM-as-a-judge or human-in-the-loop scoring.
  • •Lead build-vs-buy evaluations for observability and evaluation tooling, owning SDKs, connectors, and APIs for vendor integration.
  • •Deliver dashboarding, alerting, and SLO/SLI building blocks so teams can track model performance, latency, cost, and reliability.

Pay and Benefits

Salary: USD 388,000 - 619,000 annually
Equity and Bonus:Equity
Perks:Health Insurance401kEquityPaid Leave

Key Requirements

  • •Experience in software, AI/ML, or platform engineering with hands-on production observability, monitoring, or ML/LLM evaluation.
  • •Proven track record designing standards, libraries, SDKs, or frameworks that other teams adopt.
  • •Strong coding skills in Python and at least one of Java, Go, or Scala, with experience building production services.
  • •Practical experience operating ML models in production (online serving and/or batch), including drift, model performance, and data quality.
  • •Hands-on experience with modern observability stacks such as Prometheus/Grafana, Datadog, OpenTelemetry, ELK/OpenSearch, or Jaeger/Tempo.
Experience:AI/MLGenAILLM evaluationObservabilityPlatform engineering
Skills:Cross-functional collaborationClear communicationHands-on execution
Tech Stack:PythonJavaGoScalaPrometheusGrafanaDatadogOpenTelemetryELKOpenSearchJaegerTempoAWSGCPAzureMicroservicesDistributed systems

Company Brief

Netflix
Global subscription streaming service that produces and distributes movies, TV series, and games across multiple devices, operating worldwide with extensive original content and localized offerings.
Industry: Streaming Platforms
Company Size: Enterprise (1,001+ employees)
Revenue: USD 1B+
Growth: Public Company
Valuation: Public Company (Market Cap in USD)
Funding: IPO / Publicly Listed
Headquarters: Los Gatos, United States
Founded: 1997
Glassdoor
Glassdoor: 4.1
WebsiteLinkedInGlassdoor