Scale AI

1 week ago

Senior Software Engineer, Agent Oversight

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI

Last checked: 3 hours agoStatus: Live

Job Summary

Build platform infrastructure for production agent oversight, enabling agents to be observed, evaluated, and improved at scale. You’ll design core capabilities for deploying, monitoring, and evaluating agentic applications, including observability tooling, evaluation harnesses, and data pipelines for telemetry and performance metrics. Partner closely with ML engineers on evaluation and improvement workflows while owning end-to-end system design, implementation, debugging, and testing for reliable, scalable, and observable services.

Location: San Francisco, New York

Workplace: Onsite

Employment Type: Full time

Job Function: Software Engineering

Seniority: Mid level

Key Responsibilities

•Design and build core platform capabilities for deploying, monitoring, and evaluating agentic applications in production.
•Build reliable APIs and data pipelines to capture agent telemetry, evaluation signals, and performance metrics at scale.
•Collaborate with ML engineers on evaluation and improvement systems, using enough ML fluency to reason about model behavior and improvement loops.
•Own reliability, scalability, and observability of platform components serving multiple concurrent enterprise and government customers.
•Deliver features end-to-end—system design, implementation, debugging, and testing—and participate in high-velocity experimentation to validate capabilities against real usage.

Pay and Benefits

Salary: USD 216,000 - 270,000 annually

Equity and Bonus:Equity

Perks:Health InsuranceDentalVisionRetirement BenefitsLearning BudgetPaid LeaveCommuter Benefits

Key Requirements

•4+ years of professional software engineering experience with strong fundamentals in backend/distributed systems, APIs, and data pipeline design.
•Hands-on experience building production software for ML/LLM-powered products or platforms (e.g., evaluation systems, observability/monitoring, experimentation infrastructure, telemetry/data pipelines).
•Working knowledge of how LLM/ML systems behave in production, including evaluation signals, failure modes, prompt/tool-calling workflows, experiment results, and data quality tradeoffs.
•Partner closely with ML engineers/applied researchers to turn prototypes, eval loops, or model-improvement workflows into reliable platform capabilities.
•Experience building infrastructure/platforms that other teams build on top of, and a track record owning features end-to-end from design through production.

Experience:ML/LLM-powered productsAgentic AIInfrastructure platforms

Skills:Problem-solvingCross-functional collaborationIndependent workMentoring

Languages:English

Company Brief

Scale AI

Provides data labeling, annotation, and infrastructure services to accelerate machine learning and AI development. Supplies high-quality training data, tooling, and APIs for customers in autonomous vehicles, mapping, robotics, and enterprise AI applications.

Industry: AI & Machine Learning

Company Size: Enterprise (1,001+ employees)

Growth: Scaleup

Valuation: Unicorn (USD 1B+)

Funding: Series E+

Headquarters: San Francisco, United States

Founded: 2016

Website LinkedIn

Loading job details...

Preparing the role view and application actions.

Scale AI

1 week ago

Senior Software Engineer, Agent Oversight

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI

Last checked: 3 hours agoStatus: Live

Job Summary

Location: San Francisco, New York

Workplace: Onsite

Employment Type: Full time

Job Function: Software Engineering

Seniority: Mid level

Key Responsibilities

•Design and build core platform capabilities for deploying, monitoring, and evaluating agentic applications in production.
•Build reliable APIs and data pipelines to capture agent telemetry, evaluation signals, and performance metrics at scale.
•Collaborate with ML engineers on evaluation and improvement systems, using enough ML fluency to reason about model behavior and improvement loops.
•Own reliability, scalability, and observability of platform components serving multiple concurrent enterprise and government customers.
•Deliver features end-to-end—system design, implementation, debugging, and testing—and participate in high-velocity experimentation to validate capabilities against real usage.

Pay and Benefits

Salary: USD 216,000 - 270,000 annually

Equity and Bonus:Equity

Perks:Health InsuranceDentalVisionRetirement BenefitsLearning BudgetPaid LeaveCommuter Benefits

Key Requirements

•4+ years of professional software engineering experience with strong fundamentals in backend/distributed systems, APIs, and data pipeline design.
•Hands-on experience building production software for ML/LLM-powered products or platforms (e.g., evaluation systems, observability/monitoring, experimentation infrastructure, telemetry/data pipelines).
•Working knowledge of how LLM/ML systems behave in production, including evaluation signals, failure modes, prompt/tool-calling workflows, experiment results, and data quality tradeoffs.
•Partner closely with ML engineers/applied researchers to turn prototypes, eval loops, or model-improvement workflows into reliable platform capabilities.
•Experience building infrastructure/platforms that other teams build on top of, and a track record owning features end-to-end from design through production.

Experience:ML/LLM-powered productsAgentic AIInfrastructure platforms

Skills:Problem-solvingCross-functional collaborationIndependent workMentoring

Languages:English

Company Brief

Scale AI

Industry: AI & Machine Learning

Company Size: Enterprise (1,001+ employees)

Growth: Scaleup

Valuation: Unicorn (USD 1B+)

Funding: Series E+

Headquarters: San Francisco, United States

Founded: 2016

Website LinkedIn