Senior Technical Program Manager, Human-in-the-Loop Operations

Roblox
San Mateo
Workplace: OnsiteFull timeUSD 226,400 - 277,350 annuallyFunction: Program & Project Management (PMO)Skills: ["Written communication","Verbal communication","Cross-functional alignment","Critical thinking","Judgment"]

Own the end-to-end lifecycle of Roblox’s AI data labeling and model evaluation operations. Lead data programs that translate ML requirements into structured annotation tasks, oversee contractor execution and quality review, and deliver datasets to AI teams. Define ground-truth labeling guidelines and quality frameworks, drive SLAs and KPIs, and partner cross-functionally to evolve workflows from manual to AI-assisted, LLM-accelerated operations.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Roblox
Roblox
3 days ago

Senior Technical Program Manager, Human-in-the-Loop Operations

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 3 hours agoStatus: Live

Job Summary

Own the end-to-end lifecycle of Roblox’s AI data labeling and model evaluation operations. Lead data programs that translate ML requirements into structured annotation tasks, oversee contractor execution and quality review, and deliver datasets to AI teams. Define ground-truth labeling guidelines and quality frameworks, drive SLAs and KPIs, and partner cross-functionally to evolve workflows from manual to AI-assisted, LLM-accelerated operations.
Location: San Mateo
Workplace: Onsite
Employment Type: Full time
Job Function: Program & Project Management (PMO)
Seniority: Manager level

Key Responsibilities

  • •Lead end-to-end labeling and model evaluation lifecycle workflows across Roblox’s AI teams, from structured annotation tasks to contractor execution, quality review, and dataset delivery.
  • •Develop and iterate labeling guidelines, annotation schemas, and quality frameworks for Roblox’s data types (e.g., 3D mesh quality, texture coherence, search relevance, novelty detection, NPC behavior, AI-generated Luau code assessment).
  • •Oversee a multi-vendor, distributed remote contractor workforce, managing allocation, productivity SLAs, quality calibration, and budget forecasting across teams.
  • •Drive the shift to AI-assisted workflows by partnering with Engineering to implement LLM pre-labeling, LLM-as-a-Judge evaluation, and automated QA routing.
  • •Own operational excellence and KPIs by defining and tracking throughput, inter-annotator agreement, cost per label, and data quality metrics; resolve bottlenecks and improve processes.

Pay and Benefits

Salary: USD 226,400 - 277,350 annually
Equity and Bonus:Equity

Key Requirements

  • •5+ years in Technical Program Management, Data Operations, or Operations Management in an AI/ML or data-intensive environment.
  • •Hands-on experience with data labeling, annotation, or model evaluation, including annotation schemas, labeling guidelines, and large-scale quality control.
  • •Experience managing external vendors and distributed contractor workforces, including workforce planning, quality oversight, and budget management.
  • •Strong understanding of the ML lifecycle and the role of human-labeled data in training, fine-tuning, and evaluating models.
  • •Proficiency in SQL and Python for querying datasets, analyzing label quality, and building operational dashboards; strong written and verbal communication skills.
Experience:AI/MLData operationsOperations management
Skills:Written communicationVerbal communicationCross-functional alignmentCritical thinkingJudgment
Languages:English
Tech Stack:SQLPythonLLMLuauCursorClaudeGemini

Company Brief

Roblox
Operates an online platform for user-generated games and virtual experiences, enabling creators to build, publish, and monetize interactive 3D experiences for a global community of players.
Industry: Gaming
Company Size: Enterprise (1,001+ employees)
Revenue: USD 1B+
Growth: Public Company
Valuation: Public Company (Market Cap in USD)
Funding: IPO / Publicly Listed
Headquarters: San Mateo, United States
Founded: 2004
WebsiteLinkedIn