Senior Product Operations Manager, Evaluation Quality

Harvey
San Francisco
Workplace: HybridFull timeUSD 155,400 - 233,200 annuallyFunction: Business OperationsExperience: 6+ yearsSkills: ["Cross-functional coordination","High ownership","Clear communication","Documentation mindset","Bias to action"]

Own the quality bar behind human evaluations so evaluation outputs are trusted, decision-grade, and reproducible at scale. Lead evaluation guidelines, rubrics, gold references, and QA procedures; arbitrate ambiguous judgments; and manage contract-attorney onboarding, calibration, and inter-rater reliability. Define failure-mode taxonomies, run vendor and rater QA against a defined bar, and ensure quality holds across jurisdictions and languages while building lightweight operational dashboards.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Harvey
Harvey
1 week ago

Senior Product Operations Manager, Evaluation Quality

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 4 hours agoStatus: Live

Job Summary

Own the quality bar behind human evaluations so evaluation outputs are trusted, decision-grade, and reproducible at scale. Lead evaluation guidelines, rubrics, gold references, and QA procedures; arbitrate ambiguous judgments; and manage contract-attorney onboarding, calibration, and inter-rater reliability. Define failure-mode taxonomies, run vendor and rater QA against a defined bar, and ensure quality holds across jurisdictions and languages while building lightweight operational dashboards.
Location: San Francisco
Workplace: Hybrid
Employment Type: Full time
Job Function: Business Operations
Seniority: Manager level

Key Responsibilities

  • •Own the quality bar for Harvey’s human evaluations by defining what “good” looks like and ensuring outputs are decision-grade for EPD.
  • •Author and maintain evaluation guidelines, instructions, benchmarks, and gold references, serving as the arbiter for ambiguous or disputed judgments.
  • •Standardize rubric and evaluation design into repeatable templates and a documented methodology, partnering with Applied Legal Research (ALR).
  • •Own contract-attorney quality: onboarding, calibration training, inter-rater reliability, and the feedback loop to keep judgment consistent over time.
  • •Define failure-mode/error taxonomy, run QA on vendor and contract-attorney deliverables, and ensure quality holds across jurisdictions and non-English geographies as coverage expands.

Pay and Benefits

Salary: USD 155,400 - 233,200 annually
Equity and Bonus:Equity

Key Requirements

  • •6+ years in product operations, research operations, evaluation/QA operations, or quality program management.
  • •Own quality inputs (guidelines, instructions, benchmarks, QA procedures) for complex, expert-driven or human-in-the-loop work.
  • •Onboard, train, and calibrate a distributed pool of expert raters/annotators/reviewers and run feedback loops to improve quality over time.
  • •Grounding in measurement concepts like calibration, inter-rater reliability, sampling, and rubric design to independently set up and own quality.
  • •Strong cross-functional coordination across Product, Engineering, Research, ALR, and data providers/vendors.
Experience:6+ yearsHuman evaluationsLegal techQuality assuranceResearch operationsProduct operations
Skills:Cross-functional coordinationHigh ownershipClear communicationDocumentation mindsetBias to action
Tech Stack:AI tool supportLLM-as-judgeOperational dashboards

Company Brief

Harvey
Harvey builds domain-specific generative AI for legal and professional services, automating contract analysis, due diligence, compliance, and litigation workflows for law firms and corporate legal teams.
Industry: LegalTech
Company Size: Large (251 to 1,000 employees)
Revenue: USD 100M to 250M
Growth: Scaleup
Valuation: Unicorn (USD 1B+)
Funding: Series E+
Headquarters: San Francisco, United States
Founded: 2022
Glassdoor
Glassdoor: 4.1
WebsiteLinkedInGlassdoor