Staff Software Engineer, Model Infrastructure

Harvey
San Francisco
Workplace: OnsiteFull timeUSD 236,000 - 290,000 annuallyFunction: Software EngineeringExperience: 7+ yearsSkills: ["Communication","Collaboration","Mentoring","Cross-functional leadership","Pragmatic execution"]

Lead the design and development of a Model Infrastructure platform powering every AI request. Build highly available, low-latency systems for AI inference, including Harvey’s Unified Model Controller (UMC) and Model Selector for model health detection and intelligent routing. Develop provisioning, capacity management, failover, and traffic engineering across multiple model providers while improving observability, cost visibility, and operational excellence. Mentor engineers and partner with AI Research and Product Engineering on production AI workloads.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Harvey
Harvey
1 month ago

Staff Software Engineer, Model Infrastructure

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 11 hours agoStatus: Live

Job Summary

Lead the design and development of a Model Infrastructure platform powering every AI request. Build highly available, low-latency systems for AI inference, including Harvey’s Unified Model Controller (UMC) and Model Selector for model health detection and intelligent routing. Develop provisioning, capacity management, failover, and traffic engineering across multiple model providers while improving observability, cost visibility, and operational excellence. Mentor engineers and partner with AI Research and Product Engineering on production AI workloads.
Location: San Francisco
Workplace: Onsite
Employment Type: Full time
Job Function: Software Engineering
Seniority: Mid level

Key Responsibilities

  • •Lead the design and implementation of Harvey’s Model Infrastructure platform powering AI requests.
  • •Build systems for high availability, low latency, and operational excellence for AI inference.
  • •Design and improve the Unified Model Controller (UMC) and Model Selector for detecting model degradations and routing traffic based on reliability, latency, quality, compliance, and cost.
  • •Develop model provisioning, capacity management, failover, and traffic engineering across multiple AI providers, including maintaining provider APIs and SDKs.
  • •Improve observability with health dashboards, alerting, token usage analytics, cost reporting, and end-to-end telemetry; mentor engineers and partner with AI Research and Product Engineering.

Pay and Benefits

Salary: USD 236,000 - 290,000 annually
Equity and Bonus:Equity
Perks:Annual BonusEquity

Key Requirements

  • •7+ years of software engineering experience building large-scale distributed systems.
  • •Experience designing and operating highly available production services.
  • •Strong programming skills in Go, Java, Python, Rust, or C++.
  • •Deep understanding of distributed systems, cloud infrastructure, networking, and observability.
  • •Experience leading technical projects across multiple engineering teams.
Experience:7+ years
Skills:CommunicationCollaborationMentoringCross-functional leadershipPragmatic execution
Tech Stack:GoJavaPythonRustC++OpenAIAnthropicAzure OpenAIFireworksBasetenKubernetesService meshKafkaSparkFlinkAirflowIceberg

Company Brief

Harvey
Harvey builds domain-specific generative AI for legal and professional services, automating contract analysis, due diligence, compliance, and litigation workflows for law firms and corporate legal teams.
Industry: LegalTech
Company Size: Large (251 to 1,000 employees)
Revenue: USD 100M to 250M
Growth: Scaleup
Valuation: Unicorn (USD 1B+)
Funding: Series E+
Headquarters: San Francisco, United States
Founded: 2022
Glassdoor
Glassdoor: 4.1
WebsiteLinkedInGlassdoor