Senior Software Engineer, Model Infrastructure

Harvey
San Francisco
Workplace: HybridFull timeUSD 193,400 - 290,000 annuallyFunction: Software EngineeringExperience: 4+ yearsSkills: ["Communication","Collaboration","Technical leadership"]

Build and lead Harvey’s Model Infrastructure platform that powers every AI request, focusing on reliability, low latency, observability, and operational excellence. Design the Unified Model Controller and Model Selector for intelligent, policy-based routing across multiple providers. Develop provisioning, capacity management, failover, traffic engineering, and dashboards/telemetry to monitor model health, token usage, costs, and performance while enabling rapid adoption of new frontier models.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Harvey
Harvey
2 days ago

Senior Software Engineer, Model Infrastructure

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 1 hour agoStatus: Live

Job Summary

Build and lead Harvey’s Model Infrastructure platform that powers every AI request, focusing on reliability, low latency, observability, and operational excellence. Design the Unified Model Controller and Model Selector for intelligent, policy-based routing across multiple providers. Develop provisioning, capacity management, failover, traffic engineering, and dashboards/telemetry to monitor model health, token usage, costs, and performance while enabling rapid adoption of new frontier models.
Location: San Francisco
Workplace: Hybrid
Employment Type: Full time
Job Function: Software Engineering
Seniority: Mid level

Key Responsibilities

  • •Lead the design and implementation of the Model Infrastructure platform for Harvey’s AI requests.
  • •Build systems to ensure high availability, low latency, and operational excellence for AI inference.
  • •Design and improve the Unified Model Controller (UMC) and Model Selector for policy-based model routing based on reliability, latency, quality, compliance, and cost.
  • •Develop provisioning, capacity management, failover, and traffic engineering across multiple AI providers.
  • •Improve observability with health dashboards, alerting, token usage analytics, cost reporting, and end-to-end telemetry.

Pay and Benefits

Salary: USD 193,400 - 290,000 annually
Equity and Bonus:Equity

Key Requirements

  • •4+ years of software engineering experience building large-scale distributed systems.
  • •Experience designing and operating highly available production services.
  • •Strong programming skills in Go, Java, Python, Rust, or C++.
  • •Deep understanding of distributed systems, cloud infrastructure, networking, and observability.
  • •Experience leading technical projects across multiple engineering teams.
Experience:4+ yearsDistributed systemsCloud infrastructureObservabilitySRE best practices
Skills:CommunicationCollaborationTechnical leadership
Tech Stack:GoJavaPythonRustC++OpenAIAnthropicAzure OpenAIFireworksBasetenKubernetesService meshKafkaSparkFlinkAirflowIcebergGPULLM servingMachine learning platforms

Company Brief

Harvey
Harvey builds domain-specific generative AI for legal and professional services, automating contract analysis, due diligence, compliance, and litigation workflows for law firms and corporate legal teams.
Industry: LegalTech
Company Size: Large (251 to 1,000 employees)
Revenue: USD 100M to 250M
Growth: Scaleup
Valuation: Unicorn (USD 1B+)
Funding: Series E+
Headquarters: San Francisco, United States
Founded: 2022
Glassdoor
Glassdoor: 4.1
WebsiteLinkedInGlassdoor