Staff Software Engineer, Model Infrastructure

Harvey
San Francisco
Workplace: HybridFull timeUSD 231,000 - 340,000 annuallyFunction: Software EngineeringExperience: 7+ yearsSkills: ["Technical leadership","Communication","Collaboration","Ownership","Pragmatic execution"]

Lead the design and development of the systems powering every AI request, focusing on a highly reliable, scalable, observable, and efficient model infrastructure platform. Build model routing and health monitoring via the Unified Model Controller and Model Selector, deliver high-availability inference operations with failover and traffic engineering, and integrate new model providers. Partner with AI Research and Product Engineering to support launches, experimentation, and proactive monitoring while improving capacity, utilization, and cost visibility.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Harvey
Harvey
2 days ago

Staff Software Engineer, Model Infrastructure

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 1 hour agoStatus: Live
Reposted: similar role first listed 1 month ago

Job Summary

Lead the design and development of the systems powering every AI request, focusing on a highly reliable, scalable, observable, and efficient model infrastructure platform. Build model routing and health monitoring via the Unified Model Controller and Model Selector, deliver high-availability inference operations with failover and traffic engineering, and integrate new model providers. Partner with AI Research and Product Engineering to support launches, experimentation, and proactive monitoring while improving capacity, utilization, and cost visibility.
Location: San Francisco
Workplace: Hybrid
Employment Type: Full time
Job Function: Software Engineering
Seniority: Mid level

Key Responsibilities

  • •Lead design and implementation of the Model Infrastructure platform.
  • •Build systems for high availability, low latency, and operational excellence for AI inference.
  • •Design and improve the Unified Model Controller (UMC) and Model Selector for detecting degradations and routing traffic based on reliability, latency, quality, compliance, and cost.
  • •Develop systems for model provisioning, capacity management, failover, and traffic engineering across multiple AI providers.
  • •Improve observability with health dashboards, alerting, token usage analytics, cost reporting, and end-to-end telemetry; support model launches and proactively monitor production AI workloads.

Pay and Benefits

Salary: USD 231,000 - 340,000 annually

Key Requirements

  • •7+ years of software engineering experience building large-scale distributed systems.
  • •Experience designing and operating highly available production services.
  • •Strong programming skills in Go, Java, Python, Rust, or C++.
  • •Deep understanding of distributed systems, cloud infrastructure, networking, and observability.
  • •Ability to balance long-term architecture with pragmatic execution and lead technical projects across teams.
Experience:7+ yearsDistributed systems
Skills:Technical leadershipCommunicationCollaborationOwnershipPragmatic execution
Tech Stack:GoJavaPythonRustC++OpenAIAnthropicAzure OpenAIFireworksBasetenKubernetesService meshKafkaSparkFlinkAirflowIcebergGPU infrastructureLLM servingMachine learning platforms

Company Brief

Harvey
Harvey builds domain-specific generative AI for legal and professional services, automating contract analysis, due diligence, compliance, and litigation workflows for law firms and corporate legal teams.
Industry: LegalTech
Company Size: Large (251 to 1,000 employees)
Revenue: USD 100M to 250M
Growth: Scaleup
Valuation: Unicorn (USD 1B+)
Funding: Series E+
Headquarters: San Francisco, United States
Founded: 2022
Glassdoor
Glassdoor: 4.1
WebsiteLinkedInGlassdoor