Digital Ocean

1 day ago

Senior Engineer, Inference Data Plane

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI

Last checked: 21 hours agoStatus: Live

Job Summary

Design and deliver high-scale, resilient data plane services for “Inference as a Service,” partnering with Product and engineering teams to align technical roadmaps with customer needs. Lead system design and performance optimization for distributed LLM inference across multi-tenant environments, including tensor/data parallelism, KV-cache routing, autoscaling, and Kubernetes-native serving frameworks. Contribute upstream to inference open source projects and mentor junior engineers while maintaining SLO-driven operational excellence.

Location: Denver

Workplace: Remote

Employment Type: Full time

Job Function: Solutions Engineering & Sales Engineering

Seniority: Mid level

Key Responsibilities

•Act as a technical leader driving end-to-end design, development, and delivery of AI inference data plane components.
•Architect and refine system designs for a high-scale, multi-tenant AI inference cloud ecosystem with availability and resiliency requirements.
•Implement and optimize distributed inference hosting, including tensor/data parallelism, KV-cache optimizations, and smart routing.
•Build and scale Kubernetes-native distributed inference frameworks for prefill/decode disaggregation, KV-cache-aware routing, prefix caching, and MoE expert parallelism.
•Maintain critical high-scale services using observability tools and SLOs, and contribute upstream to inference projects while mentoring junior engineers.

Pay and Benefits

Salary: USD 139,200 - 174,000 annually

Equity and Bonus:Equity

Perks:EquityRemote WorkLearning BudgetPaid LeaveEmployee Assistance

Key Requirements

•Hands-on experience hosting large language or multimodal models using inference engines such as vLLM, SGLang, or TensorRT.
•Strong familiarity with distributed inference serving frameworks including llm-d, NVIDIA Dynamo, or Ray Serve.
•Experience with inference engine internals (e.g., continuous batching, paged attention, prefix caching) using vLLM or alternatives like SGLang or TensorRT-LLM.
•Understanding of distributed LLM serving challenges such as KV-cache locality, routing to preserve cache hit rates and tail latency, and disaggregated prefill/decode with fast KV transfer.
•Expert proficiency in GoLang or Python, with familiarity with gRPC, plus demonstrated experience running critical high-scale services and shipping customer-facing products.

Experience:AI/MLLLM inferenceDistributed systemsKubernetes

Skills:Technical leadershipCollaborationMentorshipSystem designPerformance optimization

Tech Stack:GoLangPythonGRPCKubernetesLlm-dNVIDIA DynamoRay ServeKServeVLLMSGLangTensorRTTensorRT-LLMTGIModular MAXNIXL

Company Brief

Digital Ocean

Provides cloud infrastructure and developer-focused cloud services including scalable droplets, managed databases, Kubernetes, object storage, and networking to simplify deploying and managing applications for developers and small-to-medium businesses.

Industry: Cloud Computing

Company Size: Enterprise (1,001+ employees)

Revenue: USD 250M to 500M

Growth: Public Company

Valuation: Public Company (Market Cap in USD)

Funding: IPO / Publicly Listed

Headquarters: New York, United States

Founded: 2011

Glassdoor: 3.8

Website LinkedIn

Loading job details...

Preparing the role view and application actions.

Digital Ocean

1 day ago

Senior Engineer, Inference Data Plane

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI

Last checked: 21 hours agoStatus: Live

Job Summary

Location: Denver

Workplace: Remote

Employment Type: Full time

Job Function: Solutions Engineering & Sales Engineering

Seniority: Mid level

Key Responsibilities

•Act as a technical leader driving end-to-end design, development, and delivery of AI inference data plane components.
•Architect and refine system designs for a high-scale, multi-tenant AI inference cloud ecosystem with availability and resiliency requirements.
•Implement and optimize distributed inference hosting, including tensor/data parallelism, KV-cache optimizations, and smart routing.
•Build and scale Kubernetes-native distributed inference frameworks for prefill/decode disaggregation, KV-cache-aware routing, prefix caching, and MoE expert parallelism.
•Maintain critical high-scale services using observability tools and SLOs, and contribute upstream to inference projects while mentoring junior engineers.

Pay and Benefits

Salary: USD 139,200 - 174,000 annually

Equity and Bonus:Equity

Perks:EquityRemote WorkLearning BudgetPaid LeaveEmployee Assistance

Key Requirements

•Hands-on experience hosting large language or multimodal models using inference engines such as vLLM, SGLang, or TensorRT.
•Strong familiarity with distributed inference serving frameworks including llm-d, NVIDIA Dynamo, or Ray Serve.
•Experience with inference engine internals (e.g., continuous batching, paged attention, prefix caching) using vLLM or alternatives like SGLang or TensorRT-LLM.
•Understanding of distributed LLM serving challenges such as KV-cache locality, routing to preserve cache hit rates and tail latency, and disaggregated prefill/decode with fast KV transfer.
•Expert proficiency in GoLang or Python, with familiarity with gRPC, plus demonstrated experience running critical high-scale services and shipping customer-facing products.

Experience:AI/MLLLM inferenceDistributed systemsKubernetes

Skills:Technical leadershipCollaborationMentorshipSystem designPerformance optimization

Tech Stack:GoLangPythonGRPCKubernetesLlm-dNVIDIA DynamoRay ServeKServeVLLMSGLangTensorRTTensorRT-LLMTGIModular MAXNIXL

Company Brief

Digital Ocean

Industry: Cloud Computing

Company Size: Enterprise (1,001+ employees)

Revenue: USD 250M to 500M

Growth: Public Company

Valuation: Public Company (Market Cap in USD)

Funding: IPO / Publicly Listed

Headquarters: New York, United States

Founded: 2011

Glassdoor: 3.8

Website LinkedIn