Software Engineer, Spark Platform

DoorDash US
San Francisco, Seattle, Sunnyvale, New York
Workplace: HybridFull timeUSD 130,600 - 285,000 annuallyFunction: Software EngineeringExperience: 24+ yearsEducation: bachelorsSkills: ["Ownership","Collaboration","Problem-solving","Measurement","Reducing toil"]

Build and operate DoorDash’s in-house Apache Spark platform at company-wide scale, including Spark runtime upgrades, performance, and multi-tenant scheduling. You’ll work on Kubernetes-based executor bin-packing, cluster lifecycle automation (provisioning, upgrades, capacity changes, node-failure handling), and end-to-end observability and incident automation to keep on-call sustainable. Collaborate with engineers and Spark consumers across the company to tackle high-leverage platform problems.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
DoorDash US
DoorDash US
2 months ago

Software Engineer, Spark Platform

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 3 hours agoStatus: Live

Job Summary

Build and operate DoorDash’s in-house Apache Spark platform at company-wide scale, including Spark runtime upgrades, performance, and multi-tenant scheduling. You’ll work on Kubernetes-based executor bin-packing, cluster lifecycle automation (provisioning, upgrades, capacity changes, node-failure handling), and end-to-end observability and incident automation to keep on-call sustainable. Collaborate with engineers and Spark consumers across the company to tackle high-leverage platform problems.
Location: San Francisco, Seattle, Sunnyvale, New York
Workplace: Hybrid
Employment Type: Full time
Job Function: Software Engineering

Key Responsibilities

  • •Build and operate an in-house Spark platform spanning runtime, scheduler, reliability, and user-facing tooling.
  • •Drive multi-tenant scheduling, executor bin-packing, and cost-aware placement to serve multiple consumer teams.
  • •Own cluster lifecycle automation including provisioning, runtime upgrades, capacity changes, and node-failure handling.
  • •Build observability and incident automation to make the platform debuggable end-to-end and keep on-call sustainable.
  • •Collaborate with engineers on shuffle, runtime, and architecture work, gradually taking deeper ownership areas over time.

Pay and Benefits

Salary: USD 130,600 - 285,000 annually
Equity and Bonus:Equity
Perks:401kParental LeaveHealth InsuranceDentalVisionPaid LeaveWellness StipendCommuter BenefitsDisability InsuranceLife InsurancePaid Holidays

Key Requirements

  • •B.S., M.S., or PhD in Computer Science (or equivalent).
  • •24+ years of industry experience operating production distributed systems.
  • •Experience operating Apache Spark at scale (Amazon EMR, Databricks, or in-house), focused on platform operations like runtime upgrades, cluster lifecycle, shuffle, observability, and multi-tenant scheduling.
  • •Hands-on experience operating production systems on Kubernetes, including controllers, operators, custom resources, and multi-tenant failure modes.
  • •Familiarity with batch/big-data schedulers (YuniKorn, Volcano, Kueue or equivalent) and observability stacks (Prometheus, OpenTelemetry, distributed tracing, structured logging).
Experience:24+ yearsDistributed systemsBig dataMachine learningCloudKubernetes
Education:Bachelor's
Skills:OwnershipCollaborationProblem-solvingMeasurementReducing toil
Languages:English
Tech Stack:Apache SparkAmazon EMRDatabricksKubernetesRemote ShuffleYuniKornVolcanoKueueSpark-on-Kubernetes operatorPrometheusOpenTelemetryDistributed tracingStructured loggingAWSVPC networkingInstance lifecycleSpot/preemptibleAutoscalingPythonGo

Company Brief

DoorDash US
On-demand logistics platform connecting consumers with local restaurants, grocers, and retailers for food delivery, pickup, and convenience services. Operates a marketplace and last-mile delivery network while offering tools and analytics for merchants and couriers.
Industry: Online Marketplaces
Company Size: Enterprise (1,001+ employees)
Revenue: USD 1B+
Growth: Public Company
Valuation: Public Company (Market Cap in USD)
Funding: IPO / Publicly Listed
Headquarters: San Francisco, United States
Founded: 2013
WebsiteLinkedIn