Senior Software Engineer, Spark Platform

DoorDash US
San Francisco, Sunnyvale, Seattle, New York
Workplace: HybridFull timeUSD 130,600 - 285,000 annuallyFunction: Software EngineeringExperience: 6+ yearsEducation: bachelorsSkills: ["Architectural leadership","Mentorship","Distributed systems problem-solving","Technical roadmap partnering","Design reviews"]

Lead the technical direction for an in-house Spark deployment that powers large-scale data, analytics, and ML workloads. Own cross-cutting distributed systems challenges across runtime, remote shuffle, scheduling, and service reliability, and make architectural decisions that compound over the platform’s lifetime. Partner with Engineering Management and collaborate with Data Engineering, ML Platform, and product teams to ensure workloads land effectively on the platform while raising team standards through design reviews and mentorship.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
DoorDash US
DoorDash US
2 months ago

Senior Software Engineer, Spark Platform

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 3 hours agoStatus: Live

Job Summary

Lead the technical direction for an in-house Spark deployment that powers large-scale data, analytics, and ML workloads. Own cross-cutting distributed systems challenges across runtime, remote shuffle, scheduling, and service reliability, and make architectural decisions that compound over the platform’s lifetime. Partner with Engineering Management and collaborate with Data Engineering, ML Platform, and product teams to ensure workloads land effectively on the platform while raising team standards through design reviews and mentorship.
Location: San Francisco, Sunnyvale, Seattle, New York
Workplace: Hybrid
Employment Type: Full time
Job Function: Software Engineering

Key Responsibilities

  • •Set multi-year technical direction for an in-house Spark-on-Kubernetes platform, including runtime, shuffle, scheduling, and reliability.
  • •Own distributed-systems problems such as shuffle architecture, multi-tenant scheduling, runtime performance, and large-scale failure modes.
  • •Partner with the Engineering Manager on technical roadmap, hiring, interview design, and team shape.
  • •Uplevel the team through design reviews, mentorship, and raising the bar for what they ship.
  • •Represent Spark Platform in cross-team architecture forums and shape how data, analytics, and ML workloads land on the platform.

Pay and Benefits

Salary: USD 130,600 - 285,000 annually
Equity and Bonus:Equity
Perks:401kPaid ParentalWellness StipendCommuter BenefitsPaid LeaveHealth InsuranceDentalVisionPaid HolidaysDisability InsuranceLife Insurance

Key Requirements

  • •B.S., M.S., or PhD in Computer Science (or equivalent).
  • •6+ years designing and operating distributed systems at scale.
  • •Deep hands-on experience with Apache Spark internals, query execution, shuffle, and executor/driver model at platform scale.
  • •Production experience with remote/external shuffle systems, batch/big-data schedulers, and/or observability/SRE patterns for distributed compute platforms.
  • •Strong fluency operating workloads on Kubernetes in production with knowledge of operator patterns and multi-tenant failure modes.
Experience:6+ years
Education:Bachelor's in Computer Science
Skills:Architectural leadershipMentorshipDistributed systems problem-solvingTechnical roadmap partneringDesign reviews
Tech Stack:Apache SparkSpark-on-KubernetesKubernetesAmazon EMRDatabricksScalaJavaPythonGoSQLCelebornMagnetCoscoYuniKornVolcanoKueueApache IcebergDelta Lake

Company Brief

DoorDash US
On-demand logistics platform connecting consumers with local restaurants, grocers, and retailers for food delivery, pickup, and convenience services. Operates a marketplace and last-mile delivery network while offering tools and analytics for merchants and couriers.
Industry: Online Marketplaces
Company Size: Enterprise (1,001+ employees)
Revenue: USD 1B+
Growth: Public Company
Valuation: Public Company (Market Cap in USD)
Funding: IPO / Publicly Listed
Headquarters: San Francisco, United States
Founded: 2013
WebsiteLinkedIn