Staff Software Engineer, Lakeflow Pipelines DR

Databricks
Mountain View, San Francisco
Workplace: OnsiteFull timeUSD 192,000 - 260,000 annuallyFunction: Software EngineeringExperience: 8+ yearsSkills: ["Multi-year planning","Production-quality delivery","Technical vision"]

Design and implement distributed disaster recovery for Lakeflow pipelines, replicating and recovering streaming and batch ETL across regions. Tackle correctness-critical challenges like consistency, idempotency, causal ordering, failover/failback, and safe recovery with no silent data loss or duplication. Work on cross-region replication, distributed consistency across dependencies and logs, observability and failure-injection testing, and high-fidelity recovery simulations for resilient Lakehouse computing.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Databricks
Databricks
14 hours ago

Staff Software Engineer, Lakeflow Pipelines DR

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 1 hour agoStatus: Live

Job Summary

Design and implement distributed disaster recovery for Lakeflow pipelines, replicating and recovering streaming and batch ETL across regions. Tackle correctness-critical challenges like consistency, idempotency, causal ordering, failover/failback, and safe recovery with no silent data loss or duplication. Work on cross-region replication, distributed consistency across dependencies and logs, observability and failure-injection testing, and high-fidelity recovery simulations for resilient Lakehouse computing.
Location: Mountain View, San Francisco
Workplace: Onsite
Employment Type: Full time
Job Function: Software Engineering
Seniority: Sr. Manager level

Key Responsibilities

  • •Design and implement distributed systems that replicate and recover pipelines across regions for Lakeflow disaster recovery.
  • •Handle challenging recovery correctness problems including consistency, idempotency, causal ordering, failover, failback, and safe recovery without data loss or duplication.
  • •Lead cross-region replication and recovery for Lakeflow pipelines, streaming tables, and materialized views.
  • •Build distributed consistency across pipeline dependencies, table versions, and transaction logs.
  • •Create observability, failure-injection testing, and high-fidelity recovery simulations (game-day testing and reasoning about failure modes).

Pay and Benefits

Salary: USD 192,000 - 260,000 annually
Equity and Bonus:Equity

Key Requirements

  • •Strong software engineering skills in Java, Scala, C++, Go, Python, or a similar production language.
  • •Understanding of consistency, transactions, idempotency, replication, checkpointing, and data lineage.
  • •A passion for distributed systems, databases, storage systems, streaming systems, or reliability engineering.
  • •Ability to define and work toward a multi-year technical vision with incremental, production-quality deliverables.
  • •8+ years of experience working on related systems (preferred).
Experience:8+ yearsDistributed systemsDatabasesStreaming systemsReliability engineering
Skills:Multi-year planningProduction-quality deliveryTechnical vision
Languages:En
Tech Stack:JavaScalaC++GoPython

Company Brief

Databricks
Provides a unified data analytics platform powered by Apache Spark to simplify building, deploying, and scaling data engineering, data science, and machine learning workloads for enterprises.
Industry: Data Infrastructure
Company Size: Enterprise (1,001+ employees)
Revenue: USD 1B+
Growth: Scaleup
Valuation: Unicorn (USD 1B+)
Funding: Series E+
Headquarters: San Francisco, United States
Founded: 2013
WebsiteLinkedIn