NTT

1 month ago

Data Engineer Sr (Databricks)

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI

Last checked: 12 hours agoStatus: Live

Job Summary

Design and build Databricks notebooks, jobs, and workflows to migrate and enhance DB2/Guidewire data pipelines. Create Delta Lake tables using medallion architecture and support ACID, time travel, schema evolution, incremental loads, and CDC. Integrate Databricks with AWS/S3 or Azure ADLS, ADF/Synapse, Key Vault, and Snowflake, optimizing performance and cost. Collaborate with Snowflake and dbt teams on data models, validation, and post go-live stabilization.

Location: Bengaluru

Workplace: Onsite

Employment Type: Full time

Job Function: Data Analytics & Business Intelligence

Seniority: Mid level

Key Responsibilities

•Develop Databricks notebooks, jobs, and workflows to replicate and enhance DB2/Guidewire pipelines and transformations.
•Implement Delta Lake medallion architecture (bronze/silver/gold) and patterns including ACID, time travel, schema evolution, incremental loads, and CDC.
•Integrate Databricks with AWS/S3 or Azure ADLS, ADF/Synapse, Key Vault, and Snowflake.
•Optimize Databricks clusters, jobs, and queries for performance and cost.
•Collaborate on consistent data models/data contracts with Snowflake and dbt teams; perform data validation and reconciliation and provide post go-live support.

Key Requirements

•8+ years of Databricks experience with legacy DB2/400 and Guidewire data ingestion into cloud storage.
•Experience with Change Data Capture (CDC) or scheduled batch extraction from DB2 via JDBC, including re-platforming legacy SQL for distributed computing.
•Build ETL/ELT pipelines using medallion architecture (bronze/silver/gold) with Delta Lake features like ACID transactions, time travel, schema evolution, and upserts/SCD Type 2.
•Integrate Databricks with AWS/S3 or Azure ADLS, ADF/Synapse, Key Vault, and Snowflake as needed.
•Implement data governance and quality using Unity Catalog and automated validation/reconciliation between legacy sources and Databricks outputs.

Experience:Data engineeringDatabricksDelta lakeCloud data pipelinesLegacy migrationInsuranceETL/ELT

Skills:CollaborationPerformance optimizationData validationVersion controlCI/CD

Tech Stack:DatabricksDelta LakeApache SparkPySparkSpark SQLScalaAWSS3AzureAzure ADLSAzure ADFAzure SynapseKey VaultSnowflakeDbtJDBCGitAzure DevOpsUnity CatalogDatabricks Auto Loader

Company Brief

NTT

Dimension Data, operating under NTT Ltd, provides managed IT services, cloud and data center solutions, networking, cybersecurity, and digital transformation services to enterprise customers worldwide.

Industry: Professional Services

Company Size: Enterprise (1,001+ employees)

Revenue: USD 1B+

Growth: Established Company

Valuation: Public Company (Market Cap in USD)

Headquarters: London, United Kingdom

Founded: 1983

Website LinkedIn

Loading job details...

Preparing the role view and application actions.

NTT