Data Engineer Sr (Databricks)

NTT
Bengaluru
Workplace: OnsiteFull timeFunction: Data Analytics & Business IntelligenceSkills: ["Collaboration","Performance optimization","Data validation","Version control","CI/CD"]

Design and build Databricks notebooks, jobs, and workflows to migrate and enhance DB2/Guidewire data pipelines. Create Delta Lake tables using medallion architecture and support ACID, time travel, schema evolution, incremental loads, and CDC. Integrate Databricks with AWS/S3 or Azure ADLS, ADF/Synapse, Key Vault, and Snowflake, optimizing performance and cost. Collaborate with Snowflake and dbt teams on data models, validation, and post go-live stabilization.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
NTT
NTT
3 months ago

Data Engineer Sr (Databricks)

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 20 hours agoStatus: Live

Job Summary

Design and build Databricks notebooks, jobs, and workflows to migrate and enhance DB2/Guidewire data pipelines. Create Delta Lake tables using medallion architecture and support ACID, time travel, schema evolution, incremental loads, and CDC. Integrate Databricks with AWS/S3 or Azure ADLS, ADF/Synapse, Key Vault, and Snowflake, optimizing performance and cost. Collaborate with Snowflake and dbt teams on data models, validation, and post go-live stabilization.
Location: Bengaluru
Workplace: Onsite
Employment Type: Full time
Job Function: Data Analytics & Business Intelligence
Seniority: Mid level

Key Responsibilities

  • •Develop Databricks notebooks, jobs, and workflows to replicate and enhance DB2/Guidewire pipelines and transformations.
  • •Implement Delta Lake medallion architecture (bronze/silver/gold) and patterns including ACID, time travel, schema evolution, incremental loads, and CDC.
  • •Integrate Databricks with AWS/S3 or Azure ADLS, ADF/Synapse, Key Vault, and Snowflake.
  • •Optimize Databricks clusters, jobs, and queries for performance and cost.
  • •Collaborate on consistent data models/data contracts with Snowflake and dbt teams; perform data validation and reconciliation and provide post go-live support.

Key Requirements

  • •8+ years of Databricks experience with legacy DB2/400 and Guidewire data ingestion into cloud storage.
  • •Experience with Change Data Capture (CDC) or scheduled batch extraction from DB2 via JDBC, including re-platforming legacy SQL for distributed computing.
  • •Build ETL/ELT pipelines using medallion architecture (bronze/silver/gold) with Delta Lake features like ACID transactions, time travel, schema evolution, and upserts/SCD Type 2.
  • •Integrate Databricks with AWS/S3 or Azure ADLS, ADF/Synapse, Key Vault, and Snowflake as needed.
  • •Implement data governance and quality using Unity Catalog and automated validation/reconciliation between legacy sources and Databricks outputs.
Experience:Data engineeringDatabricksDelta lakeCloud data pipelinesLegacy migrationInsuranceETL/ELT
Skills:CollaborationPerformance optimizationData validationVersion controlCI/CD
Tech Stack:DatabricksDelta LakeApache SparkPySparkSpark SQLScalaAWSS3AzureAzure ADLSAzure ADFAzure SynapseKey VaultSnowflakeDbtJDBCGitAzure DevOpsUnity CatalogDatabricks Auto Loader

Company Brief

NTT
Dimension Data, operating under NTT Ltd, provides managed IT services, cloud and data center solutions, networking, cybersecurity, and digital transformation services to enterprise customers worldwide.
Industry: Professional Services
Company Size: Enterprise (1,001+ employees)
Revenue: USD 1B+
Growth: Established Company
Valuation: Public Company (Market Cap in USD)
Headquarters: London, United Kingdom
Founded: 1983
WebsiteLinkedIn