Dimension Data

1 week ago

Data Engineer (Python AND Kafka AND Hadoop OR HDFS OR Hive OR Snowflake AND apache AND iceberg)

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI

Last checked: 17 hours agoStatus: Live

Job Summary

Build and migrate end-to-end datastore pipelines from an on-prem DataLake to an AWS-hosted Lakehouse for a high-visibility program. Refactor extraction logic and scheduling, transfer datasets with data integrity, and convert legacy SQL/Spark consumption patterns for Snowflake and Iceberg. Own reconciliation and quality validation to ensure migrated assets match production usage, while partnering with data owners through handoff and sign-off across global teams.

Location: Anywhere

Workplace: Onsite

Employment Type: Full time

Job Function: Software Engineering

Seniority: Mid level

Key Responsibilities

•Refactor and migrate extraction logic and job scheduling from legacy frameworks to the new Lakehouse environment.
•Execute physical migration of datasets from on-prem DataLake to AWS Lakehouse while ensuring data integrity.
•Convert and optimize legacy SQL and Spark-based consumption patterns for compatibility with Snowflake and Iceberg.
•Perform data reconciliation and quality validation using reconciliation frameworks to ensure functional equivalence to production.
•Act as a technical liaison to data owners, facilitating handoff and sign-off to confirm migrated assets meet business requirements.

Key Requirements

•3-5 years of hands-on coding experience in a collaborative environment.
•Professional proficiency in Python or Java, with troubleshooting experience in SQL and basic scripting.
•Deep familiarity with SDLC and CI/CD best practices, including Kubernetes (K8s) deployment experience.
•Expertise in data correctness for reconciliation, including temporal data modeling (e.g., SCD Type 2) and schema evolution with Iceberg/Apache.
•Advanced knowledge of performance optimization (partitioning/clustering) and architectural tradeoffs (normalization vs denormalization; natural vs surrogate keys).

Experience:3-5 years

Education:Bachelor's in Computer Science, Applied Mathematics, Engineering, or a related quantitative field

Skills:IntegrityCollaborationClear communicationStakeholder managementOwnership

Tech Stack:PythonJavaKafkaSQLFTPApache SparkJSONAvroParquetHadoopHDFSHiveSnowflakeApache IcebergSybase IQCI/CDSDLCKubernetesK8s

Company Brief

Dimension Data

Dimension Data, operating under NTT Ltd, provides managed IT services, cloud and data center solutions, networking, cybersecurity, and digital transformation services to enterprise customers worldwide.

Industry: Professional Services

Company Size: Enterprise (1,001+ employees)

Revenue: USD 1B+

Growth: Established Company

Valuation: Public Company (Market Cap in USD)

Headquarters: London, United Kingdom

Founded: 1983

Website LinkedIn

Loading job details...

Preparing the role view and application actions.

Dimension Data

1 week ago

Data Engineer (Python AND Kafka AND Hadoop OR HDFS OR Hive OR Snowflake AND apache AND iceberg)

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI

Last checked: 17 hours agoStatus: Live

Job Summary

Location: Anywhere

Workplace: Onsite

Employment Type: Full time

Job Function: Software Engineering

Seniority: Mid level

Key Responsibilities

•Refactor and migrate extraction logic and job scheduling from legacy frameworks to the new Lakehouse environment.
•Execute physical migration of datasets from on-prem DataLake to AWS Lakehouse while ensuring data integrity.
•Convert and optimize legacy SQL and Spark-based consumption patterns for compatibility with Snowflake and Iceberg.
•Perform data reconciliation and quality validation using reconciliation frameworks to ensure functional equivalence to production.
•Act as a technical liaison to data owners, facilitating handoff and sign-off to confirm migrated assets meet business requirements.

Key Requirements

•3-5 years of hands-on coding experience in a collaborative environment.
•Professional proficiency in Python or Java, with troubleshooting experience in SQL and basic scripting.
•Deep familiarity with SDLC and CI/CD best practices, including Kubernetes (K8s) deployment experience.
•Expertise in data correctness for reconciliation, including temporal data modeling (e.g., SCD Type 2) and schema evolution with Iceberg/Apache.
•Advanced knowledge of performance optimization (partitioning/clustering) and architectural tradeoffs (normalization vs denormalization; natural vs surrogate keys).

Experience:3-5 years

Education:Bachelor's in Computer Science, Applied Mathematics, Engineering, or a related quantitative field

Skills:IntegrityCollaborationClear communicationStakeholder managementOwnership

Tech Stack:PythonJavaKafkaSQLFTPApache SparkJSONAvroParquetHadoopHDFSHiveSnowflakeApache IcebergSybase IQCI/CDSDLCKubernetesK8s

Company Brief

Dimension Data

Industry: Professional Services

Company Size: Enterprise (1,001+ employees)

Revenue: USD 1B+

Growth: Established Company

Valuation: Public Company (Market Cap in USD)

Headquarters: London, United Kingdom

Founded: 1983

Website LinkedIn