Dimension Data

1 week ago

Data Engineer (Python AND Kafka AND Hadoop OR HDFS OR Hive AND Snowflake AND apache AND iceberge)

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI

Last checked: 17 hours agoStatus: Live

Job Summary

Join the datastore-migration Factory team to deliver end-to-end migration of on-prem DataLake to an AWS Lakehouse for a high-visibility project. Refactor extraction logic and scheduling from legacy frameworks, execute physical dataset migration with data integrity, and convert legacy SQL and Spark consumption patterns for Snowflake and Iceberg. Drive reconciliation and quality validation using migration frameworks, collaborate with data owners on handoff/sign-off, and analyze usage to deliver required data products.

Location: Anywhere

Workplace: Onsite

Employment Type: Full time

Job Function: Software Engineering

Seniority: Mid level

Key Responsibilities

•Perform end-to-end datastore migration from on-prem DataLake to an AWS hosted Lakehouse, ensuring data integrity.
•Refactor and migrate extraction logic and job scheduling from legacy frameworks to the Lakehouse environment.
•Translate and optimize legacy SQL and Apache Spark consumption patterns for compatibility with Snowflake and Iceberg.
•Conduct reconciliation and data validation using reconciliation frameworks to confirm migrated data matches production flows.
•Act as a technical liaison to internal clients, facilitating data-owner handoff and sign-off conversations and delivering required data products based on usage analysis.

Key Requirements

•3-5 years of hands-on coding experience in a collaborative, team-based environment.
•Proficiency troubleshooting SQL and basic scripting experience, with professional proficiency in Python or Java.
•Deep familiarity with SDLC and CI/CD best practices, including K8s deployment experience.
•Strong data engineering competency in temporal data modeling (e.g., SCD Type 2) and schema evolution with Iceberg/Apache.
•Advanced knowledge of data partitioning/clustering and architectural tradeoffs (normalization vs denormalization; natural vs surrogate keys).

Experience:3-5 years

Education:Bachelor's in Computer Science, Applied Mathematics, Engineering, or a related quantitative field

Skills:IntegrityEthical decision-makingCollaborationClear communicationStakeholder management

Tech Stack:PythonJavaKafkaSQLFTPApache SparkJSONAvroParquetHadoopHDFSHiveSnowflakeApache IcebergSybase IQCI/CDKubernetes

Company Brief

Dimension Data

Dimension Data, operating under NTT Ltd, provides managed IT services, cloud and data center solutions, networking, cybersecurity, and digital transformation services to enterprise customers worldwide.

Industry: Professional Services

Company Size: Enterprise (1,001+ employees)

Revenue: USD 1B+

Growth: Established Company

Valuation: Public Company (Market Cap in USD)

Headquarters: London, United Kingdom

Founded: 1983

Website LinkedIn

Loading job details...

Preparing the role view and application actions.

Dimension Data

1 week ago

Data Engineer (Python AND Kafka AND Hadoop OR HDFS OR Hive AND Snowflake AND apache AND iceberge)

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI

Last checked: 17 hours agoStatus: Live

Job Summary

Location: Anywhere

Workplace: Onsite

Employment Type: Full time

Job Function: Software Engineering

Seniority: Mid level

Key Responsibilities

•Perform end-to-end datastore migration from on-prem DataLake to an AWS hosted Lakehouse, ensuring data integrity.
•Refactor and migrate extraction logic and job scheduling from legacy frameworks to the Lakehouse environment.
•Translate and optimize legacy SQL and Apache Spark consumption patterns for compatibility with Snowflake and Iceberg.
•Conduct reconciliation and data validation using reconciliation frameworks to confirm migrated data matches production flows.
•Act as a technical liaison to internal clients, facilitating data-owner handoff and sign-off conversations and delivering required data products based on usage analysis.

Key Requirements

•3-5 years of hands-on coding experience in a collaborative, team-based environment.
•Proficiency troubleshooting SQL and basic scripting experience, with professional proficiency in Python or Java.
•Deep familiarity with SDLC and CI/CD best practices, including K8s deployment experience.
•Strong data engineering competency in temporal data modeling (e.g., SCD Type 2) and schema evolution with Iceberg/Apache.
•Advanced knowledge of data partitioning/clustering and architectural tradeoffs (normalization vs denormalization; natural vs surrogate keys).

Experience:3-5 years

Education:Bachelor's in Computer Science, Applied Mathematics, Engineering, or a related quantitative field

Skills:IntegrityEthical decision-makingCollaborationClear communicationStakeholder management

Tech Stack:PythonJavaKafkaSQLFTPApache SparkJSONAvroParquetHadoopHDFSHiveSnowflakeApache IcebergSybase IQCI/CDKubernetes

Company Brief

Dimension Data

Industry: Professional Services

Company Size: Enterprise (1,001+ employees)

Revenue: USD 1B+

Growth: Established Company

Valuation: Public Company (Market Cap in USD)

Headquarters: London, United Kingdom

Founded: 1983

Website LinkedIn