NTT

4 weeks ago

AWS / Kafka / Airflow

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI

Last checked: 5 hours agoStatus: Live

Job Summary

Design and maintain ETL workflows using Apache Airflow, building directed acyclic graph (DAG) dependencies that respond to events from Apache Kafka. Provision and scale Kafka and Airflow on AWS using services such as Amazon MSK and MWAA (or self-hosting on EKS). Integrate Kafka with Airflow for event-driven triggering, and implement monitoring, logging, alerting, and performance optimization using AWS CloudWatch and Kafka tooling.

Location: Bengaluru

Workplace: Onsite

Employment Type: Full time

Job Function: DevOps, Cloud & Infrastructure

Key Responsibilities

•Design and maintain Apache Airflow DAGs for complex ETL pipelines and dependencies.
•Build, configure, and monitor real-time data pipelines using Apache Kafka for ingestion and processing.
•Provision, scale, and manage Kafka and Airflow using AWS services such as Amazon MSK and MWAA (or self-hosting on EKS).
•Connect Airflow with Kafka to create event-driven workflows that trigger DAGs on message arrival.
•Set up alerting and logging with AWS CloudWatch and Kafka tooling and optimize broker configurations for throughput and fault tolerance.

Key Requirements

•Deep understanding of core AWS services (S3, EC2, VPC, IAM) and managed streaming/orchestration tools such as Amazon MSK and MWAA.
•Hands-on experience with Apache Kafka architecture, including topics, partitions, and consumer groups (including KSQL/Kafka Connect).
•Proficiency in Apache Airflow, including writing Python-based DAGs and operators and using the Apache Kafka Provider for Airflow.
•Advanced proficiency in Python and SQL, including scripting and building data integration layers.
•Experience integrating with data warehouses (e.g., Snowflake, Redshift) and stream-processing engines such as Apache Spark.

Tech Stack:AWSAmazon MSKMWAAEKSApache KafkaKSQLKafka ConnectApache AirflowPythonSQLDAGsAWS CloudWatchKafka Control CenterAmazon S3Amazon EC2VPCIAMSnowflakeRedshiftApache Spark

Company Brief

NTT

Dimension Data, operating under NTT Ltd, provides managed IT services, cloud and data center solutions, networking, cybersecurity, and digital transformation services to enterprise customers worldwide.

Industry: Professional Services

Company Size: Enterprise (1,001+ employees)

Revenue: USD 1B+

Growth: Established Company

Valuation: Public Company (Market Cap in USD)

Headquarters: London, United Kingdom

Founded: 1983

Website LinkedIn

Loading job details...

Preparing the role view and application actions.

NTT