NTT

1 month ago

Senior Data Engineer - Teradata/Databricks & AI/ML

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI

Last checked: 5 hours agoStatus: Live

Job Summary

Ingest and validate Teradata DBQL logs, then integrate metadata from Autosys, DataStage, and MagicWand to power log-driven analysis. Build Databricks pipelines to identify unused datasets, read-only datasets, and unused partitions, while measuring CPU/IO usage and workload statistics to quantify cost-reduction opportunities. Apply AI/ML and LLM-assisted anomaly detection to classify data tiers and produce stakeholder-ready reports, workshops, and readouts.

Location: Bengaluru

Workplace: Onsite

Employment Type: Full time

Job Function: Data Science & Machine Learning

Seniority: Mid level

Key Responsibilities

•Ingest and validate 18+ months of Teradata DBQL logs, including SQL text, object usage, timestamps, user/application IDs, row counts, and steps.
•Integrate metadata from Autosys, DataStage (orchestration), and MagicWand (observability) to supplement DBQL analysis.
•Build end-to-end, log-driven analysis pipelines in Databricks to identify unused datasets, read-only datasets, and unused partitions within active datasets.
•Analyze CPU/IO resource usage and workload statistics (ResUsage) to quantify cost-reduction opportunities and classify data into cold/warm/hot tiers.
•Use AI/ML models or LLM-assisted analysis to detect access-pattern anomalies, predict cold data candidates, and automate classification; deliver Observation Reports, workshop notes/action logs, and final readouts.

Key Requirements

•Experience ingesting and validating large-scale Teradata DBQL logs with SQL text, usage, timestamps, and row counts.
•Hands-on experience building end-to-end log-driven analytics pipelines in Databricks to detect unused datasets, partitions, and access patterns.
•Ability to integrate and analyze metadata from Autosys (scheduling), DataStage (orchestration), and MagicWand (observability).
•Experience analyzing CPU/IO resource usage and workload statistics (ResUsage) to quantify cost-reduction opportunities.
•Familiarity with AI/ML or LLM-assisted techniques for anomaly detection, cold data prediction, and automated data classification.

Skills:Stakeholder communicationData analysisPipeline developmentProblem-solving

Tech Stack:TeradataDBQLSQLDatabricksAutosysDataStageMagicWandResUsageCPUIOLLMAI/MLHeatmaps

Company Brief

NTT

Dimension Data, operating under NTT Ltd, provides managed IT services, cloud and data center solutions, networking, cybersecurity, and digital transformation services to enterprise customers worldwide.

Industry: Professional Services

Company Size: Enterprise (1,001+ employees)

Revenue: USD 1B+

Growth: Established Company

Valuation: Public Company (Market Cap in USD)

Headquarters: London, United Kingdom

Founded: 1983

Website LinkedIn

Loading job details...

Preparing the role view and application actions.

NTT