Lead, Data Engineering

S&P Global
Hyderabad, Chennai
Workplace: HybridFull timeFunction: Administration & Executive AssistanceExperience: 5+ yearsSkills: ["Problem-solving","Communication","Collaboration","Troubleshooting","Mentoring"]

Lead and support an offshore team in migrating and operating enterprise data pipelines within a Databricks-on-AWS lakehouse environment. Own data onboarding, including Bronze/Silver/Gold layer standardization, metadata-driven workflows, and secure governed access using AWS and Databricks tooling. Troubleshoot pipeline issues, collaborate on data quality and lineage controls, and drive operational excellence through monitoring, incident management, CI/CD, and cloud-native orchestration.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
S&P Global
S&P Global
1 month ago

Lead, Data Engineering

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 43 days agoStatus: Live

Job Summary

Lead and support an offshore team in migrating and operating enterprise data pipelines within a Databricks-on-AWS lakehouse environment. Own data onboarding, including Bronze/Silver/Gold layer standardization, metadata-driven workflows, and secure governed access using AWS and Databricks tooling. Troubleshoot pipeline issues, collaborate on data quality and lineage controls, and drive operational excellence through monitoring, incident management, CI/CD, and cloud-native orchestration.
Location: Hyderabad, Chennai
Workplace: Hybrid
Employment Type: Full time
Job Function: Administration & Executive Assistance
Seniority: Manager level

Key Responsibilities

  • •Migrate existing data pipelines to the Databricks platform and support ongoing operation, troubleshooting, and optimization post-migration.
  • •Support data onboarding for new assets using standardized processes and help standardize Bronze/Silver/Gold lakehouse layers.
  • •Manage metadata-driven onboarding workflows using AWS Glue Data Catalog and Databricks governance capabilities.
  • •Collaborate on platform integration support, including data quality checks, metadata management, reconciliation, and auditability/lineage controls.
  • •Ensure operational excellence through deployment, monitoring, incident management, documentation, and delivery practices (CI/CD, release management, and observability).

Pay and Benefits

Perks:Health InsuranceRetirementLearning BudgetPaid LeaveStudent Loan

Key Requirements

  • •5+ years of experience in data engineering, pipeline development, or related roles.
  • •Proven experience supporting data onboarding, migration, and pipeline operations, including ongoing troubleshooting and optimization.
  • •Hands-on expertise with data engineering tools such as Spark, SQL, and Python.
  • •Experience working with Databricks (or similar cloud data platforms) and enterprise AWS data engineering services like Amazon S3 and AWS Glue.
  • •Strong understanding of secure data access, governance, and data quality/metadata/lineage concepts.
Experience:5+ yearsData engineeringData pipelinesDatabricksAWSLakehouse
Skills:Problem-solvingCommunicationCollaborationTroubleshootingMentoring
Certifications:DatabricksAWS
Tech Stack:DatabricksSparkSQLPythonPySparkAmazon S3AWS GlueAWS IAMAWS Lake FormationDatabricks Unity CatalogUnity CatalogDelta LakeApache IcebergAmazon AthenaAmazon CloudWatchCI/CDInfrastructure as codeAWS Step FunctionsAWS LambdaAmazon Kinesis

Company Brief

S&P Global
Provides financial information, analytics, benchmarks, and credit ratings to markets and institutions worldwide, offering data, research, indices, and risk assessment tools across the financial services and commodities sectors.
Industry: Data Infrastructure
Company Size: Enterprise (1,001+ employees)
Revenue: USD 1B+
Growth: Public Company
Valuation: Public Company (Market Cap in USD)
Funding: IPO / Publicly Listed
Headquarters: New York, United States
Founded: 1917
WebsiteLinkedIn