Python with Spark Developer (5.1-7 years)-Chennai

Capco
Chennai
Workplace: OnsiteFull timeFunction: Software EngineeringExperience: 5-7 yearsEducation: bachelorsSkills: ["Teamwork","Collaboration","Attention to detail","Results driven","Communication"]

Design, develop, test, and maintain high-performance data-processing pipelines using Python, PySpark, and SQL. Transform market, trade, and client data into trusted datasets feeding downstream analytics, reporting, and machine-learning models for CEFS. Optimize complex SQL, build CI/CD for automated job deployment, monitor production workloads, and troubleshoot performance issues. Collaborate with data architects, data scientists, and analysts in an Agile environment and mentor engineers.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Capco
Capco
6 days ago

Python with Spark Developer (5.1-7 years)-Chennai

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 5 hours agoStatus: Live

Job Summary

Design, develop, test, and maintain high-performance data-processing pipelines using Python, PySpark, and SQL. Transform market, trade, and client data into trusted datasets feeding downstream analytics, reporting, and machine-learning models for CEFS. Optimize complex SQL, build CI/CD for automated job deployment, monitor production workloads, and troubleshoot performance issues. Collaborate with data architects, data scientists, and analysts in an Agile environment and mentor engineers.
Location: Chennai
Workplace: Onsite
Employment Type: Full time
Job Function: Software Engineering
Seniority: Mid level

Key Responsibilities

  • •Design and develop robust ingestion, transformation, and enrichment pipelines using Python, PySpark, and SQL.
  • •Write and optimize complex SQL queries, analytical UDFs, and window functions for data aggregation and reporting.
  • •Collaborate with CEFS data architects, data scientists, and business analysts to translate functional requirements into technical specifications.
  • •Maintain CI/CD pipelines (Git, Jenkins, Docker) and ensure automated build, test, and deployment of data jobs.
  • •Monitor production workloads and troubleshoot performance bottlenecks, memory issues, and job failures.

Key Requirements

  • •5–7 years of experience as a Python Developer with Spark, designing and building data-processing pipelines.
  • •Strong Python, PySpark, and SQL skills, including complex SQL queries, window functions, and analytical UDFs.
  • •Experience with ETL and data ingestion/transformation/enrichment pipelines; ability to troubleshoot production job failures and performance bottlenecks.
  • •Hands-on development using Python packages such as NumPy and pandas, plus working knowledge of Linux/Unix, testing, documentation, and automation scripts.
  • •Knowledge of Agile (Scrum/Kanban) and CI/CD tooling, including Git and Jenkins (plus Docker mentioned).
Experience:5-7 yearsData engineeringData pipelinesETLMachine learning
Education:Bachelor's
Skills:TeamworkCollaborationAttention to detailResults drivenCommunication
Tech Stack:PythonPySparkSQLETLNumPyPandasRESTful APIsAPIsMicroservicesDataFramesSpark SQLStructured StreamingCI/CDGitJenkinsDockerConfluenceSharePointLinux/UnixShell scripting

Company Brief

Capco
Global management and technology consultancy specializing in financial services. Capco advises banks, insurers, and capital markets firms on digital transformation, risk and compliance, technology implementation, and operational improvement, combining industry expertise with advisory and engineering capabilities.
Industry: Management Consulting
Company Size: Enterprise (1,001+ employees)
Growth: Established Company
Headquarters: London, United Kingdom
Founded: 1998
Glassdoor
Glassdoor: 3.7
WebsiteLinkedIn