Databricks Data Engineer Senior Analyst

Avanade
Brisbane, Sydney, Melbourne
Workplace: OnsiteFull timeFunction: Solutions Engineering & Sales EngineeringEducation: bachelorsSkills: ["Analytical and problem-solving","Communication","Stakeholder management","Collaboration","Agile mindset"]

Design, develop, and optimize scalable data ingestion pipelines and transformation workflows on Azure Databricks. Build PySpark-based Databricks notebooks to prepare and enrich datasets for analytics and Data Science use cases. Implement data quality, monitoring, and validation, integrate structured/semi-structured/unstructured sources, and ensure data governance, security, and compliance for banking regulations. Collaborate with Data Scientists, participate in agile ceremonies and code reviews, and optimize Spark jobs for performance and cost-efficiency.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Avanade
Avanade
1 day ago

Databricks Data Engineer Senior Analyst

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 11 hours agoStatus: Live

Job Summary

Design, develop, and optimize scalable data ingestion pipelines and transformation workflows on Azure Databricks. Build PySpark-based Databricks notebooks to prepare and enrich datasets for analytics and Data Science use cases. Implement data quality, monitoring, and validation, integrate structured/semi-structured/unstructured sources, and ensure data governance, security, and compliance for banking regulations. Collaborate with Data Scientists, participate in agile ceremonies and code reviews, and optimize Spark jobs for performance and cost-efficiency.
Location: Brisbane, Sydney, Melbourne
Workplace: Onsite
Employment Type: Full time
Job Function: Solutions Engineering & Sales Engineering
Seniority: Mid level

Key Responsibilities

  • •Design, develop, and optimize scalable data ingestion pipelines using Azure Data Factory, Databricks, and Apache Spark.
  • •Build and maintain Databricks notebooks with PySpark for data preparation, transformation, and enrichment.
  • •Collaborate with Data Scientists to understand data requirements and deliver clean, structured datasets.
  • •Implement data quality checks, validation rules, and monitoring mechanisms.
  • •Ensure data security, governance, and compliance with banking regulations; optimize Spark jobs for performance and cost-efficiency in Azure.

Key Requirements

  • •2+ years hands-on experience in Azure Databricks data engineering.
  • •Strong Apache Spark and PySpark skills for distributed data processing.
  • •Experience with Azure Data Factory, Azure Synapse, Azure Blob Storage, and Delta Lake.
  • •Proficiency in Python and/or Scala, plus solid ETL/ELT, data modeling, and data warehousing concepts.
  • •Understanding of data governance, security, and compliance in financial services (Australian citizenship required; eligible to obtain security clearance).
Experience:Data engineeringBankingFinancial services
Education:Bachelor's in Computer Science, Information Systems, Engineering, or related field
Skills:Analytical and problem-solvingCommunicationStakeholder managementCollaborationAgile mindset
Certifications:Databricks Certified Data Engineer AssociateDatabricks Certified Data Engineer Professional
Tech Stack:Azure DatabricksAzure Data FactoryApache SparkPySparkDatabricks notebooksAzure SynapseAzure Blob StorageDelta LakePythonScalaAzure DevOpsGitTerraformSQLNoSQLData lake architectures

Eligibility

Nationality:Australian National
Security Clearance:Security clearance

Company Brief

Avanade
Global digital innovation and IT services company delivering Microsoft-based cloud, digital and application services. Partners with enterprises to design, build and run business solutions across cloud, data, AI, modern workplace and industry-specific transformations.
Industry: Consulting
Company Size: Enterprise (1,001+ employees)
Revenue: USD 1B+
Growth: Established Company
Headquarters: Seattle, United States
Founded: 2000
Glassdoor
Glassdoor: 3.9
WebsiteLinkedIn