Senior Data Engineer

Red Hat
Raleigh
Workplace: RemoteFull timeUSD 158,309 - 180,000 annuallyFunction: Solutions Engineering & Sales EngineeringEducation: mastersSkills: []

Architect and implement high-volume data pipelines between Snowflake and Databricks using PySpark and dbt, with Airflow-based orchestration and automated recovery. Administer Databricks and build secure, governance-ready environments. Deliver AI-driven retrieval and end-to-end ML solutions using MLflow, vector search, and monitoring. Manage CI/CD and containerized deployments on Red Hat OpenShift, while leading security compliance (SAST, scans, threat modeling) and centralized logging.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Red Hat
Red Hat
4 days ago

Senior Data Engineer

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 7 hours agoStatus: Live

Job Summary

Architect and implement high-volume data pipelines between Snowflake and Databricks using PySpark and dbt, with Airflow-based orchestration and automated recovery. Administer Databricks and build secure, governance-ready environments. Deliver AI-driven retrieval and end-to-end ML solutions using MLflow, vector search, and monitoring. Manage CI/CD and containerized deployments on Red Hat OpenShift, while leading security compliance (SAST, scans, threat modeling) and centralized logging.
Location: Raleigh
Workplace: Remote
Employment Type: Full time
Job Function: Solutions Engineering & Sales Engineering
Seniority: Mid level

Key Responsibilities

  • •Orchestrate and monitor high-volume data pipeline scheduling, dependency management, and automated failure recovery using Apache Airflow.
  • •Administer the Databricks platform, configuring IAM roles for S3 access, managing credentials/secrets, and setting workspace governance and cluster policies.
  • •Design and deploy intelligent retrieval and AI-driven workflows (vector search, Transformer/transfer learning approaches) for marketing retrieval and decision automation using multiple data sources and APIs.
  • •Operationalize MLOps with MLflow (experiment tracking, model registry) and lakehouse monitoring to improve predictive accuracy and marketing ROI.
  • •Manage CI/CD and containerized deployments on Red Hat OpenShift (OpenShift/Kubernetes, TLS termination, routing) and lead application security work including SAST, vulnerability scanning, PIA, and STRIDE threat modeling.

Pay and Benefits

Salary: USD 158,309 - 180,000 annually
Equity and Bonus:Equity

Key Requirements

  • •Master’s in Computer Science (or related) with 3 years of relevant experience, or Bachelor’s with 5 years of relevant experience.
  • •3+ years architecting high-volume pipelines between Snowflake and Databricks using PySpark and dbt, including macro-driven data quality tests and validation logic.
  • •3+ years orchestrating/scheduling workflows with Apache Airflow (dependency management, automated failure recovery, pipeline monitoring).
  • •Experience administering Databricks (IAM/S3 access), managing secrets via vault and OpenShift secrets, and setting workspace governance and cluster policies.
  • •Experience building ML/IR workflows including NER with TextBlob/gensim/fastText, predictive models (XGBoost/Scikit-learn), deep learning with Keras, and operationalizing MLOps with MLflow and monitoring.
Experience:B2B marketing attributionMLOpsLakehouse
Education:Master's in Computer Science or related field
Tech Stack:SnowflakeDatabricksPySparkDbtApache AirflowAmazon S3IAMOpenShiftKubernetesMLflowGitTektonBuildahSkopeoTextBlobGensimFastTextXGBoostScikit-learnKeras

Company Brief

Red Hat
Provides enterprise open-source software solutions, including Red Hat Enterprise Linux, middleware, cloud, container, and Kubernetes technologies, delivering support, services, and solutions for hybrid cloud environments.
Industry: Enterprise Software
Company Size: Enterprise (1,001+ employees)
Revenue: USD 1B+
Growth: Established Company
Valuation: Unicorn (USD 1B+)
Funding: IPO / Publicly Listed
Headquarters: Raleigh, United States
Founded: 1993
WebsiteLinkedIn