Manager, AI Engineering (Tester )

Mastercard
United States
Workplace: OnsiteFull timeUSD 140,000 - 231,000 annuallyFunction: QA, Test & Release EngineeringEducation: mastersSkills: ["Analytical","Communication","Stakeholder management"]

Lead AI quality engineering for the Operational Intelligence Program within Business & Market Insights, ensuring Generative AI, LLM, and agentic systems are accurate, safe, reliable, and enterprise-ready. Define LLM evaluation frameworks, red-teaming strategies, and LLMOps quality gates; build test suites for agent workflows and RAG pipelines; run fairness and Responsible AI audits; and implement monitoring, drift detection, and CI/CD quality gates using Databricks on AWS.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Mastercard
Mastercard
1 month ago

Manager, AI Engineering (Tester )

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Status: Live

Job Summary

Lead AI quality engineering for the Operational Intelligence Program within Business & Market Insights, ensuring Generative AI, LLM, and agentic systems are accurate, safe, reliable, and enterprise-ready. Define LLM evaluation frameworks, red-teaming strategies, and LLMOps quality gates; build test suites for agent workflows and RAG pipelines; run fairness and Responsible AI audits; and implement monitoring, drift detection, and CI/CD quality gates using Databricks on AWS.
Location: United States
Workplace: Onsite
Employment Type: Full time
Job Function: QA, Test & Release Engineering
Seniority: Manager level

Key Responsibilities

  • •Design and own end-to-end LLM evaluation frameworks, including automated prompt regression pipelines, output scoring, semantic benchmarking, and hallucination detection.
  • •Build test suites for agentic AI systems to validate tool selection, inter-agent coordination, task decomposition, goal completion, and failure handling.
  • •Develop RAG pipeline evaluation frameworks for retrieval precision, chunk relevance, context faithfulness, answer grounding, and hallucination rates using RAGAS, TruLens, and DeepEval.
  • •Lead red-teaming and adversarial testing for prompt injection, jailbreaks, data leakage, context poisoning, and model manipulation; maintain an adversarial test library.
  • •Implement inference performance benchmarks, production monitoring, drift detection, and observability-based quality gates within CI/CD pipelines on Databricks (AWS).

Pay and Benefits

Salary: USD 140,000 - 231,000 annually
Perks:Health Insurance401kPaid Leave

Key Requirements

  • •Master’s/Bachelor’s degree in Computer Science, AI/ML, or Software Engineering, with hands-on experience leading AI/ML quality engineering or LLM testing programs in production.
  • •Demonstrated expertise testing LLM and Gen AI systems, including prompt testing, output evaluation, hallucination detection, RAG pipeline assessment, and agentic workflow validation.
  • •Deep hands-on knowledge of AI evaluation frameworks and tooling such as RAGAS, DeepEval, TruLens, LangSmith, PromptFlow, and Weights & Biases Evals.
  • •Strong Python skills to build test automation scripts, evaluation pipelines, and API-level integration tests; SQL proficiency required.
  • •Familiarity with LLM ecosystems and MLOps/LLMOps pipelines (including Databricks, MLflow, and SageMaker) and experience integrating quality gates into CI/CD workflows.
Experience:Generative AILLMAgentic AIMLOps/LLMOps
Education:Master's in Computer Science, AI/ML, or Software Engineering
Skills:AnalyticalCommunicationStakeholder management
Tech Stack:PythonSQLDatabricksAWSRAGASTruLensDeepEvalGrafanaDatadogCloudWatchLangSmithPromptFlowWeights & Biases EvalsOpenAIAnthropicHugging FaceLangChainLangGraphMLflowSageMaker

Company Brief

Mastercard
Mastercard is a global payments and technology company that processes transactions, provides fraud prevention and data services, and builds payment infrastructure for consumers, businesses, merchants and governments across 200+ countries.
Industry: Payments
Company Size: Enterprise (1,001+ employees)
Revenue: USD 1B+
Growth: Public Company
Valuation: Public Company (Market Cap in USD)
Funding: IPO / Publicly Listed
Headquarters: Purchase, New York, United States
Founded: 1966
Glassdoor
Glassdoor: 4.1
WebsiteLinkedInGlassdoor