Mastercard

2 days ago

Manager, AI Engineering (Tester )

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI

Last checked: 5 minutes agoStatus: Live

Job Summary

Lead AI quality engineering for the Operational Intelligence Program within Business & Market Insights, ensuring Generative AI, LLM, and agentic systems are accurate, safe, reliable, and enterprise-ready. Define LLM evaluation frameworks, red-teaming strategies, and LLMOps quality gates; build test suites for agent workflows and RAG pipelines; run fairness and Responsible AI audits; and implement monitoring, drift detection, and CI/CD quality gates using Databricks on AWS.

Location: United States

Workplace: Onsite

Employment Type: Full time

Job Function: QA, Test & Release Engineering

Seniority: Manager level

Key Responsibilities

•Design and own end-to-end LLM evaluation frameworks, including automated prompt regression pipelines, output scoring, semantic benchmarking, and hallucination detection.
•Build test suites for agentic AI systems to validate tool selection, inter-agent coordination, task decomposition, goal completion, and failure handling.
•Develop RAG pipeline evaluation frameworks for retrieval precision, chunk relevance, context faithfulness, answer grounding, and hallucination rates using RAGAS, TruLens, and DeepEval.
•Lead red-teaming and adversarial testing for prompt injection, jailbreaks, data leakage, context poisoning, and model manipulation; maintain an adversarial test library.
•Implement inference performance benchmarks, production monitoring, drift detection, and observability-based quality gates within CI/CD pipelines on Databricks (AWS).

Pay and Benefits

Salary: USD 140,000 - 231,000 annually

Perks:Health Insurance401kPaid Leave

Key Requirements

•Master’s/Bachelor’s degree in Computer Science, AI/ML, or Software Engineering, with hands-on experience leading AI/ML quality engineering or LLM testing programs in production.
•Demonstrated expertise testing LLM and Gen AI systems, including prompt testing, output evaluation, hallucination detection, RAG pipeline assessment, and agentic workflow validation.
•Deep hands-on knowledge of AI evaluation frameworks and tooling such as RAGAS, DeepEval, TruLens, LangSmith, PromptFlow, and Weights & Biases Evals.
•Strong Python skills to build test automation scripts, evaluation pipelines, and API-level integration tests; SQL proficiency required.
•Familiarity with LLM ecosystems and MLOps/LLMOps pipelines (including Databricks, MLflow, and SageMaker) and experience integrating quality gates into CI/CD workflows.

Experience:Generative AILLMAgentic AIMLOps/LLMOps

Education:Master's in Computer Science, AI/ML, or Software Engineering

Skills:AnalyticalCommunicationStakeholder management

Tech Stack:PythonSQLDatabricksAWSRAGASTruLensDeepEvalGrafanaDatadogCloudWatchLangSmithPromptFlowWeights & Biases EvalsOpenAIAnthropicHugging FaceLangChainLangGraphMLflowSageMaker

Company Brief

Mastercard

Mastercard is a global payments and technology company that processes transactions, provides fraud prevention and data services, and builds payment infrastructure for consumers, businesses, merchants and governments across 200+ countries.

Industry: Payments

Company Size: Enterprise (1,001+ employees)

Revenue: USD 1B+

Growth: Public Company

Valuation: Public Company (Market Cap in USD)

Funding: IPO / Publicly Listed

Headquarters: Purchase, New York, United States

Founded: 1966

Glassdoor: 4.1

Website LinkedIn Glassdoor

Loading job details...

Preparing the role view and application actions.

Mastercard

2 days ago

Manager, AI Engineering (Tester )

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI

Last checked: 5 minutes agoStatus: Live

Job Summary

Location: United States

Workplace: Onsite

Employment Type: Full time

Job Function: QA, Test & Release Engineering

Seniority: Manager level

Key Responsibilities

•Design and own end-to-end LLM evaluation frameworks, including automated prompt regression pipelines, output scoring, semantic benchmarking, and hallucination detection.
•Build test suites for agentic AI systems to validate tool selection, inter-agent coordination, task decomposition, goal completion, and failure handling.
•Develop RAG pipeline evaluation frameworks for retrieval precision, chunk relevance, context faithfulness, answer grounding, and hallucination rates using RAGAS, TruLens, and DeepEval.
•Lead red-teaming and adversarial testing for prompt injection, jailbreaks, data leakage, context poisoning, and model manipulation; maintain an adversarial test library.
•Implement inference performance benchmarks, production monitoring, drift detection, and observability-based quality gates within CI/CD pipelines on Databricks (AWS).

Pay and Benefits

Salary: USD 140,000 - 231,000 annually

Perks:Health Insurance401kPaid Leave

Key Requirements

•Master’s/Bachelor’s degree in Computer Science, AI/ML, or Software Engineering, with hands-on experience leading AI/ML quality engineering or LLM testing programs in production.
•Demonstrated expertise testing LLM and Gen AI systems, including prompt testing, output evaluation, hallucination detection, RAG pipeline assessment, and agentic workflow validation.
•Deep hands-on knowledge of AI evaluation frameworks and tooling such as RAGAS, DeepEval, TruLens, LangSmith, PromptFlow, and Weights & Biases Evals.
•Strong Python skills to build test automation scripts, evaluation pipelines, and API-level integration tests; SQL proficiency required.
•Familiarity with LLM ecosystems and MLOps/LLMOps pipelines (including Databricks, MLflow, and SageMaker) and experience integrating quality gates into CI/CD workflows.

Experience:Generative AILLMAgentic AIMLOps/LLMOps

Education:Master's in Computer Science, AI/ML, or Software Engineering

Skills:AnalyticalCommunicationStakeholder management

Tech Stack:PythonSQLDatabricksAWSRAGASTruLensDeepEvalGrafanaDatadogCloudWatchLangSmithPromptFlowWeights & Biases EvalsOpenAIAnthropicHugging FaceLangChainLangGraphMLflowSageMaker

Company Brief

Mastercard

Industry: Payments

Company Size: Enterprise (1,001+ employees)

Revenue: USD 1B+

Growth: Public Company

Valuation: Public Company (Market Cap in USD)

Funding: IPO / Publicly Listed

Headquarters: Purchase, New York, United States

Founded: 1966

Glassdoor: 4.1

Website LinkedIn Glassdoor