Quality Assurance Engineer

AI71
Abu Dhabi
Workplace: OnsiteFull timeFunction: QA, Test & Release EngineeringExperience: 5+ yearsSkills: ["Analytical mindset","Collaboration","Defining pass/fail criteria","Communication","Traceability management"]

Lead validation and verification of non-deterministic AI systems, defining pass/fail criteria for probabilistic outputs and “confidence levels” for stakeholders in a defense-grade environment. Build automated testing frameworks for LLM behavior (including hallucination checks, RAG evaluation, and prompt regression), integrate system and ERP verification, and enforce secure API validation. Map test cases to V-Model requirements, prepare Stage Gate readiness evidence, and manage defect traceability across model versions and datasets.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
AI71
AI71
3 months ago

Quality Assurance Engineer

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 4 hours agoStatus: Live
Reposted: similar role first listed 11 months ago

Job Summary

Lead validation and verification of non-deterministic AI systems, defining pass/fail criteria for probabilistic outputs and “confidence levels” for stakeholders in a defense-grade environment. Build automated testing frameworks for LLM behavior (including hallucination checks, RAG evaluation, and prompt regression), integrate system and ERP verification, and enforce secure API validation. Map test cases to V-Model requirements, prepare Stage Gate readiness evidence, and manage defect traceability across model versions and datasets.
Location: Abu Dhabi
Workplace: Onsite
Employment Type: Full time
Job Function: QA, Test & Release Engineering
Seniority: Mid level

Key Responsibilities

  • •Architect automated non-deterministic testing frameworks for Generative AI outputs, validating consistency and factual accuracy against Ground Truth/Gold Standard datasets.
  • •Implement automated RAG evaluation metrics to verify retrieval accuracy, citation correctness, and answer relevance/faithfulness to internal technical documentation and regulatory texts.
  • •Create regression suites to detect prompt drift and ensure AI-generated engineering documents maintain quality as models and system instructions change.
  • •Build integration tests for AI agents with enterprise systems (e.g., SAP S/4HANA, Ariba) to ensure data consistency and prevent corruption of BOM/financial data.
  • •Map automated test cases to System Requirements/User Needs for V-Model alignment, prepare Stage Gate test readiness packages, and manage defect traceability back to specific model versions or datasets.

Key Requirements

  • •5+ years of QA Automation experience, including at least 2 years testing data-driven applications, ML models, or AI agents.
  • •Expert proficiency in Python test harnesses (Pytest) and automation libraries such as Selenium/Playwright and Requests for API testing.
  • •Expertise in performance testing plans and implementations (e.g., Locust, JMeter, K6) to validate latency and throughput.
  • •Experience evaluating LLMs using frameworks such as DeepEval and TruLens (or custom Python evaluators), including Ground Truth dataset creation/management.
  • •Knowledge of CI/CD test integration (GitLab CI/CD) and requirements traceability tools (e.g., Jira, Linear, Jama, Polarion, etc.), plus test reporting/artifacts tools (e.g., TestRail, Allure).
Experience:5+ yearsMLAIData-driven applicationsLLMsDefenseAerospaceRegulated industries
Skills:Analytical mindsetCollaborationDefining pass/fail criteriaCommunicationTraceability management
Languages:English
Tech Stack:PythonPytestSeleniumPlaywrightRequestsLocustJmeterK6DeepEvalTruLensSQLGreat ExpectationsGitGitlabGitLab CI/CDJiraLinearJamaPolarionTestRail

Company Brief

AI71
AI71 builds generative AI and machine learning products and solutions, focusing on applying large language models and conversational AI to automate workflows, enhance decision-making, and deliver domain-specific AI applications for enterprise customers.
Industry: AI & Machine Learning
Website