Lead Software Development Engineer in Test - AI Tooling & Automation

Salesforce
San Francisco, Bellevue
Full timeUSD 172,500 - 260,100 annuallyFunction: Data Science & Machine LearningSkills: ["Communication","Cross-functional collaboration","Analytical thinking","Problem-solving"]

Build and maintain AI-first test frameworks and automation suites to validate Revenue Action Orchestration (RAO) workflows and agentic AI capabilities. Translate quality strategies into automation plans focused on LLM/prompt accuracy and deterministic behavior, using hands-on white-box testing to diagnose code gaps and reduce test flakiness. Implement human-in-the-loop evaluation processes for LLM response quality, safety, and contextual accuracy while partnering across product, engineering, data science, and AI teams to integrate quality early across CI/CD delivery.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Salesforce
Salesforce
4 days ago

Lead Software Development Engineer in Test - AI Tooling & Automation

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 4 hours agoStatus: Live

Job Summary

Build and maintain AI-first test frameworks and automation suites to validate Revenue Action Orchestration (RAO) workflows and agentic AI capabilities. Translate quality strategies into automation plans focused on LLM/prompt accuracy and deterministic behavior, using hands-on white-box testing to diagnose code gaps and reduce test flakiness. Implement human-in-the-loop evaluation processes for LLM response quality, safety, and contextual accuracy while partnering across product, engineering, data science, and AI teams to integrate quality early across CI/CD delivery.
Location: San Francisco, Bellevue
Employment Type: Full time
Job Function: Data Science & Machine Learning

Key Responsibilities

  • •Design, build, and maintain scalable, AI-first testing frameworks and automation suites to validate RAO workflows and agentic capabilities.
  • •Translate quality strategies into actionable automation plans, emphasizing validation of LLM integrations, prompt accuracy, and deterministic system behavior.
  • •Perform hands-on white-box testing by inspecting application code to identify quality gaps, fix code issues, and reduce test flakiness.
  • •Implement and support human-in-the-loop evaluation processes for subjective AI quality dimensions such as response relevance, user safety, and contextual accuracy.
  • •Collaborate with developers, product managers, and engineers in the RAO ecosystem to ensure quality is integrated early and continuously throughout agile delivery.

Pay and Benefits

Salary: USD 172,500 - 260,100 annually
Perks:Health InsuranceDentalVisionParental LeaveLife InsuranceDisability Insurance401k

Key Requirements

  • •Strong fundamentals in Java, especially data structures and algorithms.
  • •Hands-on experience evaluating Large Language Models (LLMs) for quality responses, including familiarity with GPT-like models and AI evaluation frameworks.
  • •Proven ability to deliver automation solutions in fast-paced CI/CD environments without compromising quality.
  • •Excellent collaboration and communication skills working cross-functionally with product managers, engineers, AI/ML engineers, and data scientists.
  • •Strong analytical and problem-solving skills focused on identifying quality gaps and driving improvements.
Experience:AILLMAgenticSalesforceCI/CD
Skills:CommunicationCross-functional collaborationAnalytical thinkingProblem-solving
Tech Stack:JavaLLMGPTCI/CDAWSAzureGCPSalesforce

Company Brief

Salesforce
Provides a leading cloud-based customer relationship management (CRM) platform with sales, service, marketing, analytics, and integration tools that empower businesses to manage customer relationships and digital transformation at scale.
Industry: SaaS
Company Size: Enterprise (1,001+ employees)
Revenue: USD 1B+
Growth: Public Company
Valuation: Public Company (Market Cap in USD)
Funding: IPO / Publicly Listed
Headquarters: San Francisco, United States
Founded: 1999
Glassdoor
Glassdoor: 4.1
WebsiteLinkedInGlassdoor