Senior Software Development Engineer in Test - AI Tooling & Automation

Salesforce
San Francisco, Bellevue
Workplace: HybridFull timeUSD 148,500 - 223,900 annuallyFunction: Data Science & Machine LearningSkills: ["Communication","Cross-functional collaboration","Problem-solving","Analytical thinking"]

Build and maintain AI-first testing frameworks and automation suites to validate complex workflows and agentic capabilities. Translate enterprise quality strategies into actionable automation plans focused on LLM integration, prompt accuracy, and deterministic behaviors. Perform hands-on white-box testing, fix code and test flakiness, and implement human-in-the-loop model evaluation for response relevance, safety, and contextual accuracy. Collaborate with product development and AI teams throughout the agile CI/CD lifecycle.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Salesforce
Salesforce
2 days ago

Senior Software Development Engineer in Test - AI Tooling & Automation

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 11 hours agoStatus: Live

Job Summary

Build and maintain AI-first testing frameworks and automation suites to validate complex workflows and agentic capabilities. Translate enterprise quality strategies into actionable automation plans focused on LLM integration, prompt accuracy, and deterministic behaviors. Perform hands-on white-box testing, fix code and test flakiness, and implement human-in-the-loop model evaluation for response relevance, safety, and contextual accuracy. Collaborate with product development and AI teams throughout the agile CI/CD lifecycle.
Location: San Francisco, Bellevue
Workplace: Hybrid
Employment Type: Full time
Job Function: Data Science & Machine Learning
Seniority: Director level

Key Responsibilities

  • •Design, build, and maintain scalable AI-first testing frameworks and automation suites for complex workflows and agentic capabilities.
  • •Translate high-level quality strategies into actionable automation plans, focusing on validating LLM integrations, prompt accuracy, and deterministic behaviors.
  • •Perform hands-on white-box testing by analyzing application code to identify quality gaps, fix code, and address test flakiness.
  • •Implement and support evaluation processes for subjective quality dimensions of AI features, including response relevance, user safety, and contextual accuracy.
  • •Collaborate with developers, product managers, and engineers to integrate quality early and continuously throughout the agile delivery lifecycle.

Pay and Benefits

Salary: USD 148,500 - 223,900 annually
Perks:MedicalDentalVision401kPaid Parental

Key Requirements

  • •Strong Java skills, including data structures and algorithms.
  • •Hands-on experience evaluating Large Language Models (LLMs) for quality responses, including familiarity with GPT-like models and AI evaluation frameworks.
  • •Proven ability delivering automation solutions in fast-paced CI/CD environments without compromising quality.
  • •Excellent communication and ability to work cross-functionally with product managers, engineers, AI/ML engineers, and data scientists.
  • •Strong analytical and problem-solving skills to identify quality gaps and drive improvements.
Experience:AI/LLMAgentic productsCI/CDEnterprise applications
Skills:CommunicationCross-functional collaborationProblem-solvingAnalytical thinking
Tech Stack:JavaLLMsGPT-like modelsCI/CDWhite-box testingAI evaluation frameworksAWSAzureGCPSalesforce platform

Company Brief

Salesforce
Provides a leading cloud-based customer relationship management (CRM) platform with sales, service, marketing, analytics, and integration tools that empower businesses to manage customer relationships and digital transformation at scale.
Industry: SaaS
Company Size: Enterprise (1,001+ employees)
Revenue: USD 1B+
Growth: Public Company
Valuation: Public Company (Market Cap in USD)
Funding: IPO / Publicly Listed
Headquarters: San Francisco, United States
Founded: 1999
Glassdoor
Glassdoor: 4.1
WebsiteLinkedInGlassdoor