Senior AI Engineer, Quality

Fieldguide
San Francisco
Workplace: RemoteFull timeUSD 170,000 - 220,000 annuallyFunction: Data Science & Machine LearningExperience: 2+ yearsSkills: ["Problem-solving","Communication","Teamwork","Ownership"]

Engineering role focused on building and maintaining a unified evaluation platform for AI agents, including observability, production feedback loops, and evaluation infrastructure to test models against audit workflows at enterprise scale. Works at the intersection of ML engineering, observability, and quality assurance to ensure AI agents meet rigorous customer standards in a remote-first, SF-origin company.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Fieldguide
Fieldguide
3 months ago

Senior AI Engineer, Quality

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 4 hours agoStatus: Live

Job Summary

Engineering role focused on building and maintaining a unified evaluation platform for AI agents, including observability, production feedback loops, and evaluation infrastructure to test models against audit workflows at enterprise scale. Works at the intersection of ML engineering, observability, and quality assurance to ensure AI agents meet rigorous customer standards in a remote-first, SF-origin company.
Location: San Francisco
Workplace: Remote
Employment Type: Full time
Job Function: Data Science & Machine Learning

Key Responsibilities

  • •Design and build a unified evaluation platform that serves as the single source of truth for all agentic systems and audit workflows.
  • •Build observability systems that surface agent behavior, trace execution, and failure modes in production, and feedback loops that turn production failures into first-class evaluation cases.
  • •Own the evaluation infrastructure stack including integration with LangSmith and LangGraph.
  • •Translate customer problems into concrete agent behaviors and workflows.
  • •Integrate and orchestrate LLMs, tools, retrieval systems, and logic into cohesive, reliable agent experiences.

Pay and Benefits

Salary: USD 170,000 - 220,000 annually
Perks:401kRemote WorkWellness StipendFlexible Work

Key Requirements

  • •Multiple years of experience shipping production software in complex, real-world systems.
  • •Experience with TypeScript, React, Python, and PostgreSQL.
  • •Built and deployed LLM-powered features serving production traffic.
  • •Implemented evaluation frameworks for model outputs and agent behaviors.
  • •Designed observability or tracing infrastructure for AI/ML systems.
Experience:2+ yearsAISoftware engineeringEnterprise software
Skills:Problem-solvingCommunicationTeamworkOwnership
Languages:English
Tech Stack:TypeScriptReactPythonPostgreSQLLangSmithLangGraphLLMVector databasesRAG

Company Brief

Fieldguide
Provides an agentic AI platform that automates and manages end-to-end audit, advisory, and compliance engagements for CPA and consulting firms, improving efficiency, margins, and client outcomes.
Industry: Enterprise Software
Company Size: Medium (51 to 250 employees)
Revenue: USD 1M to 5M
Growth: Growth Stage Startup
Funding: Series B
Headquarters: San Francisco, United States
Founded: 2020
Glassdoor
Glassdoor: 4.6
WebsiteLinkedInGlassdoor