Senior Product Engineer (Agent Training and Evals)

Prolific
London
Workplace: RemoteFull timeFunction: Education & TrainingSkills: ["Communication","Collaboration","Product mindset","Ability to work with ambiguity"]

Build infrastructure for evaluating AI agents, capturing agent traces at scale alongside real human expert work in secure environments. Join a 0→1 Agentic Training & Evals team as an early engineer, turning ambiguous problems into scalable systems with product and AI research partners. Create tooling and approaches for training, benchmarking, and evaluation, making pragmatic decisions across compute, orchestration, scaling, security, and reliability.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Prolific
Prolific
4 hours ago

Senior Product Engineer (Agent Training and Evals)

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 1 hour agoStatus: Live

Job Summary

Build infrastructure for evaluating AI agents, capturing agent traces at scale alongside real human expert work in secure environments. Join a 0→1 Agentic Training & Evals team as an early engineer, turning ambiguous problems into scalable systems with product and AI research partners. Create tooling and approaches for training, benchmarking, and evaluation, making pragmatic decisions across compute, orchestration, scaling, security, and reliability.
Location: London
Workplace: Remote
Employment Type: Full time
Job Function: Education & Training
Seniority: Mid level

Key Responsibilities

  • •Build tooling to capture and process data from agents and humans performing real-world tasks at significant scale.
  • •Solve compute, orchestration, scaling, security, and reliability challenges for agent evaluation infrastructure.
  • •Develop approaches for training, benchmarking, and evaluating AI agents.
  • •Work with emerging agentic AI technologies and rapidly evolving models.
  • •Partner closely with product engineering and AI research to deliver working products and drive direct impact in a cross-functional team.

Key Requirements

  • •Strong experience building scalable infrastructure or distributed systems in the cloud.
  • •Solid understanding of how systems are deployed, scaled, and secured.
  • •Hands-on experience integrating and evaluating AI models or agents.
  • •Interest in how AI agents are trained, evaluated, and improved.
  • •Comfort with ambiguity and moving quickly from idea to working system.
Skills:CommunicationCollaborationProduct mindsetAbility to work with ambiguity
Languages:English
Tech Stack:Google Cloud PlatformPythonJavaScriptTypeScriptFastAPIVue.jsPostgres SQLGitHub ActionsKubernetesDataDog

Company Brief

Prolific
Operates an online participant recruitment platform connecting researchers and companies with vetted participants for academic and market research studies, offering survey hosting, participant management, and data quality tools.
Industry: SaaS
Company Size: Medium (51 to 250 employees)
Growth: Growth Stage Startup
Funding: Series A
Headquarters: Oxford, United Kingdom
Founded: 2014
WebsiteLinkedIn