Software Engineer, Evals

Glean
Bengaluru
Workplace: OnsiteFull timeFunction: Software EngineeringExperience: 4+ yearsSkills: ["Analytical rigor","Quality focus","Customer-focused collaboration","Cross-functional teamwork","Mentorship"]

Build large-scale evaluation and observability systems that measure and improve assistant and agent quality for real and synthetic workflows. Own backend platforms for running evaluations, processing traces, powering dashboards and debugging, and producing signals for regressions and launch readiness across frontier and OSS model releases. Partner with product, ML, and infrastructure teams to make evals a first-class part of shipping AI features while balancing reliability, security, performance, and cost.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Glean
Glean
2 months ago

Software Engineer, Evals

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 4 hours agoStatus: Live

Job Summary

Build large-scale evaluation and observability systems that measure and improve assistant and agent quality for real and synthetic workflows. Own backend platforms for running evaluations, processing traces, powering dashboards and debugging, and producing signals for regressions and launch readiness across frontier and OSS model releases. Partner with product, ML, and infrastructure teams to make evals a first-class part of shipping AI features while balancing reliability, security, performance, and cost.
Location: Bengaluru
Workplace: Onsite
Employment Type: Full time
Job Function: Software Engineering
Seniority: Mid level

Key Responsibilities

  • •Design and build large-scale evaluation pipelines measuring assistant and agent quality across real user and synthetic workflows.
  • •Evaluate frontier model releases and open-source model drops, creating infrastructure and quality signals for regressions, tradeoffs, and launch readiness.
  • •Build agent observability infrastructure including trace enrichment, telemetry pipelines, dashboards, and debugging workflows to make AI behavior inspectable.
  • •Own backend systems end-to-end from architecture and design docs through production rollout, reliability, monitoring, and iteration.
  • •Partner with product, ML, and infrastructure engineers to make evals a first-class part of how the company ships AI features and connect eval results to product improvements.

Key Requirements

  • •4+ years of software engineering experience building backend systems, infrastructure, distributed systems, or data platforms.
  • •Strong coding skills in Go, Python, Java, C++, or similar languages with an emphasis on reliability, scale, and well-tested components.
  • •Experience with distributed data pipelines, production services, observability systems, or cloud-native infrastructure.
  • •Analytical rigor and a focus on metrics that reflect real user experience.
  • •Experience with LLM applications, evals, tracing, data warehouses, workflow orchestration, or ML infrastructure is a strong plus.
Experience:4+ yearsLLM applicationsDistributed systemsData infrastructureObservabilityML infrastructure
Skills:Analytical rigorQuality focusCustomer-focused collaborationCross-functional teamworkMentorship
Languages:English
Tech Stack:GoPythonJavaC++LLM applicationsEvalsTracingObservabilityData pipelinesData warehousesWorkflow orchestrationDistributed systemsCloud-native infrastructureTelemetryDashboardsDebugging workflowsAgent observabilityFrontier model releasesOpen-source modelsMicroservices

Company Brief

Glean
Provides an enterprise search and knowledge discovery platform that helps organizations find information across apps, drives, and internal tools, improving employee productivity and onboarding through AI-powered search and relevance ranking.
Industry: Enterprise Software
Company Size: Large (251 to 1,000 employees)
Growth: Scaleup
Headquarters: San Francisco, United States
Founded: 2017
WebsiteLinkedIn