AI Data Analyst

Glean
Bengaluru
Workplace: HybridFull timeFunction: Data Analytics & Business IntelligenceExperience: 3-5 yearsSkills: ["Analytical judgment","Attention to detail","Communication","Collaboration","Independent work"]

Evaluate AI output quality for a human evaluation system powering response quality, competitive benchmarking, and release readiness. Run human labeling for key AI workflows, triage quality issues, and perform qualitative analysis to spot failure patterns like hallucinations and retrieval failures. Maintain benchmark datasets and regression slices, participate in calibration and LLM-as-a-judge validation, and collaborate with QA, Engineering, Product, and eval owners on quality reporting and release decisions.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Glean
Glean
1 day ago

AI Data Analyst

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 17 hours agoStatus: Live

Job Summary

Evaluate AI output quality for a human evaluation system powering response quality, competitive benchmarking, and release readiness. Run human labeling for key AI workflows, triage quality issues, and perform qualitative analysis to spot failure patterns like hallucinations and retrieval failures. Maintain benchmark datasets and regression slices, participate in calibration and LLM-as-a-judge validation, and collaborate with QA, Engineering, Product, and eval owners on quality reporting and release decisions.
Location: Bengaluru
Workplace: Hybrid
Employment Type: Full time
Job Function: Data Analytics & Business Intelligence
Seniority: Mid level

Key Responsibilities

  • •Run ongoing human labeling for priority AI workflows, including response quality, task success, MCP and tool use, and end-to-end Cowork-style workflows.
  • •Triage bad-query reports, downvotes, escalations, and routed quality issues; categorize failure modes and share analysis with partner teams.
  • •Perform qualitative analysis to identify recurring patterns such as hallucinations, retrieval failures, tool-use issues, weak grounding, and poor workflow completion.
  • •Maintain high-quality labeled datasets, golden sets, and regression slices; monitor coverage, drift, leakage, and difficulty distribution.
  • •Participate in calibration exercises with human graders and validate LLM-as-a-judge outputs against human labels; support pre/post-change quality reads and release readiness decisions.

Key Requirements

  • •3–5 years of experience in data labeling, data analysis, QA, or a related field.
  • •Strong analytical judgment and attention to detail to apply nuanced rubrics consistently.
  • •Experience evaluating AI-generated outputs or working with NLP, search, recommendation, or other ML systems.
  • •Familiarity with qualitative analysis, basic SQL/data retrieval, and spreadsheet-based workflows.
  • •Clear written and verbal communication skills to explain findings to technical and non-technical audiences.
Experience:3-5 years
Skills:Analytical judgmentAttention to detailCommunicationCollaborationIndependent work
Languages:English
Tech Stack:SQLNLP

Company Brief

Glean
Provides an enterprise search and knowledge discovery platform that helps organizations find information across apps, drives, and internal tools, improving employee productivity and onboarding through AI-powered search and relevance ranking.
Industry: Enterprise Software
Company Size: Large (251 to 1,000 employees)
Growth: Scaleup
Headquarters: San Francisco, United States
Founded: 2017
WebsiteLinkedIn