User Researcher, AI Evaluations

Notion Labs
San Francisco, New York
Workplace: HybridFull timeFunction: Research & Scientific (R&D)Experience: 5+ yearsSkills: ["Communication","Collaboration","Prioritization","Problem-solving","Research operations"]

Lead the evaluation framework for Notion's AI-powered experiences, defining rubrics for usefulness, trust, tone, and transparency; run longitudinal and feature-specific studies; anchor evaluation in real workflows; identify failure modes and guardrails; and collaborate with Product, Design, Engineering, and Data Science to operationalize measurement across teams.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Notion Labs
Notion Labs
3 months ago

User Researcher, AI Evaluations

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 6 hours agoStatus: Live

Job Summary

Lead the evaluation framework for Notion's AI-powered experiences, defining rubrics for usefulness, trust, tone, and transparency; run longitudinal and feature-specific studies; anchor evaluation in real workflows; identify failure modes and guardrails; and collaborate with Product, Design, Engineering, and Data Science to operationalize measurement across teams.
Location: San Francisco, New York
Workplace: Hybrid
Employment Type: Full time
Job Function: Research & Scientific (R&D)

Key Responsibilities

  • •Define and implement reusable evaluation criteria that reflect real user expectations (helpfulness, trust, tone, control, transparency).
  • •Run longitudinal and feature-specific studies to measure experience quality over time against defined rubrics and benchmark improvements.
  • •Anchor evaluations in real workflows (jobs-to-be-done, user intent, full interaction journey such as goal setting, delegation, review, iteration).
  • •Identify failure modes and recovery behaviors across the system and translate insights into guardrails and prioritization.
  • •Collaborate with Product, Design, Engineering, and Data Science to build scalable evaluation loops and calibrate automated/LLM-based evaluation approaches against human judgments.

Key Requirements

  • •5+ years of UX research experience in industry.
  • •Experience evaluating AI-enabled products (LLMs, agents, generative UI/workflow automation) and collaborating with Data Science/ML partners on measurement strategy.
  • •Ability to operationalize soft user expectations (trust, tone, usefulness, clarity) into concrete rubrics, scoring guidelines, and observable metrics.
  • •Clear communication and impact orientation to align diverse partners and translate insights into product changes.
  • •Strong UX research craft (qual + quant) with method flexibility (interviews, benchmarking, surveys, experiments) and the ability to prioritize and work through ambiguity.
Experience:5+ yearsAIUX researchSaaS
Skills:CommunicationCollaborationPrioritizationProblem-solvingResearch operations
Languages:English
Tech Stack:DovetailSQLPythonLLMsAI evaluation tooling

Company Brief

Notion Labs
Notion builds a customizable all-in-one workspace for notes, docs, databases, and collaboration, now adding AI-powered features and enterprise capabilities to serve teams and organizations worldwide.
Industry: Enterprise Software
Company Size: Large (251 to 1,000 employees)
Revenue: USD 500M to 1B
Growth: Scaleup
Valuation: Unicorn (USD 1B+)
Funding: Series C
Headquarters: San Francisco, United States
Founded: 2016
Glassdoor
Glassdoor: 4.5
WebsiteLinkedInGlassdoor