Notion Labs

1 month ago

User Researcher, AI Evaluations

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI

Last checked: 23 hours agoStatus: Live

Job Summary

Lead the evaluation framework for Notion's AI-powered experiences, defining rubrics for usefulness, trust, tone, and transparency; run longitudinal and feature-specific studies; anchor evaluation in real workflows; identify failure modes and guardrails; and collaborate with Product, Design, Engineering, and Data Science to operationalize measurement across teams.

Location: San Francisco, New York

Workplace: Hybrid

Employment Type: Full time

Job Function: Research & Scientific (R&D)

Key Responsibilities

•Define and implement reusable evaluation criteria that reflect real user expectations (helpfulness, trust, tone, control, transparency).
•Run longitudinal and feature-specific studies to measure experience quality over time against defined rubrics and benchmark improvements.
•Anchor evaluations in real workflows (jobs-to-be-done, user intent, full interaction journey such as goal setting, delegation, review, iteration).
•Identify failure modes and recovery behaviors across the system and translate insights into guardrails and prioritization.
•Collaborate with Product, Design, Engineering, and Data Science to build scalable evaluation loops and calibrate automated/LLM-based evaluation approaches against human judgments.

Key Requirements

•5+ years of UX research experience in industry.
•Experience evaluating AI-enabled products (LLMs, agents, generative UI/workflow automation) and collaborating with Data Science/ML partners on measurement strategy.
•Ability to operationalize soft user expectations (trust, tone, usefulness, clarity) into concrete rubrics, scoring guidelines, and observable metrics.
•Clear communication and impact orientation to align diverse partners and translate insights into product changes.
•Strong UX research craft (qual + quant) with method flexibility (interviews, benchmarking, surveys, experiments) and the ability to prioritize and work through ambiguity.

Experience:5+ yearsAIUX researchSaaS

Skills:CommunicationCollaborationPrioritizationProblem-solvingResearch operations

Languages:English

Tech Stack:DovetailSQLPythonLLMsAI evaluation tooling

Company Brief

Notion Labs

Notion builds a customizable all-in-one workspace for notes, docs, databases, and collaboration, now adding AI-powered features and enterprise capabilities to serve teams and organizations worldwide.

Industry: Enterprise Software

Company Size: Large (251 to 1,000 employees)

Revenue: USD 500M to 1B

Growth: Scaleup

Valuation: Unicorn (USD 1B+)

Funding: Series C

Headquarters: San Francisco, United States

Founded: 2016

Glassdoor: 4.5

Website LinkedIn Glassdoor

Loading job details...

Preparing the role view and application actions.

Notion Labs

1 month ago

User Researcher, AI Evaluations

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI

Last checked: 23 hours agoStatus: Live

Job Summary

Location: San Francisco, New York

Workplace: Hybrid

Employment Type: Full time

Job Function: Research & Scientific (R&D)

Key Responsibilities

•Define and implement reusable evaluation criteria that reflect real user expectations (helpfulness, trust, tone, control, transparency).
•Run longitudinal and feature-specific studies to measure experience quality over time against defined rubrics and benchmark improvements.
•Anchor evaluations in real workflows (jobs-to-be-done, user intent, full interaction journey such as goal setting, delegation, review, iteration).
•Identify failure modes and recovery behaviors across the system and translate insights into guardrails and prioritization.
•Collaborate with Product, Design, Engineering, and Data Science to build scalable evaluation loops and calibrate automated/LLM-based evaluation approaches against human judgments.

Key Requirements

•5+ years of UX research experience in industry.
•Experience evaluating AI-enabled products (LLMs, agents, generative UI/workflow automation) and collaborating with Data Science/ML partners on measurement strategy.
•Ability to operationalize soft user expectations (trust, tone, usefulness, clarity) into concrete rubrics, scoring guidelines, and observable metrics.
•Clear communication and impact orientation to align diverse partners and translate insights into product changes.
•Strong UX research craft (qual + quant) with method flexibility (interviews, benchmarking, surveys, experiments) and the ability to prioritize and work through ambiguity.

Experience:5+ yearsAIUX researchSaaS

Skills:CommunicationCollaborationPrioritizationProblem-solvingResearch operations

Languages:English

Tech Stack:DovetailSQLPythonLLMsAI evaluation tooling

Company Brief

Notion Labs

Notion builds a customizable all-in-one workspace for notes, docs, databases, and collaboration, now adding AI-powered features and enterprise capabilities to serve teams and organizations worldwide.

Industry: Enterprise Software

Company Size: Large (251 to 1,000 employees)

Revenue: USD 500M to 1B

Growth: Scaleup

Valuation: Unicorn (USD 1B+)

Funding: Series C

Headquarters: San Francisco, United States

Founded: 2016

Glassdoor: 4.5

Website LinkedIn Glassdoor