Qualitative Evaluation Engineer

Luma
Redwood City
Workplace: HybridFull timeUSD 150,000 - 250,000 annuallyFunction: Data Science & Machine LearningEducation: mastersSkills: ["Systems thinking","Written communication","Synthesis","Collaboration"]

Own qualitative evaluation for Luma’s generative models—building frameworks that capture believability, identity retention, and scene coherence beyond surface metrics. Evaluate performance across tasks, prompts, and modalities, uncover failure modes and regressions, and translate product goals into testable evaluative criteria. Lead human-in-the-loop studies and side-by-side comparisons, then turn nuanced judgments into actionable feedback for fine-tuning, datasets, and UX.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Luma
Luma
1 week ago

Qualitative Evaluation Engineer

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 4 hours agoStatus: Live

Job Summary

Own qualitative evaluation for Luma’s generative models—building frameworks that capture believability, identity retention, and scene coherence beyond surface metrics. Evaluate performance across tasks, prompts, and modalities, uncover failure modes and regressions, and translate product goals into testable evaluative criteria. Lead human-in-the-loop studies and side-by-side comparisons, then turn nuanced judgments into actionable feedback for fine-tuning, datasets, and UX.
Location: Redwood City
Workplace: Hybrid
Employment Type: Full time
Job Function: Data Science & Machine Learning
Seniority: Mid level

Key Responsibilities

  • •Evaluate generative model performance across diverse tasks, prompts, and modalities, surfacing failure modes, regressions, and edge cases.
  • •Build and maintain qualitative evaluation frameworks that are scalable and reusable.
  • •Translate high-level product goals into concrete evaluative criteria.
  • •Lead qualitative studies, side-by-side comparisons, and human-in-the-loop evaluations.
  • •Convert nuanced judgments into feedback that informs fine-tuning, dataset curation, and product UX.

Pay and Benefits

Salary: USD 150,000 - 250,000 annually
Equity and Bonus:Equity

Key Requirements

  • •5+ years in product evaluation, UX research, model testing, or similar structured qualitative assessment.
  • •Master’s or higher in Cognitive Science, HCI, Design Research, Psychology, Media Studies, or a related field.
  • •Deep familiarity with creative workflows for generative models (animation, filmmaking, digital art, VFX).
  • •Systems thinking to define abstract qualities (e.g., believability, scene coherence) in clear evaluative terms.
  • •Excellent written communication and the ability to synthesize nuanced judgment into actionable insight.
Experience:Product evaluationUX researchModel testingGenerative mediaHuman-in-the-loop
Education:Master's in Cognitive Science, HCI, Design Research, Psychology, Media Studies, or a related field
Skills:Systems thinkingWritten communicationSynthesisCollaboration

Company Brief

Luma
Develops AI-powered tools for capturing, editing, and rendering high-quality 3D scenes from photos and videos, enabling creators to generate photorealistic 3D assets and spatial experiences.
Industry: AR/VR & Spatial Computing
Website