Multilingual AI Quality Specialist

Spotify
London, Stockholm
Workplace: HybridFull timeFunction: Quality & Regulatory (Non-Software)Skills: ["Communication","Cross-functional collaboration","Working with ambiguity","Qualitative analysis","Quantitative analysis"]

Define quality frameworks and evaluation rubrics for multilingual AI experiences. Design structured evaluations for AI-generated, translated, curated, and recommendation-driven content, and lead dataset curation, annotation, enrichment, and ground-truth creation. Analyze results to find quality gaps and recommend improvements, including LLM-as-a-judge and human-AI agreement studies. Partner cross-functionally across Product, Engineering, Data Science, Research, Localization, and external experts to scale evaluation capabilities across Spotify.

This position is no longer accepting applications.

  • See live roles at Spotify
  • Search all live jobs
  • Browse companies, collections, and locations hiring now
Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa

This position is no longer accepting applications.

See live roles at SpotifySearch all live jobsBrowse companies, collections, and locations hiring now

Spotify
Spotify
1 month ago

Multilingual AI Quality Specialist

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 3 hours agoStatus: Closed

Job Summary

Define quality frameworks and evaluation rubrics for multilingual AI experiences. Design structured evaluations for AI-generated, translated, curated, and recommendation-driven content, and lead dataset curation, annotation, enrichment, and ground-truth creation. Analyze results to find quality gaps and recommend improvements, including LLM-as-a-judge and human-AI agreement studies. Partner cross-functionally across Product, Engineering, Data Science, Research, Localization, and external experts to scale evaluation capabilities across Spotify.
Location: London, Stockholm
Workplace: Hybrid
Employment Type: Full time
Job Function: Quality & Regulatory (Non-Software)
Seniority: Mid level

Key Responsibilities

  • •Define multilingual AI quality frameworks, evaluation rubrics, thresholds, and methodologies.
  • •Design and execute structured evaluations for AI-generated, AI-translated, AI-curated, and recommendation-driven experiences.
  • •Lead multilingual dataset curation, annotation, enrichment, and ground-truth creation to support AI model evaluation.
  • •Analyze evaluation results, identify quality gaps, and provide actionable recommendations to improve multilingual AI quality.
  • •Support LLM-as-a-judge workflows, evaluator calibration, and human-AI agreement studies.

Key Requirements

  • •Experience in multilingual quality evaluation, localization, data curation, annotation, or AI evaluation (e.g., text-to-text or text-to-speech).
  • •Design or conduct structured evaluations using quality rubrics, audits, annotation projects, or review methodologies.
  • •Comfort using qualitative and quantitative data to identify trends, measure quality, and make recommendations.
  • •Familiarity with large language models (LLMs), generative AI evaluation, human-in-the-loop workflows, or LLM-as-a-judge methodologies.
  • •Experience with recommendation systems/personalization/search/ranking, machine translation, dataset creation, evaluator calibration, prompt testing, model evaluation, SQL, Python, dashboards, or annotation platforms (plus).
Experience:Multilingual evaluationLocalizationData curationAnnotationGenerative AILLM evaluationHuman-in-the-loop
Skills:CommunicationCross-functional collaborationWorking with ambiguityQualitative analysisQuantitative analysis
Tech Stack:LLMsGenerative AIMachine translationRecommendation systemsPersonalizationSearchRankingSQLPythonDashboardsAnnotation platformsLLM-as-a-judgeHuman-in-the-loop

Company Brief

Spotify
Provides a music, podcast and audio streaming service offering on-demand access to millions of tracks and podcasts, personalized recommendations, and advertising and subscription-based tiers for listeners and creators worldwide.
Industry: Streaming Platforms
Company Size: Enterprise (1,001+ employees)
Revenue: USD 1B+
Growth: Public Company
Valuation: Public Company (Market Cap in USD)
Funding: IPO / Publicly Listed
Headquarters: Stockholm, Sweden
Founded: 2006
Glassdoor
Glassdoor: 4.1
WebsiteLinkedInGlassdoor