Spotify

1 week ago

Multilingual AI Quality Specialist

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI

Last checked: 3 hours agoStatus: Live

Job Summary

Define quality frameworks and evaluation rubrics for multilingual AI experiences. Design structured evaluations for AI-generated, translated, curated, and recommendation-driven content, and lead dataset curation, annotation, enrichment, and ground-truth creation. Analyze results to find quality gaps and recommend improvements, including LLM-as-a-judge and human-AI agreement studies. Partner cross-functionally across Product, Engineering, Data Science, Research, Localization, and external experts to scale evaluation capabilities across Spotify.

Location: London, Stockholm

Workplace: Hybrid

Employment Type: Full time

Job Function: Quality & Regulatory (Non-Software)

Seniority: Mid level

Key Responsibilities

•Define multilingual AI quality frameworks, evaluation rubrics, thresholds, and methodologies.
•Design and execute structured evaluations for AI-generated, AI-translated, AI-curated, and recommendation-driven experiences.
•Lead multilingual dataset curation, annotation, enrichment, and ground-truth creation to support AI model evaluation.
•Analyze evaluation results, identify quality gaps, and provide actionable recommendations to improve multilingual AI quality.
•Support LLM-as-a-judge workflows, evaluator calibration, and human-AI agreement studies.

Key Requirements

•Experience in multilingual quality evaluation, localization, data curation, annotation, or AI evaluation (e.g., text-to-text or text-to-speech).
•Design or conduct structured evaluations using quality rubrics, audits, annotation projects, or review methodologies.
•Comfort using qualitative and quantitative data to identify trends, measure quality, and make recommendations.
•Familiarity with large language models (LLMs), generative AI evaluation, human-in-the-loop workflows, or LLM-as-a-judge methodologies.
•Experience with recommendation systems/personalization/search/ranking, machine translation, dataset creation, evaluator calibration, prompt testing, model evaluation, SQL, Python, dashboards, or annotation platforms (plus).

Experience:Multilingual evaluationLocalizationData curationAnnotationGenerative AILLM evaluationHuman-in-the-loop

Skills:CommunicationCross-functional collaborationWorking with ambiguityQualitative analysisQuantitative analysis

Tech Stack:LLMsGenerative AIMachine translationRecommendation systemsPersonalizationSearchRankingSQLPythonDashboardsAnnotation platformsLLM-as-a-judgeHuman-in-the-loop

Company Brief

Spotify

Provides a music, podcast and audio streaming service offering on-demand access to millions of tracks and podcasts, personalized recommendations, and advertising and subscription-based tiers for listeners and creators worldwide.

Industry: Streaming Platforms

Company Size: Enterprise (1,001+ employees)

Revenue: USD 1B+

Growth: Public Company

Valuation: Public Company (Market Cap in USD)

Funding: IPO / Publicly Listed

Headquarters: Stockholm, Sweden

Founded: 2006

Glassdoor: 4.1

Website LinkedIn Glassdoor

Loading job details...

Preparing the role view and application actions.

Spotify

1 week ago

Multilingual AI Quality Specialist

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI

Last checked: 3 hours agoStatus: Live

Job Summary

Location: London, Stockholm

Workplace: Hybrid

Employment Type: Full time

Job Function: Quality & Regulatory (Non-Software)

Seniority: Mid level

Key Responsibilities

•Define multilingual AI quality frameworks, evaluation rubrics, thresholds, and methodologies.
•Design and execute structured evaluations for AI-generated, AI-translated, AI-curated, and recommendation-driven experiences.
•Lead multilingual dataset curation, annotation, enrichment, and ground-truth creation to support AI model evaluation.
•Analyze evaluation results, identify quality gaps, and provide actionable recommendations to improve multilingual AI quality.
•Support LLM-as-a-judge workflows, evaluator calibration, and human-AI agreement studies.

Key Requirements

•Experience in multilingual quality evaluation, localization, data curation, annotation, or AI evaluation (e.g., text-to-text or text-to-speech).
•Design or conduct structured evaluations using quality rubrics, audits, annotation projects, or review methodologies.
•Comfort using qualitative and quantitative data to identify trends, measure quality, and make recommendations.
•Familiarity with large language models (LLMs), generative AI evaluation, human-in-the-loop workflows, or LLM-as-a-judge methodologies.
•Experience with recommendation systems/personalization/search/ranking, machine translation, dataset creation, evaluator calibration, prompt testing, model evaluation, SQL, Python, dashboards, or annotation platforms (plus).

Experience:Multilingual evaluationLocalizationData curationAnnotationGenerative AILLM evaluationHuman-in-the-loop

Skills:CommunicationCross-functional collaborationWorking with ambiguityQualitative analysisQuantitative analysis

Tech Stack:LLMsGenerative AIMachine translationRecommendation systemsPersonalizationSearchRankingSQLPythonDashboardsAnnotation platformsLLM-as-a-judgeHuman-in-the-loop

Company Brief

Spotify

Industry: Streaming Platforms

Company Size: Enterprise (1,001+ employees)

Revenue: USD 1B+

Growth: Public Company

Valuation: Public Company (Market Cap in USD)

Funding: IPO / Publicly Listed

Headquarters: Stockholm, Sweden

Founded: 2006

Glassdoor: 4.1

Website LinkedIn Glassdoor