Member of Technical Staff, Data Flywheel

Reflection AI
New York, San Francisco, London
Workplace: OnsiteFull timeFunction: Data Science & Machine LearningEducation: bachelorsSkills: ["Collaboration","Owning projects","Navigating ambiguity","Action-oriented","Defining and measuring model progress"]

Work on the Data Flywheel team to connect real-world model usage with rigorous evaluations that drive measurable improvements. You’ll identify and integrate high-value data sources, design evaluations and feedback loops, analyze model failures, and translate insights into datasets, reward signals, and training interventions. Build scalable pipelines to ingest and evaluate data, and collaborate across research, engineering, post-training, and live deployments.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Reflection AI
Reflection AI
1 day ago

Member of Technical Staff, Data Flywheel

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 1 hour agoStatus: Live

Job Summary

Work on the Data Flywheel team to connect real-world model usage with rigorous evaluations that drive measurable improvements. You’ll identify and integrate high-value data sources, design evaluations and feedback loops, analyze model failures, and translate insights into datasets, reward signals, and training interventions. Build scalable pipelines to ingest and evaluate data, and collaborate across research, engineering, post-training, and live deployments.
Location: New York, San Francisco, London
Workplace: Onsite
Employment Type: Full time
Job Function: Data Science & Machine Learning

Key Responsibilities

  • •Identify high-value data sources and partnership opportunities, translating use cases into representative evaluations.
  • •Bring new data sources online through scoping, quality validation, and integration into production evaluation and training pipelines.
  • •Design and build evaluations, graders, and feedback loops to measure priority real-world model behaviors.
  • •Analyze performance and failure modes, then translate insights into targeted datasets, reward signals, and training interventions.
  • •Develop human and synthetic data strategies and run evaluation/data-collection programs with vendors, while building the infrastructure to ingest and evaluate data at scale.

Pay and Benefits

Equity and Bonus:Equity
Perks:Health InsuranceDentalVisionLifePaid ParentalPaid LeaveWellness StipendEquityMeal Allowance

Key Requirements

  • •Degree (BS, MS, or PhD) in Computer Science, Machine Learning, or a related discipline, or equivalent practical experience.
  • •Deep technical understanding of LLM training and evaluation, with hands-on experience in evaluation design, data curation, reinforcement learning, or reward design.
  • •Strong software engineering skills building automated data/evaluation pipelines or large-scale ML systems.
  • •Track record of owning high-impact projects end to end, handling ambiguity and shifting priorities.
  • •Collaborative, action-oriented mindset with excitement about defining how a frontier lab measures and accelerates progress.
Education:Bachelor's in Computer Science, Machine Learning, or related discipline
Skills:CollaborationOwning projectsNavigating ambiguityAction-orientedDefining and measuring model progress
Tech Stack:LLM trainingReinforcement learningReward designEvaluation designData curation

Eligibility

Work Authorization:Sponsorship available.

Company Brief

Reflection AI
Builds frontier autonomous AI systems focused on autonomous coding agents (product: Asimov) to create organizational superintelligence, founded by former DeepMind/Google researchers and hiring across SF, NYC, London.
Industry: AI & Machine Learning
Company Size: Small (11 to 50 employees)
Growth: Growth Stage Startup
Funding: Series A
Headquarters: San Francisco, United States
Founded: 2024
WebsiteLinkedInGlassdoor