Research, Post-Training Evals

Thinking Machines Lab
San Francisco
Workplace: HybridFull timeFunction: Education & TrainingEducation: bachelorsSkills: ["Communication","Collaboration","Research judgment","Technical writing","Debugging"]

Build the next generation of internal evaluation and research signals for post-training, covering usability, correctness, auditing, agentic evaluation, and robustness to gaming. Create and improve graders, benchmarks, and onboarding/auditing methodologies so researchers can trust and use eval signals effectively. Develop evaluations that detect meaningful improvements and generalize across setups, plus core signal sets that support efficient internal RL research and personalized behavior dimensions.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Thinking Machines Lab
Thinking Machines Lab
1 day ago

Research, Post-Training Evals

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 1 hour agoStatus: Live

Job Summary

Build the next generation of internal evaluation and research signals for post-training, covering usability, correctness, auditing, agentic evaluation, and robustness to gaming. Create and improve graders, benchmarks, and onboarding/auditing methodologies so researchers can trust and use eval signals effectively. Develop evaluations that detect meaningful improvements and generalize across setups, plus core signal sets that support efficient internal RL research and personalized behavior dimensions.
Location: San Francisco
Workplace: Hybrid
Employment Type: Full time
Job Function: Education & Training

Key Responsibilities

  • •Create internal evaluations and research signals for capabilities and behaviors important to model research and post-training.
  • •Develop usability evaluations that measure genuine research/product usefulness and feed insights into better data and training signals.
  • •Improve evaluation correctness, including grader reliability, ambiguous ground truth, evaluator disagreement, and false positives/negatives.
  • •Build eval onboarding and auditing methodologies that help researchers understand, trust, and appropriately use internal evaluation signals.
  • •Develop specialized agentic evaluations and build signal-bearing core sets for efficient, high-quality internal RL research, including personalized behavior dimensions.

Key Requirements

  • •Bachelor’s degree (or equivalent) in Computer Science, Machine Learning, Physics, Mathematics, or a related field with strong theoretical and empirical grounding.
  • •Experience designing, building, or analyzing evaluations, benchmarks, datasets, graders, or other measurement systems.
  • •Strong written and verbal communication skills and the ability to collaborate effectively across research and engineering teams.
  • •Proficiency in Python and familiarity with at least one deep learning framework (PyTorch, TensorFlow, or JAX), including comfort debugging distributed training and writing scalable code.
  • •Experience with LLMs/post-training, reinforcement learning, or agentic systems (preferred).
Experience:LLMsPost-trainingReinforcement learningAgentic systems
Education:Bachelor's
Skills:CommunicationCollaborationResearch judgmentTechnical writingDebugging
Tech Stack:PythonPyTorchTensorFlowJAX

Company Brief

Thinking Machines Lab
Develops enterprise AI solutions, custom large language models, and ML platforms to help organizations deploy intelligent applications. Services include data engineering, model development, and AI consulting for scale and production readiness.
Industry: AI & Machine Learning
Company Size: Medium (51 to 250 employees)
Growth: Growth Stage Startup
Headquarters: Mumbai, India
Website