Head of Evaluations Research

Aaru
New York
Workplace: OnsiteFull timeUSD 450,000 - 750,000 annuallyFunction: Research & Scientific (R&D)Skills: ["Experimental design","Statistical rigor","Debugging and failure analysis","Clear technical communication","Research leadership"]

Lead evaluation research to determine how well population simulations, predictions, and agent-based systems match real-world behavior. You’ll set the evaluation charter, design experiments and longitudinal studies, build and maintain evaluation datasets, and write analysis code to diagnose failures. Partner closely with simulation engineering and research teams while preserving independent conclusions, and recruit/mentor a small team to deliver reproducible, decision-ready evidence for customers and the public.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Aaru
Aaru
1 day ago

Head of Evaluations Research

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 15 hours agoStatus: Live

Job Summary

Lead evaluation research to determine how well population simulations, predictions, and agent-based systems match real-world behavior. You’ll set the evaluation charter, design experiments and longitudinal studies, build and maintain evaluation datasets, and write analysis code to diagnose failures. Partner closely with simulation engineering and research teams while preserving independent conclusions, and recruit/mentor a small team to deliver reproducible, decision-ready evidence for customers and the public.
Location: New York
Workplace: Onsite
Employment Type: Full time
Job Function: Research & Scientific (R&D)
Seniority: Manager level

Key Responsibilities

  • •Set an end-to-end evaluation agenda across population construction, predictive systems, agent behavior, group dynamics, and simulations.
  • •Translate questions about realism, accuracy, and usefulness into measurable constructs, experiments, and decision criteria.
  • •Design quality tests for population coherence and representation, and evaluate forecasts/predictions using holdouts, calibration, ranking quality, and subgroup performance.
  • •Run longitudinal and interaction-based evaluations to test how agents change over time, respond to new information, and influence one another.
  • •Recruit, mentor, and lead a small team while staying hands-on by building datasets, writing analysis code, diagnosing failures, and producing decision-ready evidence.

Pay and Benefits

Salary: USD 450,000 - 750,000 annually
Equity and Bonus:Equity
Perks:Equity

Key Requirements

  • •Developed an evaluation or measurement agenda with rigorous methods across machine learning, behavioral science, computational social science, statistics, economics, psychometrics, or comparable fields.
  • •Built evaluations that changed research direction, model capability, product decisions, or scientific conclusions.
  • •Comfortable with experimental design, observational data, sampling, uncertainty, statistical power, leakage, and condition shift.
  • •Able to write code, analyze large datasets, design studies, and inspect individual model failures.
  • •Led researchers or a major technical direction while remaining directly involved in the work.
Experience:Machine learning evaluationBehavioral scienceComputational social scienceStatisticsEconomicsPsychometricsCausal inferenceExperimental economics
Skills:Experimental designStatistical rigorDebugging and failure analysisClear technical communicationResearch leadership
Tech Stack:Analysis codeLLM agentsMulti-agent systemsSynthetic populationsProbabilistic modelsRecommender systemsEvaluation platformsExperiment tracking systems

Company Brief

Aaru
Aaru builds multi-agent simulation software that models entire populations to predict behavior and future events, replacing traditional research with decision-ready forecasts for enterprises, governments, and agencies across industries.
Industry: AI & Machine Learning
Company Size: Small (11 to 50 employees)
Revenue: USD 1M to 5M
Growth: Early Stage Startup
Valuation: Unicorn (USD 1B+)
Funding: Series A
Headquarters: San Francisco, United States
Founded: 2024
WebsiteLinkedIn