Jump

4 months ago

Applied AI Evaluation Scientist

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI

Last checked: 13 hours agoStatus: Live

Job Summary

An applied AI evaluation scientist at Jump owns design, execution, and optimization of evaluation frameworks for AI/ML systems, with a primary focus on Agentic Retrieval-Augmented Generation (RAG) pipelines. You will design datasets, measure retrieval and generation quality, and collaborate with Product and Engineering to drive data-driven improvements for financial-advisor AI features, while balancing velocity and rigor.

Location: United States

Workplace: Remote

Employment Type: Full time

Job Function: Research & Scientific (R&D)

Seniority: Sr. Manager level

Key Responsibilities

•Design and curate evaluation datasets for retrieval quality, including synthetic query-answer-context pairs, adversarial test cases, and gold sets drawn from real user queries.
•Measure retrieval quality using metrics such as Recall@k, Precision@k, MRR, and NDCG@k and determine when each metric matters for a given use case.
•Recommend data cleaning/normalization strategies to improve discriminative power of retrieval algorithms and reduce noise.
•Evaluate and optimize chunking strategies by grid-searching chunk size, overlap, and segmentation methods and assess cascading effects on retrieval and generation quality.
•Collaborate with Product and Engineering to instrument pipelines, establish observability, and create lightweight interfaces for review and annotation.

Pay and Benefits

Salary: USD 180,000 - 270,000 annually

Equity and Bonus:Equity

Perks:Health InsuranceDentalVisionEquity

Key Requirements

•Strong product sense with ability to judge AI system outputs for user usefulness and generic vs. specification failures.
•Statistical rigor including experimental design, confidence intervals, hypothesis testing, and inter-annotator agreement metrics.
•Information retrieval fundamentals including embedding models, vector search, BM25, re-ranking, and retrieval evaluation metrics.
•Proficiency with Python and SQL and ability to write clean data analysis code and reproducible experiments.
•Hands-on experience with LLMs and data labeling/annotation, plus strong written and verbal communication skills.

Experience:5+ yearsInformation retrievalAI evaluationLLMsData science

Education:Bachelor's

Skills:CommunicationProblem-solvingCollaborationAnalytical thinkingPresentation

Languages:English

Tech Stack:PythonSQLLLMsRAGVector searchBM25OpenAI API

Company Brief

Jump

Provides an AI assistant and intelligence engine for financial advisors that automates meeting prep, note-taking, compliance documentation, CRM updates, client recaps and follow-ups to reduce admin and improve advisor productivity.

Industry: Wealth Management

Company Size: Medium (51 to 250 employees)

Growth: Growth Stage Startup

Funding: Series A

Headquarters: Salt Lake City, United States

Founded: 2023

Website LinkedIn

Loading job details...

Preparing the role view and application actions.

Jump

4 months ago

Applied AI Evaluation Scientist

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI

Last checked: 13 hours agoStatus: Live

Job Summary

Location: United States

Workplace: Remote

Employment Type: Full time

Job Function: Research & Scientific (R&D)

Seniority: Sr. Manager level

Key Responsibilities

•Design and curate evaluation datasets for retrieval quality, including synthetic query-answer-context pairs, adversarial test cases, and gold sets drawn from real user queries.
•Measure retrieval quality using metrics such as Recall@k, Precision@k, MRR, and NDCG@k and determine when each metric matters for a given use case.
•Recommend data cleaning/normalization strategies to improve discriminative power of retrieval algorithms and reduce noise.
•Evaluate and optimize chunking strategies by grid-searching chunk size, overlap, and segmentation methods and assess cascading effects on retrieval and generation quality.
•Collaborate with Product and Engineering to instrument pipelines, establish observability, and create lightweight interfaces for review and annotation.

Pay and Benefits

Salary: USD 180,000 - 270,000 annually

Equity and Bonus:Equity

Perks:Health InsuranceDentalVisionEquity

Key Requirements

•Strong product sense with ability to judge AI system outputs for user usefulness and generic vs. specification failures.
•Statistical rigor including experimental design, confidence intervals, hypothesis testing, and inter-annotator agreement metrics.
•Information retrieval fundamentals including embedding models, vector search, BM25, re-ranking, and retrieval evaluation metrics.
•Proficiency with Python and SQL and ability to write clean data analysis code and reproducible experiments.
•Hands-on experience with LLMs and data labeling/annotation, plus strong written and verbal communication skills.

Experience:5+ yearsInformation retrievalAI evaluationLLMsData science

Education:Bachelor's

Skills:CommunicationProblem-solvingCollaborationAnalytical thinkingPresentation

Languages:English

Tech Stack:PythonSQLLLMsRAGVector searchBM25OpenAI API

Company Brief

Jump

Industry: Wealth Management

Company Size: Medium (51 to 250 employees)

Growth: Growth Stage Startup

Funding: Series A

Headquarters: Salt Lake City, United States

Founded: 2023

Website LinkedIn