Senior Evaluation Algorithm Engineer

Binance
Hong Kong, Taipei, Sydney
Workplace: RemoteFull timeFunction: Solutions Engineering & Sales EngineeringEducation: mastersSkills: ["Communication","Business understanding","Cross-team collaboration","Analytical thinking"]

Design end-to-end LLM evaluation plans for dialogue and financial trading scenarios, turning subjective quality judgments into quantifiable, reproducible, and explainable results. Build evaluation metric systems, rubrics, and high-quality datasets with rigorous annotation guidelines and quality control. Analyze model failure modes and capability boundaries to guide R&D iteration, and automate/scale evaluation workflows via evaluation platforms and toolchains in partnership with algorithm, product, and data teams.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
Binance
Binance
1 day ago

Senior Evaluation Algorithm Engineer

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 2 hours agoStatus: Live

Job Summary

Design end-to-end LLM evaluation plans for dialogue and financial trading scenarios, turning subjective quality judgments into quantifiable, reproducible, and explainable results. Build evaluation metric systems, rubrics, and high-quality datasets with rigorous annotation guidelines and quality control. Analyze model failure modes and capability boundaries to guide R&D iteration, and automate/scale evaluation workflows via evaluation platforms and toolchains in partnership with algorithm, product, and data teams.
Location: Hong Kong, Taipei, Sydney
Workplace: Remote
Employment Type: Full time
Job Function: Solutions Engineering & Sales Engineering
Seniority: Mid level

Key Responsibilities

  • •Design end-to-end LLM evaluation plans for business scenarios such as dialogue and financial trading, including metric systems and rubrics.
  • •Lead evaluation dataset design and construction: define evaluation dimensions, scenario coverage, annotation guidelines, quality control, and benchmarks that reflect business needs.
  • •Analyze model capability boundaries and failure modes, then produce actionable improvement recommendations and collaborate to drive model iteration.
  • •Drive automation and scaling of evaluation workflows by building sustainable evaluation platforms and toolchains for high-frequency evaluation during rapid iteration.
  • •Collaborate with algorithm, product, and data teams to translate business and model objectives into evaluation standards and drive implementation of R&D directions from evaluation findings.

Pay and Benefits

Perks:Remote Work

Key Requirements

  • •Master’s degree or above in computer science, AI, mathematics, statistics, or related fields with strong algorithmic foundations and understanding of LLM principles plus training and fine-tuning.
  • •Hands-on LLM evaluation experience, including commercial deployment evaluation with knowledge of the evaluation pipeline from data prep and rubric design to evaluation-driven R&D.
  • •Familiarity with mainstream evaluation methods (human evaluation, LLM-as-a-judge, metric computation) and ability to define evaluation dimensions and write clear, discriminative rubrics.
  • •Systematic control of evaluation data representativeness, annotation consistency, and result reliability to ensure scientific, trustworthy conclusions.
  • •Proficient in Python with experience automating evaluation workflows, building benchmarks/evaluation platforms, and independently handling data processing, scripts, and result analysis.
Experience:LLM evaluationAI agentsFinancial tradingData annotation
Education:Master's
Skills:CommunicationBusiness understandingCross-team collaborationAnalytical thinking
Tech Stack:PythonLLMLLM-as-a-judgeRLHF

Company Brief

Binance
Operates one of the world’s largest cryptocurrency exchanges, offering spot and derivatives trading, a native token (BNB), wallet services, staking, and a broad suite of crypto financial products and infrastructure.
Industry: Trading Platforms
Company Size: Enterprise (1,001+ employees)
Revenue: USD 1B+
Growth: Established Company
Valuation: Unicorn (USD 1B+)
Funding: Bootstrapped
Headquarters: George Town, Cayman Islands
Founded: 2017
WebsiteLinkedIn