LLM Engineer (LLM Evaluation)

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI

Last checked: 1 hour agoStatus: Live

Job Summary

Design and build an end-to-end evaluation system for large language models, including benchmark datasets, human/LLM-based metrics, and evaluation protocols focused on reproducibility. Develop evaluation automation using Argo Workflows and MLflow, integrate it with ML pipelines, and add regression detection with alerting for deployment validation. Operate and continuously improve model quality validation workflows for large-scale, stable model releases in Kubernetes-based environments.

Location: South Korea

Workplace: Hybrid

Employment Type: Full time

Job Function: Data Science & Machine Learning

Seniority: Mid level

Key Responsibilities

•Design LLM evaluation and benchmarks by building benchmark datasets and defining Human/LLM-based metrics.
•Establish evaluation protocols for fair model comparisons and ensure reproducibility.
•Build evaluation automation environments using Argo Workflows and MLflow, integrating them with ML pipelines.
•Design regression detection and alerting when model performance degrades after deployment.
•Validate model quality at scale with repeatable workflows and operate continuous model quality improvement processes.

Key Requirements

•3+ years of experience in LLM training or evaluation.
•Research and development experience in Deep Learning or NLP.
•Experience using LLM evaluation frameworks such as lm-eval, HELM, or OpenAI Evals.
•Experience developing Python-based services, including async processing.
•Understanding of experiment management and reproducibility, with experience designing model evaluation/validation workflows.

Skills:Collaboration

Tech Stack:PythonAsyncKubernetesArgo WorkflowsMLflowLm-evalHELMOpenAI EvalsDatadogPrometheusContainersGPU

Company Brief

42dot

Website

Loading job details...

Preparing the role view and application actions.

42dot

1 week ago

LLM Engineer (LLM Evaluation)

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI

Last checked: 1 hour agoStatus: Live

Job Summary

Location: South Korea

Workplace: Hybrid

Employment Type: Full time

Job Function: Data Science & Machine Learning

Seniority: Mid level

Key Responsibilities

•Design LLM evaluation and benchmarks by building benchmark datasets and defining Human/LLM-based metrics.
•Establish evaluation protocols for fair model comparisons and ensure reproducibility.
•Build evaluation automation environments using Argo Workflows and MLflow, integrating them with ML pipelines.
•Design regression detection and alerting when model performance degrades after deployment.
•Validate model quality at scale with repeatable workflows and operate continuous model quality improvement processes.

Key Requirements

•3+ years of experience in LLM training or evaluation.
•Research and development experience in Deep Learning or NLP.
•Experience using LLM evaluation frameworks such as lm-eval, HELM, or OpenAI Evals.
•Experience developing Python-based services, including async processing.
•Understanding of experiment management and reproducibility, with experience designing model evaluation/validation workflows.

Skills:Collaboration

Tech Stack:PythonAsyncKubernetesArgo WorkflowsMLflowLm-evalHELMOpenAI EvalsDatadogPrometheusContainersGPU

Company Brief

42dot

Website