Instructure

2 weeks ago

Senior AI Engineer (Evaluations - Canvas Agent)

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI

Last checked: 6 hours agoStatus: Live

Job Summary

Design and implement LLM-as-a-judge evaluation pipelines for the Canvas Agent, scoring multi-step reasoning, tool usage, and conversational helpfulness. Build scalable evaluation frameworks, rubrics, and “golden” datasets, iterate on judge prompts, and translate evaluation signals into actionable engineering improvements. Integrate automated evaluations into CI/CD to enable fast, high-confidence AI releases, and contribute to full-stack AI feature development as needed.

Location: Budapest

Workplace: Hybrid

Employment Type: Full time

Job Function: Data Science & Machine Learning

Seniority: Mid level

Key Responsibilities

•Build and maintain scalable LLM-as-a-judge evaluation pipelines to score the Canvas Agent’s actions, responses, and tool usage across complex educational workflows.
•Create grading rubrics and curate high-quality golden datasets (real and synthetic) to baseline and test agent performance.
•Engineer and iterate on judge prompts so automated scoring aligns with high-quality evaluations.
•Contribute to full-stack product development as needed, collaborating to build and refine core AI features, UI components, and application architecture.
•Integrate automated evaluations into CI/CD pipelines and monitor evaluation metrics to identify failure modes, hallucination rates, and regressions, then translate findings into engineering tasks.

Pay and Benefits

Salary: HUF 1,500,000 - 2,200,000 monthly

Perks:Learning Budget

Key Requirements

•Hands-on experience designing and writing automated tests for LLMs, including proven LLM-as-a-judge evaluations in real-world scenarios.
•Strong background in Python (preferred) and willingness/ability to learn it quickly; experience in full-stack or backend engineering.
•Prompt engineering expertise for reliable classification, extraction, and grading without common model biases.
•Familiarity with LLM observability and evaluation frameworks such as LangSmith, Braintrust, Ragas, or promptfoo.
•Understanding of agentic architectures for planning and executing tool calls (e.g., ReAct) and evaluating multi-step workflows.

Experience:LLM evaluationsAI agentsEducational technology

Skills:Analytical mindsetCollaboration skills

Tech Stack:PythonTypeScriptNode.jsLangSmithBraintrustRagasPromptfooReActCI/CDLLM

Company Brief

Instructure

Builds and delivers learning management and education technology solutions, best known for the Canvas LMS platform, serving K–12 schools, higher education institutions, and corporate learning programs worldwide.

Industry: EdTech

Company Size: Enterprise (1,001+ employees)

Growth: Established Company

Funding: Private Equity Backed

Headquarters: Salt Lake City, United States

Founded: 2008

Website LinkedIn

Loading job details...

Preparing the role view and application actions.

Instructure

2 weeks ago

Senior AI Engineer (Evaluations - Canvas Agent)

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI

Last checked: 6 hours agoStatus: Live

Job Summary

Location: Budapest

Workplace: Hybrid

Employment Type: Full time

Job Function: Data Science & Machine Learning

Seniority: Mid level

Key Responsibilities

•Build and maintain scalable LLM-as-a-judge evaluation pipelines to score the Canvas Agent’s actions, responses, and tool usage across complex educational workflows.
•Create grading rubrics and curate high-quality golden datasets (real and synthetic) to baseline and test agent performance.
•Engineer and iterate on judge prompts so automated scoring aligns with high-quality evaluations.
•Contribute to full-stack product development as needed, collaborating to build and refine core AI features, UI components, and application architecture.
•Integrate automated evaluations into CI/CD pipelines and monitor evaluation metrics to identify failure modes, hallucination rates, and regressions, then translate findings into engineering tasks.

Pay and Benefits

Salary: HUF 1,500,000 - 2,200,000 monthly

Perks:Learning Budget

Key Requirements

•Hands-on experience designing and writing automated tests for LLMs, including proven LLM-as-a-judge evaluations in real-world scenarios.
•Strong background in Python (preferred) and willingness/ability to learn it quickly; experience in full-stack or backend engineering.
•Prompt engineering expertise for reliable classification, extraction, and grading without common model biases.
•Familiarity with LLM observability and evaluation frameworks such as LangSmith, Braintrust, Ragas, or promptfoo.
•Understanding of agentic architectures for planning and executing tool calls (e.g., ReAct) and evaluating multi-step workflows.

Experience:LLM evaluationsAI agentsEducational technology

Skills:Analytical mindsetCollaboration skills

Tech Stack:PythonTypeScriptNode.jsLangSmithBraintrustRagasPromptfooReActCI/CDLLM

Company Brief

Instructure

Industry: EdTech

Company Size: Enterprise (1,001+ employees)

Growth: Established Company

Funding: Private Equity Backed

Headquarters: Salt Lake City, United States

Founded: 2008

Website LinkedIn