Python Engineer, AI Coding Agent Evaluator

G2i
Anywhere
Workplace: RemoteContractUSD 100 - 200 hourlyFunction: Data Science & Machine LearningSkills: ["Engineering judgment","Opinionated feedback","Critical evaluation","Subjective rigor","Communication"]

Evaluate the end-to-end quality of AI coding agent interactions (e.g., OpenAI Codex and Claude Code) in real-world scenarios. You’ll judge whether responses make sense, whether preambles and reasoning are useful, and whether outputs reflect strong engineering judgment and taste—not syntax correctness. Provide clear, opinionated feedback on what worked, what didn’t, and what feels misleading, helping define what “great” looks like with tools like Cursor.

Loading

Loading job details...

Preparing the role view and application actions.

FursaFursa
G2i
G2i
2 days ago

Python Engineer, AI Coding Agent Evaluator

✓ Verified Job

Canonical indexed version, validated from employer's careers page.

Source: Company careers pageValidated by: Fursa AI
Last checked: 10 hours agoStatus: Live

Job Summary

Evaluate the end-to-end quality of AI coding agent interactions (e.g., OpenAI Codex and Claude Code) in real-world scenarios. You’ll judge whether responses make sense, whether preambles and reasoning are useful, and whether outputs reflect strong engineering judgment and taste—not syntax correctness. Provide clear, opinionated feedback on what worked, what didn’t, and what feels misleading, helping define what “great” looks like with tools like Cursor.
Location: Anywhere
Workplace: Remote
Employment Type: Contract
Job Function: Data Science & Machine Learning
Seniority: Mid level

Key Responsibilities

  • •Evaluate AI-generated coding interactions end-to-end in real-world scenarios.
  • •Assess whether responses make sense and whether preambles, reasoning, and explanations are useful.
  • •Judge whether outputs reflect strong engineering judgment (including usefulness and high-level correctness).
  • •Provide clear, opinionated feedback on what worked, what didn’t, and what felt off or misleading.
  • •Help define what “great” looks like for interactions with tools like Cursor.

Pay and Benefits

Salary: USD 100 - 200 hourly

Key Requirements

  • •Staff/Principal-level engineering experience (or equivalent) and a high bar for “good engineering.”
  • •Strong background in TypeScript/JavaScript or Python.
  • •Hands-on experience using OpenAI Codex and/or Claude Code (and familiarity with Cursor).
  • •Deep familiarity with modern AI-assisted development workflows.
  • •Ability to make subjective but rigorous judgments about code quality and explanations without fully executing or reviewing every line.
Experience:AI coding agentsAI-assisted development
Skills:Engineering judgmentOpinionated feedbackCritical evaluationSubjective rigorCommunication
Tech Stack:PythonTypeScriptJavaScriptOpenAI CodexClaude CodeCursor

Company Brief

G2i
G2i is a video-first hiring platform that connects companies with pre-vetted remote engineers (React, React Native, GraphQL, etc.) using technical screening, live interviews, and AI-assisted hiring infrastructure to speed hiring.
Industry: HR Tech
Company Size: Medium (51 to 250 employees)
Growth: Growth Stage Startup
Headquarters: Miami, United States
Founded: 2016
Glassdoor
Glassdoor: 3.3
WebsiteLinkedInGlassdoor