hiringremote.Get free job alerts

Applied Engineer, Evaluations Role

$100–120/hr · AI expert network · remote

Open to people living in the United States, Canada, the United Kingdom and any EU country.

Check history

CheckedStatusSlots leftChangeHow
15 Nov 2026Open––Script, platform’s board
14 Nov 2026Open––Script, platform’s board
13 Nov 2026Open––Script, platform’s board
12 Nov 2026Open––Script, platform’s board
11 Nov 2026Open––Script, platform’s board
10 Nov 2026Open––Script, platform’s board
9 Nov 2026Open––Script, platform’s board
8 Nov 2026Open––Script, platform’s board
7 Nov 2026Open––Script, platform’s board
6 Nov 2026Open––Script, platform’s board

The script reads the platform’s own job board every night. A job not confirmed for 10 days comes off the list.

An older role: first posted 531d ago, and AI expert network still listed it when it was last checked on 15 Nov 2026. Newer roles tend to fill faster. See the jobs hiring now.

What the work is

We work alongside an in-house research team and have produced industry leading benchmarks to compare the performance of frontier large language models.

In this role, you will:

  • Identify repositories with enough substantive work to support challenging evaluations, and select suitable completed PRs.
  • Investigate each problem, its reference solution, and the repository’s architecture, tests, and conventions.
  • Write task prompts and build evaluations using automated tests, shell commands, and LLM grading prompts.
  • Validate evaluations against reference solutions and deliberately flawed implementations. Find missing checks, incorrect grades, and criteria that unnecessarily constrain how a problem can be solved.
  • Run evaluations repeatedly across multiple models and harnesses. Investigate whether failures come from the agent’s solution, the environment, or the grading, and establish that tasks expose meaningful weaknesses in agent performance.
  • Refine evaluations through repeated testing and review. Document findings and grading decisions, and work through feedback.

You’ll need:

  • Strong programming fundamentals and practical experience working in substantial and complex codebases. Languages we create evals for include TypeScript/JavaScript, Python, Java, Kotlin, Go, Ruby, PHP, C++ and Rust
  • The ability to understand unfamiliar code, investigate subtle behavior, and assess whether different implementations solve the same problem correctly.
  • Experience writing meaningful tests, including edge cases and regression coverage.
  • Attention to detail and patience for repeated investigation and refinement.
  • Practical experience using AI coding tools, with the judgment to verify their output and catch mistakes.
  • Confidence using Git, test runners, and CI tooling.
  • Clear and fluent written and verbal communication with the ability to own a task independently while raising questions when requirements are ambiguous.

Relevant experience can come from open-source, private, or enterprise repositories. Familiarity with a particular language or ecosystem is helpful; the ability to learn the repository and make sound engineering judgments matters more than its popularity or your public contribution history.

We provide onboarding to the evaluation framework and ongoing review feedback. After onboarding, you’ll be expected to own task selection, evaluation development, and iteration without step-by-step direction.

The platform posted this on 2 Jun 2025.

Similar jobs hiring now

Open the full list
  1. –Generalist Expert$40–70/hrAI expert networkno degreeUS, CA, UK +1US, CA, UK +1Checked auto, 15 Nov
  2. Posted 12 NovChip Design Expert (RTL Design & Verification)$100/hrAI expert network12 slots leftUS, CA, UK +1US, CA, UK +1Checked auto, 15 Nov
  3. Posted 11 NovFab Safety and Emergency Response Expert$85/hrAI expert network18 slots leftUS, CA, UK +1US, CA, UK +1Checked auto, 15 Nov
  4. Posted 9 NovFab Safety and Emergency Response Expert (Japanese)$85/hrAI expert network19 slots leftUS, CA, UK +1US, CA, UK +1Checked auto, 15 Nov
  5. Posted 5 NovCivil Engineer Reviewer$90/hrAI expert network5 slots leftUS, CA, UK +1US, CA, UK +1Checked auto, 15 Nov
  6. Posted 5 NovMechanical Engineer Reviewer$90/hrAI expert network7 slots leftUS, CA, UK +1US, CA, UK +1Checked auto, 15 Nov
  7. Posted 5 NovElectrical Engineering Writer$80/hrAI expert network31 slots leftUS, CA, UK +1US, CA, UK +1Checked auto, 15 Nov
  8. Posted 5 NovElectrical Engineer Reviewer$90/hrAI expert network3 slots leftUS, CA, UK +1US, CA, UK +1Checked auto, 15 Nov