hiringremote.Get free job alerts

SWE-Bench Task Auditor

$70–90/hr · AI expert network · remote, 40 hrs/week

Open to people living in the United States.

Check history

CheckedStatusSlots leftChangeHow
15 Nov 2026Open30Script, platform’s board
14 Nov 2026Open30Script, platform’s board
13 Nov 2026Open30Script, platform’s board
12 Nov 2026Open30Script, platform’s board
11 Nov 2026Open30Script, platform’s board
10 Nov 2026Open30Script, platform’s board
9 Nov 2026Open30Script, platform’s board
8 Nov 2026Open30Script, platform’s board
7 Nov 2026Open30Script, platform’s board
6 Nov 2026Open3–Script, platform’s board

The script reads the platform’s own job board every night. A job not confirmed for 10 days comes off the list.

An older role: first posted 79d ago, and AI expert network still listed it when it was last checked on 15 Nov 2026. Newer roles tend to fill faster. See the jobs hiring now.

What the work is

Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate a frontier AI lab's models. You'll assess repository-level tasks, reference patches, test harnesses, and grading integrity — and provide clear, rubric-based written feedback.

Basic Qualifications

  • 3+ years professional software engineering
  • Real open-source contribution or maintainer experience (merged PRs, committer / maintainer roles)
  • Strong ability to audit reference patches, test runners, and Docker isolation, and to detect answer leakage / reward hacking
  • Fluency across common ecosystems (Python and at least one of Java / Go / TypeScript / C++)

Preferred Qualifications

  • Familiarity with SWE-Bench (Verified) or similar repository benchmarks
  • Maintainer history on major Python OSS (Django, Flask, scikit-learn, sympy, pytest, etc.)
  • Prior code-review or task-grading experience

The platform posted this on 28 Aug 2026.

Similar jobs hiring now

Open the full list
  1. –Generalist Expert$40–70/hrAI expert networkno degreeUS, CA, UK +1US, CA, UK +1Checked auto, 15 Nov
  2. Posted 12 NovChip Design Expert (RTL Design & Verification)$100/hrAI expert network12 slots leftUS, CA, UK +1US, CA, UK +1Checked auto, 15 Nov
  3. Posted 11 NovFab Safety and Emergency Response Expert$85/hrAI expert network18 slots leftUS, CA, UK +1US, CA, UK +1Checked auto, 15 Nov
  4. Posted 9 NovFab Safety and Emergency Response Expert (Japanese)$85/hrAI expert network19 slots leftUS, CA, UK +1US, CA, UK +1Checked auto, 15 Nov
  5. Posted 5 NovCivil Engineer Reviewer$90/hrAI expert network5 slots leftUS, CA, UK +1US, CA, UK +1Checked auto, 15 Nov
  6. Posted 5 NovMechanical Engineer Reviewer$90/hrAI expert network7 slots leftUS, CA, UK +1US, CA, UK +1Checked auto, 15 Nov
  7. Posted 5 NovElectrical Engineering Writer$80/hrAI expert network31 slots leftUS, CA, UK +1US, CA, UK +1Checked auto, 15 Nov
  8. Posted 5 NovElectrical Engineer Reviewer$90/hrAI expert network3 slots leftUS, CA, UK +1US, CA, UK +1Checked auto, 15 Nov