What the work actually is

You spend your day inside an agentic coding harness. On online assignments you receive predefined tasks, interact with blinded models, and then rank and grade the resulting trajectories — maintaining model blindness and keeping sessions independent so your judgments stay uncontaminated. On offline assignments you go upstream: you design multi-step coding tasks with realistic user intent and milestone structure, calibrate them through user simulation, write task-specific rubrics and binary criteria, and grade the trajectories they produce.

Verification is hands-on rather than impressionistic. You read the diff, run the tests, check the logs, inspect generated artifacts, and write scripts when the existing checks don't settle the question. Every ranking ships with a rationale that points at specific evidence — which command you ran, what it returned, which edge case the implementation silently dropped. You are also expected to notice when the environment is broken or the instructions are ambiguous, and escalate that with supporting evidence instead of quietly guessing.

What the screen is looking for

Turing is explicit that this is not entry-level annotation. The bar is 5+ years in software engineering, QA, developer tooling, or data/ML engineering, with real depth in one or two ecosystems — Python, JavaScript/TypeScript, Rust, Java, C/C++, Bash/CLI, Haskell, Swift, SQL, or similar. Screens tend to probe whether you can orient in an unfamiliar repository, tell a functional fix from a plausible-looking one, and hold a consistent standard across many runs. For offline and senior tracks, Docker fluency, experience in large legacy repositories, and demonstrated originality in framing technical problems carry extra weight.

Logistics

  • Fully remote contractor engagement, no medical or paid leave.
  • Stated commitment: 8 hours per day with a 4-hour overlap with PST.
  • Contract duration 5 weeks, with start expected roughly a week after selection.
  • Compensation is described as competitive and scaled to experience and project scope; no band is published, so treat any figure you hear as observed rather than guaranteed.