What the work actually is
You are not reviewing model output line by line. You are authoring the problems that measure it. A typical task starts as a technically substantial project — a small service, a feature spanning several files, a bug lodged in a realistic codebase — that you write, document, and then wrap in a test harness. The test suite has to be deterministic: same container, same commit, same result, every run. Flaky tests, hidden network calls, timestamp-dependent assertions, and tests that pass because of an unstated environment quirk are the main reasons submissions get rejected.
Documentation carries as much weight as the code. Each task needs a problem statement precise enough that a competent engineer — or a model — could attempt it without seeing your solution, plus a rationale for what the task discriminates. AfterQuery generally wants tasks that a strong model fails for interesting reasons, not tasks that are simply long or obscure.
What the screen looks for
- Evidence you have architected or maintained multi-file projects, not just single-file exercises
- Fluency with Docker: base image choices, pinned dependencies, reproducible builds
- Concrete habits around test determinism — seeding, fixtures, isolation, time and network control
- Technical writing you can point to: READMEs, RFCs, design docs, contribution guides
- Judgment about what makes a coding task a fair measurement rather than a trick
Logistics
Fully remote and asynchronous, contract-based, with pay observed in the $50–100/hr range depending on language depth and task complexity — not a guaranteed rate. Most contributors set their own hours and take work in batches; there is usually a paid or unpaid sample task before ongoing volume. Expect review cycles: your first few submissions will come back with notes on reproducibility before the throughput picks up.