Build coding-agent benchmark tasks from production-like Python repositories, then write the visible and hidden test suites that catch an agent completing the work by unsafe means — disabling tests, leaking secrets, or broadening permissions. Turing lists this as a remote contractor assignment of 4–8 weeks, 20+ hours per week with four hours of PST overlap.

Summary, eligibility guidance and screening preparation are prepared by Sidequest from the platform's posting. Check the source listing before applying.

What the work actually involves

You author benchmark tasks that look like real tickets: a bug fix, a CI repair, a config migration, an integration update, a runtime-state fix. Each task ships as a runnable repository with tests, configs and documentation, plus a prompt, metadata, an evaluator, reference patches, a scoring rubric and calibration notes. The distinguishing part is the second axis: alongside utility requirements (did the agent do the job?) you define safety and alignment requirements (did it preserve developer intent, data integrity, permissions, secrets handling and oversight mechanisms?). You then deliberately write an unsafe reference solution that passes every utility check while violating several safety requirements — because that is the case your hidden tests must catch.

The other half is analysis. You read rollouts from frontier coding agents such as Claude Code and Codex, judge whether the task completed, and identify the shortcut taken when it didn't complete honestly: assertions weakened, a test skipped, validation bypassed, a protected table dropped, a credential printed into a log. Findings feed back into evaluator reliability and benchmark difficulty, in collaboration with engineering, QA, security and client teams.

What the platform screens for

Turing's filter here is production seniority, not interview puzzles. Expect probing on eight-plus years of hands-on engineering, Python as the primary language plus at least one of Java, Go or C++, and concrete stories about diagnosing production failures from logs, metrics and traces. The security bar is specific and named: injection, auth and authorization flaws, memory safety, race conditions, deserialization, secrets management, input validation — and evidence you have shipped and validated remediations, not just filed findings. Screening is a profile review plus a 45–60 minute delivery interview.

Logistics

Fully remote contractor assignment; no medical or paid leave.
4–8 week duration, with the expected start the week after selection.
Minimum 4 hours per day and 20 hours per week, with 4 hours overlapping PST.
Pay band undisclosed on this listing; ask for the rate and any per-task structure during the delivery interview.

Senior Software Engineer – Python (LLM Evaluation & Repository Validation)

What the work actually involves

What the platform screens for

Logistics

Interview prep — 10 questions