Build verifiable software-engineering tasks from real public repository histories so LLMs can be trained and measured against them — triaging GitHub issues, Dockerizing repos, and judging whether a test suite actually proves a fix. Contractor assignment through Turing, fully remote, 20–40 hrs/week with four hours overlapping PST.

Summary, eligibility guidance and screening preparation are prepared by Sidequest from the platform's posting. Check the source listing before applying.

What the work actually is

You take a trending open-source Python repository and turn a slice of its history into a task an LLM can be scored on. That means reading the issue and the commit that closed it, reconstructing the pre-fix state, getting the project to build and its tests to run reproducibly in Docker, and then deciding whether the existing tests genuinely fail before the fix and pass after it. Where coverage is thin or the tests pass for the wrong reason, you strengthen them. A meaningful share of the day is unglamorous environment work — pinned dependencies, system libraries, flaky network-dependent tests, CI configs that assume a runner you don't have.

The second half is judgment. You work with researchers to pick issues that are hard for models for interesting reasons — cross-file reasoning, subtle state, ambiguous specs — rather than issues that are merely tedious. You run and modify codebases locally to see where a model's patch goes wrong, and you feed that back into task selection. There are stated opportunities to lead a group of junior engineers on repo onboarding.

What the screen looks for

Turing's process is roughly 75 minutes across two rounds: a 60-minute technical and a 30-minute technical-plus-fit conversation. Expect concrete probing on Python, Git internals (rebases, reverting a commit, bisecting), Docker, and test frameworks — and expect follow-ups on specific repositories you claim to know. Vague familiarity with "open source" is easy to expose; being able to say what a named project's test suite is like, and what breaks when you first clone it, is not.

Logistics

Contractor assignment, no medical or paid leave; 3-month initial duration with a start date roughly a week out.
Minimum 4 hours/day and 20 hours/week, with 4 hours overlapping PST; 30 and 40 hrs/week options exist.
Fully remote, but hiring is restricted to India, Pakistan, Nigeria, Kenya, Egypt, Ghana, Bangladesh, Turkey, and Mexico.
Pay band undisclosed on this listing — ask for the rate and the committed weekly hours in the first call.

Senior Software Engineer – Python (LLM Evaluation & Repository Validation)

What the work actually is

What the screen looks for

Logistics

Interview prep — 10 questions