What the work actually involves

You take a real issue from a trending open-source Python library and turn it into a task an LLM can be scored against. In practice that means finding the commit that fixed the issue, reconstructing the pre-fix state, and building a Docker image where the repository's test suite runs reproducibly — pinned dependencies, system packages, fixtures, network-free execution where possible. Then you decide whether the tests actually discriminate: do they fail before the fix and pass after it, do they fail for the right reason, and would a plausible-looking wrong patch slip past them? A task with a flaky test, a test that passes on the broken code, or a fix that leaks into a docstring the model can read is a task that gets rejected.

Alongside instance construction you will run modified codebases locally to see where models break down, and talk to researchers about which repositories and issue types are genuinely hard — dependency upgrades, concurrency bugs, subtle API contract changes — rather than one-line fixes that every model already solves. There are stated opportunities to lead a group of junior engineers on repository batches.

What the screen looks for

The platform's screen is AI-led and probes specifics: repositories you can name and describe the build system of, how you handle a project whose test suite needs a database or a GPU, how you tell a weak test from a strong one, and how you keep an environment deterministic. Vague claims about "strong Python experience" collapse under follow-up; concrete build failures you debugged do not. Expect questions about Docker layer caching, dependency pinning, and pytest behaviour, plus judgment scenarios where the right answer is to reject or rework a task rather than ship it.

Logistics

  • Fully remote, contractor assignment — no medical or paid leave.
  • Three-month duration, with the listing indicating a start roughly a week out.
  • Eligible locations as stated: India, Pakistan, Nigeria, Egypt, Ghana, Bangladesh, Turkey, Mexico.
  • Asynchronous and task-paced: pay is per accepted task at an observed $100, so throughput and rework rate matter more than hours logged. Minimum 3+ years of engineering experience is required, tech-lead-level familiarity with real codebases preferred.