What the work actually involves

This is not annotation in a browser tab. You take a public Ruby repository — often a trending gem or a well-maintained OSS library — and turn its commit and issue history into a task an LLM can be scored against. That means reproducing a historical bug, getting the project running in a clean Docker image with the right Ruby version and native extensions, confirming the fail-to-pass test genuinely fails before the fix and passes after, and writing up the issue context so the task is solvable without leaking the answer. You then run models against it and look at where they went wrong.

A large share of the effort is environment work: Bundler lock conflicts, C extension builds, database-backed test suites, flaky specs that pass locally and not in CI. The other share is judgment — deciding that a test suite's coverage is too shallow to validate a fix, or that an issue is trivially solvable by pattern-matching and therefore worthless as an evaluation item. Senior contributors also review juniors' task submissions and may lead a small pod.

What the platform screens for

  • Real Ruby depth, probed under follow-up: gem internals, metaprogramming, RSpec/Minitest idioms, Rails vs. plain-Ruby projects.
  • Containerization fluency — can you get an unfamiliar repo green in Docker without the maintainers' help?
  • Test-quality judgment: the difference between line coverage and a test that would actually catch the regression.
  • Evaluation instinct: whether you can articulate why a given issue is or isn't a fair, non-gameable task for a model.

The process is roughly 75 minutes across two rounds — a 60-minute technical and a 30-minute technical-plus-fit conversation. Expect live reasoning over code rather than algorithm puzzles.

Logistics

Fully remote contractor assignment, no benefits or paid leave. Hiring is restricted to India, Pakistan, Nigeria, Kenya, Egypt, Ghana, Bangladesh, Turkey, and Mexico. Commitment tiers are 20, 30, or 40 hours per week, with a minimum of four hours per day and at least four hours of daily overlap with PST — for most of these locations that means late-evening or overnight working hours. Pay is undisclosed on this listing; Turing typically sets an hourly rate by language, seniority, and region at offer stage.