Applied Computer Science Benchmark Specialist

$66–84/hr

rate as listed

Workload: 10+ hrs/week
Engagement: Project-based
Location: Location not stated
Field: STEM and Coding

Write and verify expert-level multiple-choice computer science questions — one correct answer, nine plausible distractors, and a step-by-step solution with academic citations — for a benchmark used to measure AI capability. Work is asynchronous and remote, with an expected commitment of 10+ hours per week; observed pay for this listing runs $66–84/hr.

What the work involves

You will be assigned one of two tracks. Authoring means producing original questions in your own subdomain: a precisely stated problem, a difficulty rating (Medium for intro undergraduate, Hard for advanced undergraduate, Expert for post-graduate and beyond), one correct answer, nine distractors that are individually defensible until you reason carefully, a markdown Chain-of-Thought solution, and one to five references from peer-reviewed or university sources. Verification means auditing pre-written items for accuracy, ambiguity, missing premises, and solvability — editing where needed and documenting exactly what you changed and why.

The hard part is rarely the computer science. It is writing a question that is fully self-contained (no implicit assumptions about compiler version, memory model, or scheduling policy), and then generating nine wrong answers that reflect genuine expert failure modes rather than arithmetic noise. Domains in scope include GPU kernel engineering, formal methods and automated reasoning, computer architecture, distributed systems, DevOps and SRE, data engineering and databases, cloud infrastructure, OS and kernel internals, ML engineering, web and API development, embedded systems, graphics and game development, and mobile engineering.

What the platform screens for

Mercor's screen is résumé parsing plus an AI-led interview. Expect it to test:

Verifiable specifics — named institutions, advisors, papers, systems you shipped, teams you were on.
Depth under follow-up — a claimed specialty will be probed two or three layers past the textbook answer, often into edge cases and failure modes.
Evaluation judgment — whether you can distinguish a question that tests reasoning from one that tests recall, and whether you can articulate why an item is ambiguous.
Calibration — realistic difficulty ratings, and honesty about the boundary of your expertise.

Credentials favored: PhD or doctoral candidate in CS, EE, or an adjacent field; a Master's with unusual depth in one subdomain is also considered. Publications, senior engineering experience at a well-known infrastructure or ML organization, or a competitive programming record all help.

Logistics

Fully remote, fully asynchronous, no fixed hours or meetings. Expected commitment is 10+ hours per week; work arrives in task batches and throughput varies with your domain's demand. Pay is hourly at the observed band above — bands on this platform reflect what candidates have reported for comparable work and are not a guarantee. Written English quality matters as much as technical accuracy, since your solution text is part of the deliverable.

What they look for

PhD or doctoral candidate in Computer Science, Electrical Engineering, or a closely related field (exceptional Master's-level depth considered)
Graduate-level command of at least one listed subdomain — e.g. GPU kernels, formal methods, distributed systems, OS internals, or ML engineering
Ability to write unambiguous, self-contained technical problems and step-by-step solutions in markdown
Skill at constructing plausible distractors that reflect real expert reasoning errors
Familiarity with peer-reviewed literature and the ability to cite primary sources accurately
Clear, concise written English and reliable async self-management at 10+ hrs/week

Before you apply

Would you pass this screen?

Drawn from what this listing actually requires. Answered in your browser — nothing is sent to us, and we do not store your answers.

Do you hold a PhD (completed or in progress) in Computer Science, Electrical Engineering, or a closely related field, or a Master's degree with significant published research or top-tier industry work in a specific CS subdomain?
PhD in Computer Science or closely related field, or advanced Master's + exceptional domain depth · required
Are you available to commit 10 or more hours per week to this role on an ongoing, asynchronous basis?
10+ hours per week availability · required
Do you have published research in a peer-reviewed venue, or professional experience in computer science at a top technology company or research organization?
Research publications or top-tier industry experience · preferred
Are you able to write clearly and concisely in English well enough to author unambiguous technical problems and solutions?
Fluent written English · required

0 of 4 answered

Included with this listing

Interview prep — 11 questions

Matched to what this platform's screening actually measures. Answer each out loud in 60–90 seconds: claim → concrete example → implication.

What it tests: Whether the candidate has conducted original research at the depth needed to author graduate-level assessment questions, and whether they can articulate technical rigor from firsthand work rather than course study alone.
A strong answer: Name a specific problem (e.g., a distributed consensus algorithm, memory optimization technique, or ML architecture choice), explain the research gap it addressed, and describe both the theoretical foundation and the engineering steps taken to validate it—showing they've worked at the boundary between concept and execution.
Where people lose it: Describing coursework, a hackathon project, or someone else's research instead of their own published or presented work; or staying too high-level without showing the technical depth that would let them evaluate whether a question is actually rigorous.

Summary, eligibility guidance and screening preparation are prepared by Sidequest from the platform's posting. Check the source listing before applying.

STEM AND CODING on Sidequest

What stem and coding AI work pays

Rates by specialty, the platforms hiring, and every open role in this field.