The work

AI labs need reliable ways to score whether a model-generated web app actually works. That scoring usually happens through browser automation — a Playwright or Cypress suite that clicks through the app and asserts on the result. If those tests are flaky, order-dependent, or assert on something ambiguous, the benchmark data is worthless. Your job is to catch that before it ships.

Day to day you'll open a task containing a web application spec, the generated app, and a proposed browser test suite. You read the tests the way you'd read a colleague's PR: Are the selectors stable or are they going to break on a re-render? Does the test wait properly, or is it leaning on a sleep? Does teardown actually reset state, or will run #2 fail because run #1 left a record in the database? Are the assertions specific enough that a pass genuinely means the feature works, and a fail points at a real defect? You then write structured feedback against a project rubric — usually a set of dimension scores plus written justification that a downstream engineer can act on.

What the screen looks for

  • Verifiable hands-on background. Three-plus years as an SE, QA engineer, SDET, or test automation engineer, with specific frameworks and specific systems you tested. Vague "familiar with Selenium" answers don't survive follow-up.
  • Flakiness diagnosis under pressure. Expect to be handed a test snippet and asked what will break and why. Race conditions, implicit vs. explicit waits, shared fixtures, and network stubbing come up constantly.
  • Judgment about assertion quality. Distinguishing a test that passes for the right reason from one that passes accidentally is the core skill here, and it's the hardest thing to fake.
  • Written clarity. Feedback is the deliverable. Reviewers who can't explain a defect in two precise sentences don't last.

Logistics

Fully remote and asynchronous — no standups, no fixed hours. Most contributors work in blocks and commit to a weekly minimum, commonly in the 10–20 hour range, though some projects run heavier during ramp. Pay is hourly and observed between $30 and $60/hr on this category; the actual offer depends on your assessed level, the specific project, and region. Mercor onboards through an AI-led interview plus a work sample, and placement onto a paying project is not guaranteed by passing the screen.