AI Agent Evaluator

$25–45/hr

rate as listed

Workload: ~15h/week
Engagement: Project-based
Location: US
Field: Generalist

Test two AI agents side by side on everyday errands, recording each attempt and evaluating completion, quality, control, satisfaction, and time. This U.S.-based contractor role uses your own phone and eligible personal AI account, with no coding or AI development background required.

What the work involves

You will run the same kinds of errands people may delegate to an AI agent, including bills, returns, email follow-ups, bookings, shopping, groceries, travel, family logistics, outings, and forms. Each run involves testing two agents, approving or stopping key actions, screen recording the attempts, and documenting what happened.

How the work is evaluated

You will score completion, output quality, control issues, user satisfaction, and time, then provide concise written evidence for those judgments. Screening is likely to focus on your daily use of AI, your ability to recall exact steps and outcomes, your judgment about when an agent should be stopped, and whether your scores remain consistent with the recorded evidence.

Practical requirements

The contract is remote, U.S.-based, and performed from a smartphone using your own accounts. You need an active personal account for a popular AI service on a Free, Go, or Plus plan, not Pro, and a smartphone capable of screen recording. Runs take about 1.5 hours including evidence and scoring, and the work is scheduled flexibly.

Timing and pay

Pay band: $25–45/hr

The listing describes onboarding on Friday, Saturday, or Sunday and a Monday kickoff, but it gives conflicting calendar dates for that kickoff. Confirm the actual onboarding and start dates with Talent Ops before committing availability.

What they look for

ChatGPT
Muse
AI agents
Attention to Detail
Following Instructions
Screen Recording

Included with this listing

Interview prep — 10 questions

Matched to what this platform's screening actually measures. Answer each out loud in 60–90 seconds: claim → concrete example → implication.

What it tests: This tests whether your professional history contains verifiable process ownership relevant to recording agent actions and defending evaluation scores.
A strong answer: Give a concise timeline with employer or project context, your specific responsibilities, and one example of documentation that another person relied on. Distinguish your own work from the work of the wider team.
Where people lose it: Candidates lose points by naming broad duties such as operations or project support without identifying what they personally tracked, checked, or produced.

Summary, eligibility guidance and screening preparation are prepared by Sidequest from the platform's posting. Check the source listing before applying.