Evaluation skillsHow-to

Prompt Writing and Reference Answers for AI Training

Design realistic tasks with checkable outcomes, then write reference answers that preserve constraints and withstand review.

4 min read · Published September 9, 2026

You'll learn

  1. A good task tests an intended skill rather than accidental ambiguity.
  2. Reference answers need independent checking, even when AI helps draft them.
  3. Define acceptable alternatives instead of treating one wording as mandatory.
  4. Keep private material and held-out evaluation content within project rules.
In this guide
  1. Design from the skill backward
  2. A complete small task
  3. What makes a reference answer usable
  4. Vary the skill demand, not just the names

Writing a task means choosing what success should demonstrate. “Write something about finance” has no clear finish line. “Using the supplied table, explain why the margin changed in two sentences” gives a reviewer an input, an operation and constraints to check.

Design from the skill backward

From skill to checked answer

  1. Choose one main capability

    Decide whether you are testing extraction, calculation, comparison, planning or another specific behavior.

  2. Provide the necessary context

    Include the evidence and assumptions a capable solver needs. Remove accidental dependence on private knowledge.

  3. Define success before drafting

    List required facts, constraints and acceptable uncertainty. Separate essentials from preferred style.

  4. Write the prompt naturally

    Use a plausible user request. Avoid piling on arbitrary constraints just to make it difficult.

  5. Solve and verify

    Produce the reference answer, then independently check its facts, reasoning and compliance.

  6. Test ambiguity and alternatives

    Ask whether another answer could also be correct. Update the criteria or clarify the prompt rather than forcing one wording.

A complete small task

Evaluation example

Prompt
Illustrative task. Fictional inventory: blue notebooks, 18 in stock; green notebooks, 7 in stock. Reorder any color below 10. State which color needs reordering and the evidence, in one sentence.
Response A
Reorder green notebooks because only 7 remain, below the threshold of 10.
Response B
Reorder green and blue notebooks to avoid running out.
Decision
Response A
Why
A follows the threshold and provides the required evidence. B invents a broader stocking objective. A valid reference need not use A’s exact wording; it must identify green, cite its count and apply the stated threshold without adding blue.

What makes a reference answer usable

A reference should demonstrate a correct result at the requested level of detail. It should not contain extra facts merely to look authoritative. If a task asks for a final answer only, keep the published reference concise and store any required checking notes in the project’s designated evidence field.

Distinguish a reference answer from a rubric. The reference shows one valid completion; the rubric explains how to judge other completions. Exact-match grading may be appropriate for a normalized identifier, but it can wrongly reject equivalent prose or a different valid implementation.

Common mistake

Writing a prompt around the answer you already want

If essential assumptions exist only in your head, the task tests guessing. Try solving from the supplied prompt alone. A reviewer who needs your private explanation has found a task-design defect.

Vary the skill demand, not just the names

Changing a customer’s name across many otherwise identical prompts adds little coverage. Vary the reasoning requirement, source structure, missing information or realistic constraint. Keep changes attributable: a task with several unrelated traps may be hard to diagnose when the model fails.

Use only material the project permits. Do not copy confidential client tasks into a public portfolio or use held-out evaluation answers as practice material for the system being tested. If AI assistance is allowed, treat its draft as unverified work and perform the required checks yourself.

Before submitting a task

  • Name the capability being tested.
  • Confirm all necessary information is present or intentionally unavailable.
  • Write criteria that accept equivalent correct answers.
  • Check the reference independently.
  • Remove arbitrary complexity unrelated to the capability.
  • Confirm source rights, confidentiality and tool-use rules.

Key takeaway

A task is ready when a capable solver can succeed for the intended reason and a reviewer can explain why.