Skip to main content

​
Read a run

Each run belongs to one version. It records each step’s inputs, outputs, checks, and changes. On the dashboard, open the run and read the step that went wrong. To give the saved run to your coding agent:
Ask your coding agent to find the smallest wrong instruction, missing input, or wrong rule. It edits the Method file and runs it again. The new run saves a new version.

​
Keep a correction as a case

A case is a recorded run plus a sentence that must be true of the output. Every later version must pass it. Cases are in cases/ beside the Method. After you accept the fixed run, your coding agent records the case:
  • --run is the run that went wrong. The case must fail on it.
  • --passing-run is the run that you accepted after the fix. The case must pass on it.
  • --rubric is a plain sentence. A model judges it, with quotes that must exist in the output.
  • --expect FILE gives exact checks instead, for example {kind: equals, ref: outputs.NAME, value}.
A case must not force the Method to break its own instructions. case new reads the instructions of the model steps and what each step received in the recorded run. It refuses a case that can pass only by breaking an instruction (case_conflicts). Example: the case needs a fact that the inputs do not give, and a prompt says not to invent facts. The error names the rule and the instruction. Decide which one is right. If the case is right, change the instruction first. To keep both, add --accept-conflict 'your answer'. If your complaint does not say what the right result is, your coding agent asks you before it changes the Method or records a case. method case list FILE lists the cases. When a rule no longer applies (for example a policy changed), retire the case. Do not retire a case to make a failing change pass.

​
Test a version

method test replays every case against the current file. Unchanged steps return their recorded outputs. This includes steps that the recorded run reused from an earlier run, so a case fails on the version it was recorded from. Changed steps run. files connections become scratch folders, so a changed step that writes files is safe. A changed step that asks a person or changes a service makes the case unverifiable. --baseline also runs each case on the old file, to show what the change fixed or broke. method publish runs the cases too and refuses a version that breaks an approved case. See Publish and versions.

​
Checks

A step needs no extra check by default: every output is already type checked. Add a check only for a concrete failure that matters to the result. See Checks and recovery.