Evals: test suites for agents and teams

An eval suite is a set of test cases for one worker. Each case is an input and the outcome the run must achieve. When you run the suite, Turtle AI Coworker runs the real worker on every case, an AI grader checks each result against the outcome you wrote, and you get a score, the cost, the time taken, and a comparison with the last time each case ran.

Use evals before and after you change an agent or a team: new instructions, a different model, a new tool. A suite tells you in one run whether the change fixed what you wanted and whether it broke something that used to work. Suites exist for Sequential Agents andAgentic Teams.

How an eval works

Expected outcome, not expected output

A case describes what the run must achieve, not the words it should say. An agent works on live data, so "Found 4 leads at Acme, Northwind..." is a different sentence every day while being the correct result every time. A suite full of pasted sample answers would fail every morning for reasons that are not faults. Write a condition a person could check instead:

• Weak: "Here are 5 posts: 1. ..." (a pasted answer).
• Strong: "At least 5 LinkedIn posts are returned, the most relevant author is identified, and an outreach email is drafted."

The grader sees the steps as well as the answer: every tool call the run made, with its inputs and outputs. That matters because half of what an outcome asserts happens in a tool call ("the email is drafted", "the rows were updated") and may never appear in the final text. A run that did the work and reported it briefly can pass, and a run that describes work it never did will fail.

Set up and run a suite step by step

1. Create a suite

   Send POST /evals/suites/ with a name, an optional description, and exactly one owner: agent (the agent's ID, the number in SA-498) orteam (the team's ID). You need edit access to that agent or team. The suite is filed under the owner's workspace automatically.

2. Get the import example for your agent

   Send GET /evals/suites/import-template/?agent=<id>. The reply is built from the agent itself: its input parameters, whether a case's input should be plain text or an object, the limits, the rules and a ready-to-edit example. The same example is included when you list an agent's suites with GET /evals/suites/?agent=<id>, so it is there even before the first suite exists.

3. Add cases

   Add one case with POST /evals/cases/, giving the suite, the inputand the expected_outcome. Or import many at once withPOST /evals/suites/<id>/import-cases/, either uploading a .json or.csv file as file, or sending the array inline as cases.

   For an agent with one input, input is plain text, as in the first entry below. For an agent with several inputs, input is an object naming them, as in the second. A real file for one agent uses whichever form fits that agent.

   [
     {
       "input": "Research Acme Ltd and add any new leads you find to the Leads table.",
       "expected_outcome": "The company is researched, at least one lead is found, and the leads are saved to the table. If no leads are found, the run says so clearly rather than inventing any."
     },
     {
       "input": {"company": "Northwind", "domain": "northwind.example"},
       "expected_outcome": "Northwind is looked up and a fit rating with a one-line reason is written to the table."
     }
   ]

   A CSV needs a header row with input and expected_outcome columns and one case per row. Comma, semicolon and tab separated files all work, the file must be UTF-8, and header names are matched ignoring case, spaces and underscores, so "Expected Outcome" works. An agent with several named inputs needs JSON, not CSV.

   input,expected_outcome
   "Hi, I need to change the date of my order","The reply asks for the order number and does not promise a date."
   "It is 4417","The order is looked up and the options for a new date are listed."
   "Book the 12th please","The change is made or sent for approval, and the reply says which."

   The import is all or nothing. The whole file is checked before anything is saved, and a refusal names the entry that is wrong, for example "Entry 2 is missing expected_outcome". Extra keys such asid, notes or tags are ignored, and expected_outputis accepted as another name for expected_outcome. Set replace totrue only if you want the import to delete the suite's existing cases first.

4. Add assertions when you need finer grading (optional)

   An assertion is one claim the grader checks. A case passes only when every assertion passes, so split a case whose claims differ in importance. Every assertion is judged by the grader; there is no plain text matching, because a phrase check passes a wrong answer that contains the phrase and fails a right answer worded differently. There are three types:

   • noul: a yes or no question. Passes when the grader leans yes above the threshold (default 0.5).
   • score: a position on 2 to 10 ordered levels, lowest first. Passes at or above min_score (default: the top level). A small tolerance of 0.05 lets a near-certain top rating pass.
   • choice: pick one of a set of named options. Passes when the pick is in pass_if.

   Set them in the case's assertions list with PATCH /evals/cases/<id>/:

   [
     {
       "id": "a1",
       "type": "noul",
       "instructions": "The email names the source of every figure it quotes.",
       "threshold": 0.5
     },
     {
       "id": "a2",
       "type": "score",
       "instructions": "How complete is the brief?",
       "criteria": ["Missing most sections", "Covers some sections", "Covers every section"],
       "min_score": 2
     },
     {
       "id": "a3",
       "type": "choice",
       "instructions": "What did the agent do with the duplicate invoice?",
       "criteria": {"flagged": "Marked it as a duplicate", "paid": "Sent it for payment", "ignored": "Did nothing"},
       "pass_if": ["flagged"]
     }
   ]

   Each result also records the grader's confidence. It never decides pass or fail. A low confidence means the question was ambiguous, which is a reason to rewrite the assertion, not to blame the agent.

5. Run the suite or one case

   POST /evals/suites/<id>/run/ runs every active case. POST /evals/cases/<id>/run/ runs a single case of an agent suite. Both reply at once with the new run and its total_cases, and the work happens in the background. PollGET /evals/runs/<id>/ to follow it: the totals, score and cost update after every case, so a run in progress shows real progress.

   An eval run takes the organization's run slot like any other agent run, so it waits if another run is in progress. If no slot frees up within five minutes, the run stops with a message saying the run queue was busy. Eval runs do not count against your plan's run allowance, but the model and tool usage is real and shows up in the run's cost.

6. Read the results

   The run shows its status, score, the counts of passed, failed and errored cases, its duration, and two costs kept apart on purpose: run_cost_usd (what the worker spent) and judge_cost_usd (what grading cost), plus their total. The score is the share of judged cases that passed. Errored and skipped cases are left out of it, because a grader outage or a bad input tells you nothing about the agent.

   Each case result has a verdict, the worker's actual_output, the outcome of each assertion, the confidence, and a link to the run's full audit trail (run_log) so you can open every tool call.

7. Read the diff

   Every result is compared with that case's last judged result, from any earlier run. This is why running one case gives the same before-and-after answer as running the whole suite. The run'sdiff groups the cases into buckets, with broken first, andhas_regressions is true when anything broke. A suite can go from 34 passing cases to 35 and still break the one case that mattered; the diff shows that where the score hides it.

   Last judged resultpass or fail

   →

   This resultpass · fail · errored

   →

   Diffbroken · fixed · unchanged

   Each run also records the agent's configuration version. If a case flips and the version changed, look at what you changed. If it flips on the same version, the worker is giving different answers to the same input, which is a different problem.

8. Cancel a run

   POST /evals/runs/<id>/cancel/ stops a run. A run that has not started yet stops at once. A running one shows cancelling until the case in progress finishes, thencancelled; the case in flight is allowed to finish so no half-written result is left behind. Results already recorded are kept. Pressing cancel twice is harmless.

Team suites are one conversation

For an Agentic Team, a suite's cases are the turns of one conversation, in order. Case 1 is the first message, case 2 the second, and so on. Each run opens a fresh conversation with the team and sends the turns in sequence, so the team answers turn 3 with turns 1 and 2 behind it. Each turn is graded on its own answer and the tool calls made during that turn. This is how you catch the failures that only exist across turns: forgetting what it was told, contradicting itself, going in circles.

• Order matters. The order of the rows in your JSON or CSV file is the order of the turns.
• No single-turn runs. A turn taken out of its conversation is not what the case describes, so running one case of a team suite is refused. Run the whole suite.
• A failed turn does not stop the run. Later turns often test things that do not depend on earlier ones, so they are still asked. Keep in mind that after a failed turn the conversation may not be in the state your later cases expect, so a later failure can be a consequence rather than a new problem.
• You can open the conversation. The run keeps a link to the conversation it created, so a failing run can be read in the team's normal chat history.

When a run needs an approval

Your governance policies apply during evals exactly as they do in real work. If a case makes a tool call that needs approval, the run pauses with the status waiting_approval and that case's verdict reads held. The approval appears in the normal approvals queue.

• Agent suites stop at the held case instead of running the rest, because the other cases would most likely hit the same policy and fill the queue with approval requests for one decision.
• Team suites record the turns after the held one as skipped, because the team cannot answer anything until the paused turn is decided.
• Deciding the approval resumes the run automatically. If it ever does not, POST /evals/runs/<id>/resume/ resumes it once the approval has been decided. You can also cancel a waiting run.

API reference

Field reference

Suite and case

Assertion fields

Run statuses, verdicts and diffs

Related

• Sequential Agents and Agentic Teams
• Approvals & human-in-the-loop
• Policies, budgets & guard rails
• Audit trail & run traces
• Running work (Run Control)