Pipeline, steps 8 and 9
Bench
The same invoices run in the same order, with one agent per arm. The verdict comes from the SQL database, never from what the agent says about itself. A pilot, not a study.
Replay of 7 November 2026, read from report and work
python scripts/90_bench.py JZMgB1UItkk --arms none,old-voice --n 8House-rule rows, failures first
Rows where the right answer depends on a house rule the 2021 tutorial never mentions. The task sent to the agent does not state the rule, except in the row 27 run: it predates that change and still gave the VAT rate.
| Row | House rule | 2021 tutorial | Run |
|---|---|---|---|
| 3, Transports Lemaire | two Transports Lemaire: take the one that has a SIREN | success | JZMgB1UItkk-skill-3-1107-195711 |
| 27, Café du Coin | Café du Coin: always VAT 5.5 % | fail | JZMgB1UItkk-skill-27-1107-164022 |
The 2021 tutorial on these rows
| Arm | n | Passed | Partial | Failed | Avg steps | Avg time | Avg cost | Skill loaded |
|---|---|---|---|---|---|---|---|---|
| 2021 tutorial | 3 | 1 | 1 | 1 | 21 | 528 s | 0.13 USD | 3/3 |
Generic data entry, per arm
| Arm | n | Passed | Partial | Failed | Avg steps | Avg time | Avg cost | Skill loaded |
|---|---|---|---|---|---|---|---|---|
| No tutorial | 1 | 1 | 0 | 0 | 23 | 161 s | 0.07 USD | 0/1 |
| 2021 tutorial | 7 | 3 | 1 | 3 | 19 | 386 s | 0.10 USD | 7/7 |
4 runs left out of the counts: the invoice already existed before the run and the agent, told never to create a record twice, created nothing. The database confirms.
Runs
One row of report/bench.csv per H session.
| Run | Arm | Invoice | Verdict | Steps | Time | USD | Skill rev. | What happened |
|---|---|---|---|---|---|---|---|---|
JZMgB1UItkk-skill-1-1107-152836 | 2021 tutorial | 1 | success | 28 | 176 s | 0.10 | 1 | |
JZMgB1UItkk-none-2-1107-153235 | No tutorial | 2 | success | 23 | 161 s | 0.07 | none | |
JZMgB1UItkk-skill-1-1107-154730 | 2021 tutorial | 1 | nothing created | 4 | 683 s | 0.01 | 1 | nothing created by this run: an invoice with this reference already existed and the agent did not duplicate it |
JZMgB1UItkk-skill-1-1107-163738 | 2021 tutorial | 1 | nothing created | 7 | 44 s | 0.03 | 1 | nothing created by this run: an invoice with this reference already existed and the agent did not duplicate it |
JZMgB1UItkk-skill-1-1107-163859 | 2021 tutorial | 1 | nothing created | 2 | 44 s | 0.01 | 1 | nothing created by this run: an invoice with this reference already existed and the agent did not duplicate it |
JZMgB1UItkk-skill-27-1107-164022 | 2021 tutorial | 27 | fail | 10 | 147 s | not recorded | 1 | no invoice with this supplier reference |
JZMgB1UItkk-skill-1-1107-164303 | 2021 tutorial | 1 | nothing created | 11 | 175 s | 0.05 | 1 | nothing created by this run: an invoice with this reference already existed and the agent did not duplicate it |
JZMgB1UItkk-skill-1-1107-192947 | 2021 tutorial | 1 | success | 27 | 137 s | 0.10 | 2 | |
JZMgB1UItkk-skill-3-1107-193206 | 2021 tutorial | 3 | partial | 18 | 1237 s | not recorded | 2 | total before tax is 0.00, still a draft |
JZMgB1UItkk-skill-3-1107-195711 | 2021 tutorial | 3 | success | 35 | 200 s | 0.13 | 2 | |
JZMgB1UItkk-skill-1-1107-212105 | 2021 tutorial | 1 | crash | 0 | 740 s | not recorded | 2 | the session failed before creating anything |
JZMgB1UItkk-skill-13-1107-220000 | 2021 tutorial | 13 | fail | 12 | 63 s | 0.05 | 2 | no invoice with this supplier reference |