The fleet test

With FailEcho vs without

Same tasks, same models, same address. One twin asks the network before it retries and does what it says; the other retries by fixed rules. Every number is a live average over the window stated just above the table, from the same ledger as the tables below; bold is the better side, blank when they are within 2%. Our own agents, in a lab: a measurement of the mechanism, not adoption.

How a run is graded, exactly. A model run counts as completed when it returned an answer rather than a failure marker; a builder or OpenCode run when the file it was asked for exists and matches the task's pattern. That is weaker than "the answer was correct": a wrong but well-formed answer counts, and a pattern can match a partly right file. Read the completion rows as runs that produced an answer, and the recovery and retry rows, which are counted per call, as the harder evidence. An outside review on 20 Sep made this point, and since 22 Sep a grader checks the answers themselves: "every checked value correct" compares each value with the truth fetched from the same API, "values declined rather than invented" counts an agent saying it could not get a value, and a run with no truth to check against is left out rather than guessed.

What these tables cover. Loading…

Askers vs blind

Attempts each cohort needed to get past a failure, on the same workloads. Lower is better. A tie means the network had nothing to say when asked. Skipped on advice is the network saying nothing works and the asker not spending the attempt (since 2026-09-17 04:40 UTC).

Latest builds

Real services, real limits: ask vs blind

The proof table. Only real services (GitHub API, PyPI, npm, crates.io, Stack Exchange) and only the controlled twins — no test endpoints, no model providers, no explorers. Each failure is a documented policy being enforced on agents behind one address: GitHub's 403 at 60 an hour, crates.io's 429 above one a second, Stack Exchange's 300 a day. The asking side spends what the network says is worth spending — a reset wait, a conditional request that a 304 does not count against the limit — and skips the rest. Counted from 2026-09-17 06:30 UTC, when the twins started trading places each cycle: before that every ask twin ran first and its blind twin met the rate limits it had used up. Even so, the twins share one address and one hourly budget, so how many failures each meets is not a fair comparison — whoever runs second meets more. What is fair is what each does once it has failed: attempts per failure, recovered, skipped, seconds.

OpenCode: a real agent, with FailEcho and without

Since 2026-09-18 12:30 UTC two personas drive OpenCode itself, headless, inside the throwaway VM, on the same model (NVIDIA's nemotron-3-super): the ask twin has FailEcho's MCP server in its opencode.json and one paragraph in AGENTS.md saying what it is for; the blind twin has neither. Same task, same project, everything else is OpenCode's own behaviour. Nothing asks or reports on the agent's behalf. Graded on the file the task asks for; its rows are in the table above.

What the advice is worth, by failure shape

Once a call has failed, what happened to the second attempt: when the network recommended it (advised), when the asker had nothing from the network and did what blind does (no advice), and when blind retried. The last column is the cost of a skip: in the same hour, on the same shape the network told askers to skip, how many of blind's retries recovered. Since 2026-09-17 06:30 UTC, twins only, shapes with at least eight events.

Does it get better with evidence?

Per half day, model-driven and cron twins: the share of an asker's failures the network had a recommendation for, what each side completed, and seconds lost inside failures per run. Twins share one address and GitHub's hourly budget, so completion moves with who met the 403s; the advised share is the network's own curve.

When the model provider fails

Groq, Gemini, OpenRouter, NVIDIA, Mistral, xKiro and Ollama fail for real every day: 429s, 503s, timeouts, a 400 when a model emits broken tool JSON. Blind personas give up. Askers do what the network recommends. Two explorers try an action nobody has evidence for yet — wait for the reset header, switch model, drop tool_choice — and report the outcome, which is how the evidence askers inherit gets made. Recovered means the run went on.

Cost and outcome per run: ask vs blind

Everything a run pays and whether it got the job done, by cohort, so a change to the product can be judged on all of it and not only on the number it moved. Completed: a real answer, every call eventually succeeding, or the build done. Tokens: what the provider billed, prompt plus completion. Ask s: time spent asking the network — the product's own overhead. Wait s: time slept on backoff or a reset. Recorded since 2026-09-17 10:30 UTC. Definitions in docs/fleet-metrics.md.

Outcome

Cost per run

Builders: whose fault was it?

Two agents write small programs and run them in a sandbox VM, fix them, run again. A local failure is their own bug — counted, never reported. A shared failure is the world's: the index timing out, GitHub's rate limit, a 503. The last column is the share of an agent's failures a shared network could ever speak to.

Coverage of the zero-code wrapper on code nobody wrote for it: for every program the builders run, the fence counts the connections the program opened and the wrapper says how many calls it observed. A run with traffic and nothing observed is an HTTP client the wrapper does not patch.

Does it repeat?

Fingerprints seen by more than one reporter. This is the premise; zero here after two days is the honest bad result.

Does the naming hold?

The same service and operation reported through three paths. More than one row per real failure means the join is broken in practice.

Onboarding from llms.txt

Every hour a free model gets a clean VM, a shell, and one sentence: Read llms.txt and set yourself up to use FailEcho. No hints. What it finds on disk is one of five scenes the document has a rule for: a clean project (write the config, verify with one query); a project with another server already configured (add, do not clobber); one where FailEcho is already there (stop, say so); a home directory full of projects (ask, do not write); a project that cannot be written (use the REST query, say so). Graded from the disk, not by a model. The failing column is the line of llms.txt to rewrite next.

Install canary

Once a day a fresh VM with nothing of ours on it installs each package from PyPI and does what the setup page says to do: import it, ask the network, run a script under it, start the stdio relay and complete an MCP handshake. The day a path stops working is the day this table says so.

Per persona