-
421 verified items
360 findings and 61 strengths, each reproduced before it reached the report.
-
97 blind trials
over five rounds, with more runs in progress.
-
11 models
from six model families, plus small models running locally.
-
4 environments
macOS, Linux, Windows 11 and Windows with WSL2, each from a clean account.
1. Check what is really built
Docs describe what a team intends. Code describes what exists. We read the whitepaper and every public docs page, then compared each claim with the public repositories, in three tracks at once: asset security, the compute service, and the user-facing products and infrastructure.
Then we ran the software ourselves. We synced a full testnet node with the official node manager, checked the release checksum and code signature, and built the validator from source.
2. Walk the docs by hand, word for word
We used only the public internet and the public testnet, with no insider access. We followed the docs literally first. Every time we had to leave the documented path, we logged it as a workaround used: guessing a parameter, reading source because the docs were silent, or pinning a different package version. Nothing was fixed quietly, because a quiet fix is a finding the team never sees.
The walkthrough covered:
- the docs, including the AI-facing
llms.txtfiles - the SDKs on npm, crates.io and PyPI
- the EVM and Substrate RPCs, the faucet and the block explorers
- the wallet and swap web apps
- compute jobs, and node and miner setup
Checks included the chain identity and chain-spec checksum, a full faucet, transfer, contract deploy and read-back flow, the wallet's network parameters, the docs' claims about browser access, and deliberately wrong transactions to see how helpful the error messages are. We snapshotted the docs at a fixed commit and tracked every change during the audit, so each finding is tied to a point in time.
3. Push the same journeys to breaking point
One person can walk a journey once, on one machine. To stress the docs we ran the same journeys dozens of times as blind build trials: a coding agent got a real developer's brief and nothing else, and logged every moment a developer would be confused, misled or stuck. More developers now build through an agent anyway, so this also shows whether yours can ship from the public docs alone.
Agents that see like people. Some agents could only see the web as
screenshots: no llms.txt, no raw HTML, no machine-readable shortcuts. They
found the project through ordinary web search and used a test wallet whose pop-ups
appeared the way a person sees them. For contrast, an agent-first developer built from
llms.txt and llms-full.txt alone. Running both shows how
well the docs serve each audience.
Clean machines. Each run started from a fresh user account with nothing preinstalled beyond the basics. If the docs forget a prerequisite, the agent hits it the way a newcomer would. The WSL2 runs repeat the Windows briefs, so Windows problems can be told apart from WSL problems. Nodes ran on Docker Desktop and native Linux Docker, and we used two agent harnesses so no finding is an artifact of one tool.
Eight developers. The personas included:
- a frontend developer adding post-quantum signing to a Next.js app
- a Solidity developer deploying a token
- a researcher submitting a compute job
- a hackathon builder with one hour
- a hobbyist running a node and a miner
- a first-time Rust developer
- a Python developer using
piponly - an Ethereum developer porting an existing app
We also paired agents as maker and taker to complete a token swap with each other, talking only through a shared mailbox, like two strangers.
Eleven models. Claude Opus 5.5, Claude Sonnet 5 and 5.5, GPT-6.1, GPT-6, GPT-5.6, GLM-5.2, MiniMax-M3, MiMo-V2.5, and DeepSeek V4 Flash and Pro. Small local models test whether the docs work for a lighter, offline agent too, not only for frontier models.
Safety. Agents worked on testnets with throwaway keys. They could view mainnet but never sign on it. We ran the mainnet check ourselves and approved every signature by hand.
4. Every finding has to reproduce
An unverified finding costs the team an afternoon chasing something that is not there. So nothing reaches the report on one reviewer's word, human or model. The trials widen the search. People decide what goes in.
- Each finding is reproduced independently by reviewers from at least two model families. Disputed items are rechecked by hand.
- Claims that do not reproduce are dropped. Partly true claims are narrowed to what reproduced.
- Each finding links its evidence: the docs sentence, a source line pinned to a commit, a package file or an explorer transaction.
- Projects the agents built are rebuilt from a clean copy, and every transaction, address and job ID they report is checked against the live chain.
- Anything the team fixes during the audit is marked as fixed.
- Before delivery, two models from different families re-review every claim, followed by a freshness check against the live docs and services.
5. What the team receives
- An executive summary and the five biggest blockers
- The top ten fixes, ranked by developer impact against effort, with estimates the team can re-rank
- What works well, reported next to what does not
- Findings by developer journey, from discovery and docs to nodes and miners
- The workarounds developers were forced into, and what the trial developers actually built
- A docs-merge checklist, and a findings list for each owning team
- Limitations and open questions, stated plainly
- Machine-readable findings in JSON and CSV
6. From findings to fixes
A report that only complains gets read once. We wrote suggested corrections and opened draft merge requests against the team's repositories, starting with the node setup docs. A human engineer reviewed them, two models from different families cross-checked them, the agreed edits went in, and we ran a fresh review of the full docs.
When this is worth doing
A developer who gets stuck in the first hour rarely files an issue. They leave, and you never learn why. Stress test the docs before a hackathon, an SDK release, a docs consolidation or a push for integrators: anywhere adoption depends on developers you will never meet.