Doc-Coverage Tests: Treating Agent-Facing Docs as a Tested Capability Contract

Executive Summary

An agent relying on a supplied, bounded documentation surface — a SKILL.md, CLI reference, or MCP tool manifest — may miss a capability that exists only in the code. Agents can also run --help, inspect source, or retrieve more references when permitted; the risk here concerns workflows that rely on the supplied surface without that exploration. A missing entry can lead to an unnecessary workaround or a mistaken claim that a feature is unavailable.

This is the opposite of how documentation drift has traditionally been treated — as a hygiene issue, a stale-docs complaint, something a reviewer should have caught. For agents relying on those docs, drift can hide a capability, so a mechanical gate is useful alongside review. The fix looks like a coverage test, not a style guide: enumerate the tool's actual capabilities from its source of truth (a command registry, a route table, an MCP tool list) and assert, in CI, that every one of them appears in the docs the agent actually reads. This article walks through a real incident that forced this realization, surveys the prior art in docs-as-code and documentation testing that doc-coverage tests build on, examines what's specific to the agent-consumption case, and lays out a concrete, minimal pattern any team building agent-facing tools can adopt.

The Incident: An Invisible Feature, Twice

The grounding case is unremarkable in the way that matters most — it's the kind of gap that happens constantly and rarely gets caught. A component that renders documents as web pages had a share-link feature in its web UI: publish a page, optionally protect it with a password, rotate that password later, remove it. The web UI supported all four operations. The CLI — the interface an agent actually drives when asked to publish or manage pages — did not. There was no enable-password, rotate-password, or disable-password subcommand.

An agent asked to password-protect a shared link did what any reasonable agent does when a CLI doesn't expose something: it looked for another way in. It found the component's internal modules, imported them directly, and wrote a one-off script that called internals never meant to be a stable interface. It worked, technically. It also meant the agent was now coupled to implementation details that could change on any refactor, bypassing whatever validation or side effects the CLI layer was supposed to provide.

The natural fix was to add the missing subcommands to the CLI — which happened. Then something more interesting happened in review: the first round of review caught that the CLI got its new subcommands, but the agent-facing docs — the SKILL.md describing the component's capabilities and the CLI reference doc — were not updated to mention them. The subcommands existed. The tests passed. And the feature was, for an agent relying on those docs alone to decide what the CLI could do, still absent from its supplied reference. The exact failure mode that motivated writing the feature in the first place would have quietly recurred, just one layer up: the CLI now had the capability, but the discovery surface agents actually use still said it didn't.

The fix that came out of that review round wasn't "remember to update the docs next time." It was a script: enumerate every subcommand the CLI actually registers, and assert that each one has a corresponding mention in the SKILL.md and CLI reference doc. Wire that into CI so a PR that adds a subcommand without touching the docs fails, not merges.

That test, run for the first time against the existing codebase, immediately found a second, unrelated gap: a keyring-related command that had been in the CLI for a while and had never been documented anywhere an agent would see it. Nobody had shipped that gap in the PR under review — it had been sitting there, leaving a real command absent from the checked agent-facing docs. That's the strongest evidence for treating this as a systemic problem rather than a one-off review miss: a small, mechanical coverage test found two independent instances of the same failure mode on its first run, one of which predated the incident that motivated writing the test.

The lesson generalizes cleanly: "docs were updated" needs to be a property CI can check, not a property a reviewer needs to remember to check. And for tools consumed primarily by agents, the cost of getting this wrong isn't a slightly-stale README — it's a capability a consumer relying on that surface may never discover, despite its existence in source code.

Prior Art: Documentation Has Been Tested Before

None of this is conceptually new — the software industry has been fighting doc-code drift for decades. What's new is who reads the docs and how unforgiving that reader is about gaps. It's worth surveying the existing toolbox, because doc-coverage tests for agents are best understood as one more point on a spectrum that already has well-understood instruments.

Doctests: examples that must keep working

The oldest and most direct answer to "how do I know my docs are still true" is to make the examples in the docs executable. Python's doctest module scans docstrings for interactive-shell-style examples and re-runs them, failing if the output no longer matches. Rust does the same thing at the language level: code fences in /// doc comments are compiled and run as part of cargo test, specifically to help keep documentation up to date with code. Elixir's ExUnit.DocTest extracts iex> examples from @doc attributes and runs them as real tests. The shared insight: an example that isn't executed isn't verified, and unverified examples rot at the same rate as any other unmaintained code — quietly and without an error message, until someone tries to actually run one.

Doctests are precise but narrow: they verify that a specific worked example still behaves as claimed. They say nothing about whether the docs are complete — whether every public function has an example at all. That completeness question is what coverage-style tools try to answer instead.

CLI snapshot and command-tree testing

For command-line tools specifically, a cluster of tools test the shape of CLI output rather than executing narrative examples. cram is a language-agnostic end-to-end tool that runs shell commands embedded in a test file and diffs actual output against expected output. Rust's trycmd (and its sibling snapbox) does the same for CLI binaries and is explicitly designed to enumerate large batches of test cases rather than hand-write assertions one at a time — real crates like clap itself, typos, and cargo-edit use it to verify that the examples in their own README stay runnable, as documented in the clap project's own discussion of the pattern. Simon Willison's cog-based workflow for Datasette and sqlite-utils takes a related but distinct approach: rather than snapshot-testing help output, it regenerates the --help block inside the README from the live CLI and fails CI (cog --check) if the committed README doesn't match what regeneration would produce. mdsh generalizes this to arbitrary shell output embedded anywhere in a markdown file.

These tools can catch new capabilities too: a root --help snapshot that lists public commands changes when a new command is added, even without a dedicated per-command block. Cog can regenerate that list or enumerate a registry directly. The blind spot is an incomplete selection of snapshots or examples, not snapshot testing itself. A separate coverage assertion is useful when the obligation is more specific: each public command must have an entry in the skill instructions or reference set, beyond its presence in a generated root list.

Generate-from-source: synchronizing reference metadata

A different strategy reduces duplication: generate reference docs from the tool's own metadata. Go's Cobra doc package walks the command tree and emits Markdown, man pages, or ReST from command definitions, including Use, Short, Long, Example, and flags. Its LLM-ready CLI documentation guide recommends meaningful what/why descriptions and worked examples. terraform-docs similarly reads .tf files to generate variable/output tables; its repository documents CI integration for keeping committed output synchronized.

Generation synchronizes the fields and files it covers when regeneration is enforced. It can preserve authored narrative: Cobra emits Long and Example, so a generated reference need not be merely a flag list. Sparse metadata or restrictive templates produce sparse docs; that is a content-design limitation, not an inherent property of generation. A curated SKILL.md can add cross-command workflow guidance or link to generated references. Neither generation nor metadata synchronization proves that the implementation behaves as described.

API-contract drift detection

At the API layer, a parallel discipline exists around OpenAPI specs. oasdiff diffs two versions of an OpenAPI spec and classifies each change as breaking or non-breaking against a documented rule set of over 500 distinct change types, and teams run it as a GitHub Action that posts a commit status on every PR. (Optic’s official repository is archived; that establishes its maintenance status, not a ranking of other tools.) These tools check spec-to-spec drift, not spec-to-implementation drift, but the CI pattern — diff against a source of truth, fail the build on unreviewed change — is identical to what a doc-coverage test needs to do against a command registry.

Coverage-as-a-lint: the closest existing analogue

The closest existing pattern to "fail CI if a public capability lacks documentation" is language-level missing-docs linting. Rust's missing_docs lint flags any public item without a doc comment; it ships allow-by-default but the Rustdoc book documents it as intended for library authors to deny explicitly via #![deny(missing_docs)] — a policy several practitioners describe adopting specifically because, past a certain project size, "it is much easier to maintain documentation coverage than to backfill it later." Kubernetes provides a related single-source example, with different guarantees. Its API conventions recommend explicit +optional or +required tags for new fields while preserving legacy optionality for omitempty fields without +required. Its v1.34.0 OpenAPI generator derives descriptions from comments, filtering marker/TODO lines, stopping at ---, normalizing whitespace, and potentially appending enum descriptions. This reuses authored comments; it neither guarantees byte-identical human and machine text nor establishes universal documentation coverage.

The missing-docs lint is the closer coverage analogue: enumerate public items and enforce a documentation obligation. Kubernetes comment-to-schema generation illustrates source reuse, a separate property. Doc-coverage tests for agent-facing docs are this pattern, ported from language-level public items to CLI-level subcommands (and, as the next section covers, to MCP tool manifests).

Review-time enforcement: Danger and its limits

One more layer worth naming because it's the default most teams reach for first: PR-time checklist bots. Danger runs arbitrary review rules during CI — flag a PR if the description is too short, if it touches a sensitive path without a specific reviewer, and so on — and teams commonly configure a rule like "if cli/commands/ changed, warn unless docs/ also changed." This catches the should-touch-docs case cheaply, but it's necessarily a blunt proxy: it can tell that some file in docs/ changed, not that the specific new subcommand is mentioned anywhere in it. A PR that edits an unrelated typo in the docs directory satisfies a Danger rule like this while shipping the exact silent gap from the incident above. Danger-style rules are a reasonable first line of defense and a poor substitute for an actual coverage assertion.

The Agent-Specific Angle: Docs Are Now the API

Everything above predates the current wave of LLM agents, and all of it still applies. What's changed is the reader. Historically, a human hitting a doc gap had recourse: skim the source, run --help, ask a colleague, infer from adjacent examples. In workflows scoped to supplied documentation, agents may not do any of that — not because they're incapable of it, but because the entire design point of structured agent-facing docs (MCP tool descriptions, SKILL.md files, llms.txt) is to give the agent a complete, bounded, cheap-to-load description of capabilities so that it doesn't have to go exploring. That's a feature when the description is accurate and a trap when it isn't.

MCP makes this documentation surface explicit, but prose is only part of it. The 2025-11-25 tools contract exposes a tool's name and inputSchema as well as an optional description. Names, parameter schemas, and prose can all inform selection and invocation; description is neither the whole API nor the only signal. FastMCP can derive description text from Python docstrings, reducing duplication for that text. This does not ensure every tool has useful prose, nor prove schema/implementation agreement.

Where MCP tool docs drift anyway, the consequences are qualitatively different from a stale README. A widely-cited account describes an MCP search tool whose parameter was renamed from query to search_query; the server kept silently accepting the old field name, defaulting to an empty string instead of erroring, and the agent "confidently explained why the search found nothing" rather than surfacing a broken contract — a failure mode the author frames as worse than REST versioning precisely because an LLM rationalizes a bad response instead of raising an exception. Tooling has started to appear specifically for this: Specmatic's MCP Auto-Test generates test calls directly from a server's declared tool schema and runs them in CI, and its own walkthrough first encounters a missing collection requirement on Postman's update_mock tool during manual MCP Inspector testing. The later automated test reproduces that same mismatch. The demonstrated benefit is repeatable local and CI regression checking, not discovery of a defect that the manual walkthrough missed.

There's a security dimension layered on top of the pure-drift problem, worth naming because it inverts the incentive: the original tool-poisoning-attack research from Invariant Labs showed that MCP's trust model assumes tool descriptions are benign, and demonstrated hidden instructions embedded in a description field that are invisible in the UI a human reviews but fully visible — and actionable — to the model consuming them. OWASP catalogs this as its own attack class, including a "rug pull" variant where a server changes an already-approved tool's description after the fact. A doc-coverage test guards against incompleteness — capabilities that exist but aren't described. Tool-poisoning defenses guard against the opposite failure mode — descriptions that exist but lie, or change underneath an agent that already trusted them, defended by pinning tool definitions and hashing them for integrity checks rather than by coverage enumeration. Both problems live at the same interface — the text an agent trusts as its map of what a tool can do — and a mature agent platform needs answers to both, but they're not the same test.

Standards for making documentation itself agent-legible have also emerged fast. llms.txt, proposed by Jeremy Howard in September 2024, gives a site a root-level markdown index specifically scoped for LLM context windows, distinct from robots.txt's crawler-permission scope. Anthropic's own Agent Skills specification formalizes SKILL.md as YAML frontmatter (name, description) plus a markdown body, explicitly loaded progressively — skill metadata (name and description) is loaded at startup, and the full body only once a skill is activated. That progressive-disclosure design is efficient, but it raises the stakes on the description field specifically: if a capability isn't reachable through that first ~100-token summary, the agent may never load the body that would have described it correctly, no matter how good that body is. None of these specs, as written, mandate a coverage check against the underlying tool's actual command or route list — that enforcement is left entirely to the implementer, which is exactly the gap doc-coverage tests are meant to fill.

Patterns and Trade-offs

These strategies check different obligations; they are not a single ladder from weak prose to strong generated output:

Review checklists / Danger rules such as "did you touch docs/?" check a cheap proxy. An unrelated documentation edit satisfies that rule without describing the new command; a custom Danger script could instead implement a real coverage assertion.
Snapshot/doctest-style testing (trycmd, cog, doctest, ExUnit.DocTest) checks selected output or examples. A root command-list snapshot can catch a new public command. Incomplete selections can miss it, and a root list alone does not establish coverage in separately authored skill guidance.
Generate-from-source (Cobra, terraform-docs, FastMCP docstrings) synchronizes covered metadata and output when CI enforces regeneration. Authored narrative and examples can be included; their usefulness depends on metadata and templates. This checks synchronization, not runtime correctness.
Coverage assertion checks that each capability in a declared public set has the required documentation entry. A cheap name-presence check can flag omissions, including the incident's pre-existing gap, but does not prove accuracy or discoverability.
Spec-to-spec compatibility diffing (oasdiff) compares two OpenAPI specifications and classifies their changes. It cannot detect runtime drift when the schema remains unchanged.
Runtime contract testing (Specmatic's MCP schema tests or targeted calls) compares declared expectations with actual behavior. A tool declaring a parameter optional while requiring it at runtime is a different failure from a missing documentation entry.

No single layer subsumes the others. A name-presence check would not catch the update_mock schema mismatch; runtime testing does not by itself establish that a separate skill guide mentions every CLI command. Choose layers according to the failure and cost: inexpensive presence checks are a useful starting point, while runtime contract tests require additional fixtures and execution.

Concrete Recommendations for Agent Platform Builders

A minimal, high-leverage doc-coverage test looks like this, and is worth building before reaching for anything heavier:

Identify one source of truth per surface. For a CLI, that's the command-registration code itself (cobra.Command tree, argparse subparsers, click group) — not a hand-maintained list. For an MCP server, it's the live tools/list response or the decorator-registered tool set. If the source of truth requires a second hand-maintained list to describe it, the drift problem has just moved one level, not disappeared.
Enumerate mechanically. Where available, walk the registry in-process (for example, recursively using Cobra’s Commands()) to obtain full public command paths. A root help snapshot remains useful for checking the user-visible list; extracting a separate coverage inventory from rendered help adds parsing assumptions to maintain.
Define the checked set and obligation. A minimal grep checks only that each declared public command/tool name occurs in the selected docs. Use full command paths to distinguish shared short names. For a stronger entry check, require a corresponding reference heading or structured entry and validate links from the agent's actual entrypoint; for skills this may traverse metadata, activated instructions, and on-demand references. Keep prose quality as a separate review obligation.
Run on every PR, or filter on changes to either the registry, checked documentation, or the coverage implementation/configuration/inputs. A docs-only deletion of the last entry can break coverage just as a new command can. Registry-only triggers miss that case.
Layer, don't choose. Pair coverage with generate-or-check steps for generated reference material. Add spec-to-spec compatibility checks and runtime contract tests where those distinct failure modes matter.
State the narrow guarantee. A green grep proves name presence for the declared set in the checked files. A negative statement, obsolete example, shared short name, or unreachable reference can still satisfy it. Structured full-path entries and verified entrypoint links strengthen coverage and reachability, but neither establishes that an agent will select the capability, that the prose is correct, or that runtime behavior matches it. Use editorial review, runnable examples, and agent discovery exercises for those further questions.

Conclusion

For agents relying on a bounded documentation surface, a missing entry can lead to an unnecessary workaround or a mistaken capability judgment. The incident motivated a cheap registry-to-docs presence check, which found both the new omission and an older gap. That is a useful, limited result: coverage of a declared set can be enforced in CI without relying solely on reviewer memory. Keep its guarantee explicit, run it when either side changes, and add entrypoint/link checks or behavioral tests when the goal extends to discoverability or correctness.