Test-Double Protocol Drift: Keeping E2E Drivers Honest Against an Evolving Wire Protocol

Executive Summary

Every team that tests a WebSocket or event-driven service without driving the real client through a real browser ends up writing a driver: a piece of test code that speaks the wire protocol directly, impersonating the client. The driver is a test double — not of a dependency, but of your own client — and like every test double, it is a second implementation of a contract that only one implementation is forced to keep. The client evolves with the product; the driver evolves only when someone remembers it exists. Nothing in a typical CI setup detects the gap. The suite stays green while the thing it simulates drifts further from the thing users actually run.

This article works from a concrete incident in an internal conversational agent system (text plus realtime voice over WebSocket). The e2e drivers had drifted from the live protocol in three independent ways: they omitted the protocol-version query parameter entirely, so a strict server would have rejected them at the door; they waited for an event type that had been removed from the protocol and replaced by incremental progress patches; and — worst — they omitted a newly added field on both the driver side and the server's own test fixtures, so the suite validated two wrong implementations against each other and passed, while the same flow against the real client reproduced a production bug 100% of the time. Verifying the eventual fix required reverse-engineering a protocol-exact driver from the production client's source, because no trustworthy driver existed.

The research behind this piece asked six questions: what contract testing offers for bidirectional protocols; whether drivers can be derived from production client code; what can detect drift; how the literature treats mutually consistent but wrong tests; when replay can substitute for a real client; and what voice testing adds. The useful positive evidence is shared contract testing of real implementations and their doubles, explicit protocol versions, and comparisons with real-client behavior. The selected sources below did not establish a turnkey driver-versus-client drift detector for ad hoc JSON-over-WebSocket; that is a limit of this survey, not evidence that no tool or vendor practice exists. Capture can inform a driver, but replay requires a separately verified implementation.

For small teams, the ranked takeaways are concrete: put a protocol version on the wire and make drivers fail loudly on mismatch; run one shared contract suite against both the driver and the real implementation; make protocol owners own the drivers; and compare driver behavior with fresh real-client captures as a staleness check. Each is cheap relative to the failure it prevents.

1. The Failure Class: A Driver Is an Unverified Test Double of Your Own Client

The incident that motivated this research is worth describing precisely, because each of its three defects is a distinct mechanism of the same disease.

Defect 1 — missing version handshake. The live protocol required clients to declare a protocol version as a connection parameter; current servers rejected unversioned connections with an explicit upgrade_required error. The drivers predated the versioning scheme and never sent the parameter. This is the best possible variant of drift, because it fails loudly and immediately — the driver cannot even connect, so nobody mistakes its results for coverage. LiveKit provides a precedent for putting a numbered protocol version on the wire as a connection query parameter, but not for this strict rejection policy. Its pinned parser leaves missing or unparseable protocol values at zero, and its capability checks use version thresholds to support older clients. Strict rejection is the policy proposed here: incompatible changes must increment the protocol version, and both server and driver must enforce compatibility checks before proceeding. This catches declared incompatibility; a driver can still drift while sending an accepted version.

Defect 2 — waiting on a removed event. The drivers awaited a discrete mode-switch event that the protocol had since replaced with a stream of incremental progress patches. A driver waiting on an event that will never arrive either hangs until a timeout or — more insidiously — passes because the assertion was written as "no error before timeout." Event-shape changes in unversioned protocols leave no artifact a test can trip over; the driver's model of the conversation state machine is simply wrong, and nothing says so.

Defect 3 — the green suite encoding the bug. A field (call it generation, used to invalidate stale responses after a mode switch) had been added to the protocol. The driver didn't send it — and neither did the server's own test fixtures. Both sides of the test boundary omitted the same field, so the suite was internally consistent and green, while the real client, which did send the field, hit a 100%-reproducible bug that the tests structurally could not see. This is the deepest failure: the test double and the system under test had co-evolved away from production. The suite wasn't stale — it was coherently wrong.

The aftermath is the tell for how expensive this class is: to verify the fix against the live deployment, a new protocol-exact driver had to be reverse-engineered from the production client's source code, message by message. That cost — rebuilding the double from the real implementation under incident pressure — is exactly what the patterns in section 5 are designed to amortize into routine maintenance.

Naming matters here. In Google's taxonomy (Software Engineering at Google, ch. 13), this is fidelity decay of an unverified fake: "A fake without tests… can diverge over time as the real implementation evolves." The chapter's ownership rule states the organizational root cause plainly: "the team that owns the real implementation should write and maintain a fake." Our drivers were owned by the test side. That is the named anti-pattern, not an incidental detail.

2. Contract Testing for Bidirectional Protocols: Mature for REST, Young for Everything Else

The obvious industry answer to "two implementations of one contract drifted apart" is contract testing. For REST it is a solved category. For WebSockets and event-driven systems, the landscape splits into two camps, neither of which fully covers ad hoc JSON-over-WS.

Camp 1: consumer-driven contracts. Pact's message pacts extend the consumer-driven contract model to asynchronous messages, and a WebSocket mode exists. The model is attractive in principle — the consumer records what it actually sends and expects, the provider verifies against those recordings in CI — but the async support is thinner than the HTTP lineage: broker delivery semantics and message ordering are explicitly out of scope, and the gRPC/protobuf plugin's published examples are unary-only despite streaming existing in the protocol. Practitioner skepticism is on the record: for many-to-many Kafka topologies, consumer-driven contracts are argued to be premature when schema-registry compatibility rules already cover the syntactic part of the problem.

Camp 2: schema registries as compatibility gates. The most operationally proven mechanism in this space is not a testing tool at all — it is the schema registry (Confluent, AWS Glue, Azure). Netflix mandates Avro schema registration for Kafka publishes; the registry rejects incompatible schema evolutions at publish time. LEGO ships event schemas as NPM packages consumed by both sides — with the honest caveat, documented in Yan Cui's survey of the space, that without enforcement nothing guarantees publishers actually run consumer tests before merging (a gap that maps exactly onto our incident: shared artifacts help only if both sides are forced through them). PostNL layers a registry with runtime validation and dead-lettering, accepting runtime cost to catch what static compatibility checks miss.

The structural gap. Every diff-based gate in this space — AsyncAPI Diff, GraphQL Inspector, oasdiff, buf breaking, Azure's openapi-diff — requires a formal schema to diff. Specmatic's AsyncAPI WebSocket support (new as of December 2025) and Microcks (the most mature multi-protocol mock-and-test tool driven from AsyncAPI definitions — WebSocket, Kafka, MQTT, AMQP, NATS) both assume you have an AsyncAPI document that is itself kept true. An ad hoc JSON-over-WS protocol — which is what most internal realtime products actually run — has no schema to register, no artifact to diff, and therefore no seat at any of these tables until someone writes the schema. That schema-fying step is a real cost, but it is also the toll gate to the entire mature tooling ecosystem.

One caveat applies across both camps, stated candidly even by vendors in the space (Specmatic among them): schema validation is not behavior validation. Ordering, side effects, and state-machine transitions — precisely the things defects 2 and 3 above lived in — are not covered by any schema compatibility check.

3. Deriving Drivers from Production Client Code

If dual implementation is the disease, the structural cure is to make the driver derived from — or identical to — the production client. Four patterns exist, at sharply different costs.

Schema-first codegen. Where the protocol has a schema (protobuf via buf, OpenAPI-generated SDKs, GraphQL codegen, AsyncAPI codegen), both client and driver are generated from one source, and drift between them is structurally impossible for message shapes (behavior can still drift). This is the right answer for new protocols; it is not retrofittable to an ad hoc protocol without the schema-fying toll mentioned above.

Drive the real client headlessly. A third-party LiveKit testing framework runs real browsers under Selenium with real WebRTC media — "not mocked," in its own words — orchestrated via Docker and Gherkin. This eliminates the driver as a separate implementation entirely: the test is the client. The costs are real (browser orchestration, media plumbing, flake) and it tests different server versions rather than detecting driver-vs-client drift — because there is no driver to drift.

Capture, then implement replay explicitly. HTTP recording tools such as VCR do not establish WebSocket replay support. mitmproxy can intercept and inspect WebSocket messages, but its official protocol documentation explicitly lists client/server replay as unsupported. mitmdump -r capture.mitm reads saved flows; adding -s addon.py runs an addon, not an automatic protocol-exact player. Its options reference defines file reading separately from replay. Captures remain useful evidence of the observed exchange; they do not cover unobserved branches, and mitmproxy does not store PING/PONG frames. To replay WebSocket sessions, implement or select and test a separate driver that handles handshake/auth refresh, correlation IDs, ordering, timing and server responses. Validate it against the real client before treating it as a substitute under section 6's conditions.

Hew to a standard. Mock Service Worker's WebSocket story is a cautionary tale from a tooling team that hit this exact problem from the other side: MSW abandoned its own multi-transport abstraction and rebuilt on strict WHATWG WebSocket compliance, precisely because an abstraction over a protocol drifts from the real protocol. WireMock, the JVM ecosystem's default mock server, only gained WebSocket support in a 4.x beta — for years the ecosystem's answer was bespoke forks, which is to say: everyone hand-wrote drivers, and everyone owned this risk.

The integration material sampled here covers transports, quality testing and media harnesses. Those documents are not an audit of the vendors' internal driver-maintenance processes. An omitted procedure in public integration documentation does not establish that the vendor lacks it.

4. Drift Detection: What the Selected Sources Establish

A useful CI question is whether a driver's outbound messages and behavior still agree with the production client. The selected tools below address parts of that problem: schema gates require a schema, verified fakes require shared behavioral checks, and traffic-derived tests require representative captures. This article proposes extracting the client's message constructors or capturing a real-client session, then comparing the driver's equivalent scenario in CI. It does not claim this combination is novel or absent from other tools.

What does exist, adjacent to the gap:

• Pact's positioning. Provider-verifies-consumer-pacts-in-CI is explicitly marketed as the fix for "mock drift" — the consumer's mock of the provider going stale. Our problem is the mirror image (the test's mock of the consumer going stale), which pact-style verification would also catch if the real client published pacts — but the client still needs a verified way to produce representative contracts.
• Traffic-derived mocks. Speedscale/Proxymock captures production traffic and regenerates local mocks plus CI replay from it, marketed as keeping test environments aligned with production. WebSocket support is unconfirmed; the architecture is the right shape regardless.
• A gateway compatibility request. IBM/mcp-context-forge#546 requests multi-version support and translation in that gateway, whose issue describes implementation-specific limitations. MCP itself already specifies version negotiation in its 2025-11-25 lifecycle. The gateway request illustrates compatibility work an implementation may need; neither it nor a proposed protocol rewrite establishes an ecosystem-wide breakage incident.
• An off-domain template worth copying. Multivon's fixture-staleness gate for LLM prompts commits a baseline artifact and reports staleness against it in CI. Nothing about that mechanism is prompt-specific; a committed baseline of client message shapes plus a staleness report is a possible basis for a driver-drift gate.

One adjacent example in the cited material is a WhatsApp Web CLI client rejected with "Client outdated (405)" following protocol changes. This concerns a non-official messaging client, not a voice-agent test driver. The sample does not establish the frequency of stale-driver release incidents or how widely teams document them.

5. "The Green Suite Encodes the Bug": The Literature Has a Name and a Fix

Defect 3 — both sides of the test boundary omitting the same field — is the richest-documented part of this failure class, and the literature converges on one fix from four independent directions.

The canonical statement is Google's SWE book, chapter 13: a fake without its own verification "can diverge over time as the real implementation evolves," and the prescribed fix is contract tests — one suite of behavioral assertions executed against both the real implementation and the fake. Fidelity is defined "from the perspective of the test": the fake need not be complete, but where it speaks, it must agree with reality, and the shared suite is what forces agreement.

The same pattern, independently named twice. Itamar Turner-Trauring's "verified fake" pattern prescribes running the same contract suite against the double and the real thing; 4comprehension's "fake drift" post gives the strongest concrete implementation shape for JVM-style codebases — an abstract test-suite class instantiated once per implementation, so every implementation (real, fake, driver) is pinned to identical behavioral assertions and a divergence fails the build. That is precisely the structure our incident lacked: the driver and the server fixtures each had their own notion of the protocol, and no shared suite forced them to agree with the client's.

The older roots. Fowler's "Mocks Aren't Stubs" flagged the failure mode two decades ago — "mockist tests… run green but mask inherent errors" — and the consumer-driven-contracts paper (Fowler/Robinson, 2006) is the lineage from which Pact grew. Hyrum's Law supplies the corollary from the consumer side: with enough consumers, every observable behavior of the protocol will be depended on by somebody, which is why "the spec didn't change, only an optional field was added" is not a safety argument (see section 6).

The citable modern war story is Red Hat's, from their OpenShift AI / Llama Stack backend-for-frontend work (May 2026), and it reads like our incident with the names changed: "our mocked tests kept passing, green checkmarks everywhere. And then someone would deploy, and the BFF would break against the real server." Their remedy stack: a real upstream instance in the test loop, record/replay for speed, and a daily "Compatibility Sentinel" job exercising the integration against both the stable and development versions of the fast-moving upstream, alerting to Slack — buying weeks of lead time on breaking changes instead of discovering them at deploy.

The ownership rule deserves restating as the organizational half of the fix: the team that owns the real implementation owns the fake. A driver owned by "the test side" has no forcing function tying it to protocol changes; a driver owned by the protocol's owners is updated in the same change set that alters the wire format, or the owners' own CI breaks.

6. When Protocol-Exact Replay Is a Valid Substitute for Full-Client E2E

A protocol-exact driver — even a perfectly synced one — is still a replay of a client, not the client. When is that good enough?

The clearest published framing is a decision framework from dev.tools comparing contract tests to end-to-end API tests: contract-level checks are excellent at catching field renames, type changes, and error-shape changes; weak at state-machine regressions, cross-service breakage, and side effects. Its sharpest point is directly on our incident: a non-breaking, additive spec change can still break a strictly-parsing or semantically-dependent consumer — adding a field is "compatible" by every schema-diff rule while being load-bearing in behavior, exactly what the generation field was.

The classical test-shape literature (the Test Pyramid; Kohavi-adjacent "Testing Trophy" arguments for mostly-integration; Google's test-sizes discipline and "Just Say No to More End-to-End Tests") all pull toward fewer full-e2e tests — but Google's own Testing-on-the-Toilet guidance attaches the honest price tag to the ones you keep: "allocate at least one week a quarter per test" to keep end-to-end tests stable. Uber's "Shifting E2E Testing Left" is the strongest recent counter-programming: they made e2e cheap enough to run pre-merge (≥90% single-attempt pass rate, sub-minute average runtime, "placebo executions" to measure flake), and report 71% fewer incidents per thousand diffs in 2023 — with the sobering note that configuration changes, which bypass most test layers entirely, cause up to 30% of incidents. Monitoring's synthetic-vs-RUM distinction is the same trade-off wearing different clothes: synthetic probes are "too predictable" to see what real users see; canarying (per the Google SRE Workbook) supplements but does not substitute.

Synthesizing across these — and this is a synthesis, not a single citable authority — protocol-exact replay is a valid stand-in for the real client when all of the following hold: the release under test is server-only with zero client-bundle change; no fields were added, renamed, or re-typed; timing, flow control, and reconnection logic are untouched; and the replay was captured from (or verified against) the current real client's message sequence. The moment any one condition fails — client code changed, a field moved, heartbeat or backpressure logic was touched — replay validity is void and a real-client pass is required. The practical value of writing these conditions down is that they turn "is the driver good enough for this release?" from a judgment call into a checklist.

7. What Voice and Streaming Agents Add to the Problem

Realtime voice raises the cost of every option above, which is why driver-based testing is so common in this domain despite the drift risk.

Driving the real client means driving real media. Chrome and Firefox fake-media flags have documented gaps (--use-fake-ui-for-media-stream is broken under full headless per public issue trackers; one fake stream per browser instance), and Daily.co's writeup of their headless WebRTC test robots is a catalog of operational pain: exact ChromeDriver-to-Chrome version pinning, hard-to-kill robot processes, custom EC2 AMIs. Server-side, GStreamer's audiotestsrc/fakeaudiosink provide CI-able synthetic audio. Vendors' own tooling concedes the cost problem: Vapi's Test Suites carry a 15-minute cap and per-minute pricing that the vendor itself admits is "impractical for wide, repeatable regression testing"; Retell sells live-call QA scoring at $0.10/minute; LiveKit's first-party test helpers run text-only — no real WebRTC, no audio timing, no turn-taking. Deepgram's Voice Agent quality harness streams audio in 50ms increments to simulate realtime input — a reminder that even "replay" in this domain must reproduce pacing, not just payloads.

One public writeup of a bidirectional-streaming test harness is Amazon's open-sourced Nova Sonic harness, whose framing sentence is the state of the industry in one line: "The only way most teams test today is to have someone physically talk to the system." The harness speaks the streaming protocol directly (session auto-renew before the ~8-minute timeout, turn-completion via speculative-text finalization, LLM-as-judge for nondeterministic output) — and its documentation does not address how the harness itself is kept in sync as the protocol evolves. This leaves driver maintenance unresolved in that particular document; it is not evidence about every vendor's practices.

The production-bug catalogs from this domain show what drivers must model to be worth anything: a Pipecat production guide documents an STT vendor's WebSocket dying silently after 60–70 seconds of silence, disconnect races against already-closed sockets, and a provider dropping 1-in-50 connections with code 1011 despite keepalives; an AWS builder's WebSocket-to-WebRTC migration postmortem reports session-affinity breakage, ICE candidates leaking VPC-internal IPs, and cold-start ICE failures — summarized as "local peer-to-peer testing masked production-only failure modes." A driver that models none of these still tests something; the danger is only in believing it tests everything.

8. Ranked Recommendations for a Small Team

Ordered by leverage per unit of effort, for a team running an ad hoc JSON-over-WS protocol:

1. Put a protocol version on the wire, and enforce a compatibility policy. LiveKit is a version-field precedent; strict rejection of unsupported or missing versions is this article's proposed policy. Increment the version for incompatible changes, and make both server and driver check compatibility. This can reject a driver declaring an incompatible version, but it cannot detect behavioral drift that retains an accepted version; shared contract checks remain necessary.
2. One shared contract suite, run against both the driver and the real implementation. The verified-fake / fake-drift pattern, and the convergent prescription of the Google SWE book and the Red Hat case. This is the structural fix for "green suite encodes the bug."
3. Ownership rule: whoever changes the protocol owns updating the drivers, in the same change set. Organizational, free, and the root-cause fix per Google's taxonomy.
4. Compare driver behavior with fresh real-client captures. mitmproxy provides capture/inspection, not WebSocket replay. Compare equivalent scenarios and account explicitly for dynamic fields; unexplained differences require investigation. Use a separately tested playback driver if replay is needed.
5. Schema-fy the protocol, then adopt schema-diff CI gates (AsyncAPI + Microcks/Specmatic, or protobuf + buf breaking). Higher upfront cost; buys entry into the entire mature tooling ecosystem.
6. Derive the driver from the client codebase — shared message-constructor module, codegen, or headless real client. Highest cost; structurally eliminates the dual implementation.
7. A daily compatibility sentinel exercising the integration against the live deployment with alerting — the Red Hat remedy, which converts deploy-time surprises into weeks of lead time.

9. Open Gaps

Scope and date. This is a selected-source survey dated 2026-07-28, with the mitmproxy protocols/options, LiveKit client protocol/source, MCP lifecycle/gateway request and Google SWE chapter 13 claims rechecked on 2026-09-10. The source list samples contract-testing tools (Pact, Specmatic, Microcks), client protocols/testing documentation (LiveKit, Vapi, Amazon Nova Sonic, Deepgram), and selected practitioner reports. It is not an exhaustive search or an audit of vendor internals.

The sample did not establish an exact public postmortem matching this incident or a turnkey tool comparing this driver's output with its production client's message constructors. Neither finding proves absence. Two academic papers remain unread behind paywalls (IEEE 10043304; Wiley STVR e70006); Speedscale/Proxymock WebSocket support remains unconfirmed. The TransferGo/Microcks adoption claim also lacks a primary source and is not used as evidence. Section 6's replay-validity conditions are this article's proposed checklist, not an externally certified standard.

Sources: Software Engineering at Google ch. 13 (abseil.io/resources/swe-book/html/ch13.html) · Red Hat, "How we built integration testing for a fast-moving AI backend" (developers.redhat.com, 2026-05-27) · pythonspeed.com/articles/verified-fakes/ · 4comprehension.com/avoiding-fake-drift/ · martinfowler.com/articles/mocksArentStubs.html · martinfowler.com/articles/consumerDrivenContracts.html · hyrumslaw.com · docs.pact.io (message pacts) · pactflow.io/blog/contract-testing-for-grpc-and-protobufs/ · docs.specmatic.io/supported_protocols/asyncapi/web-sockets.html · microcks.io/documentation/references/apis/async-api/ · theburningmonk.com/2025/04/how-to-detect-and-prevent-breaking-changes-in-event-schemas/ · medium.com/@chinthakadd (contract testing for EDA) · medium.com/billie-finanzratgeber (Erlang WS harness) · docs.livekit.io/reference/internals/client-protocol/ · stancalau.ro/livekit-testing-framework/ · docs.vapi.ai/calls/websocket-transport · mswjs.io/blog/enter-websockets/ · github.com/IBM/mcp-context-forge/issues/546 · modelcontextprotocol.io/specification/versioning · thenewstack.io/mcp-release-candidate-rewrite/ · docs.multivon.ai/guides/staleness · dev.tools/blog/contract-testing-vs-end-to-end-api-testing-a-decision-framework-for-engineering-teams/ · testing.googleblog.com ("Just Say No to More End-to-End Tests"; TotT 2016-09) · uber.com/blog/shifting-e2e-testing-left/ · Google SRE Workbook (canarying) · daily.co/blog/how-to-make-a-headless-robot-to-test-webrtc-in-your-daily-app/ · aws.amazon.com/blogs/machine-learning/evaluate-your-amazon-nova-sonic-voice-agent-at-scale-no-microphone-required/ · deepgram.com/learn/voice-agent-api-generally-available · luonghongthuan.com/en/blog/pipecat-voice-agent-production-scalable-guide/ · dev.to/aws-builders (WS→WebRTC migration) · github.com/openclaw/wacli/issues/187 · news.ycombinator.com/item?id=47048811 (voicetest.dev)