When the Send Succeeds but the Record Fails — Audit-Log Durability in Agent Communication Pipelines

Executive Summary

Agent fleets increasingly route every outbound message — Telegram replies, Slack posts, webhook calls — through a gateway that does two things per call: deliver the message, and record that it happened. These are two independent writes to two independent systems (an HTTP/API call and a local database insert), and nothing forces them to succeed or fail together. This is the classic dual-write problem, and in agent systems it shows up with a specific, painful shape: the audit record is the only evidence an agent has that it spoke on the user's behalf, so when the record silently disappears, the fleet operator loses the ability to answer "what did my agents actually say, and when."

The following anonymized operational observations motivate this note. They are not independently replicated public evidence:

Silent swallow. The gateway wrapped the audit-write call in a try/catch that logged a warning and continued, on the theory that "the message got through, don't block the user on a logging failure." The result: audit-write exceptions from an unrelated regression accumulated for days with the message-delivery success rate looking perfect, and nobody knew the audit trail had a hole in it until a reconciliation was attempted for an unrelated reason.
Environment drift breaking a dependency silently. A privilege-switched invocation (sudo -u otheruser) resolved a different Node.js installation than the one the audit-write binding had been compiled against. The compiled SQLite native addon's ABI no longer matched the running Node ABI. The message still went out — delivery and audit-write were sequential, not transactional, so the send had already completed — but the audit insert threw at the module-load or native-call boundary. Because the failure was caught and logged rather than surfaced, the sender only discovered it by accident when checking logs for an unrelated reason.

Both failures share a root cause: delivery and recording are treated as one logical operation but implemented as two independent, non-atomic writes, and the second write's failure mode is "best effort." The rest of this note surveys how distributed systems have solved structurally identical problems (dual writes, outbox pattern, at-least-once delivery, WAL durability), builds a failure taxonomy that generalizes past the two incidents above, and lays out concrete recommendations for agent communication gateways — record-before-send ordering, a durable local fallback log, mandatory alerting on audit-write failure, and periodic reconciliation between transport logs and the audit database.

The core recommendation: treat "deliver" and "record" as a single unit of work with an explicit ordering and an explicit fallback, never as two calls where the second one is allowed to fail quietly. Which of the two happens first is a real design choice with trade-offs (below), but "fail open, log a warning, move on" should never be the answer for the audit write.

The Dual-Write Problem, Restated for Agent Communication

The dual-write problem is well known in distributed systems: whenever an application needs to update two separate systems of record as part of what should be one logical operation, there is no built-in mechanism to make both updates succeed or fail together unless the two systems participate in a shared transaction. Classic examples are "write to the database, publish to the message queue" or "charge the card, create the order." A crash, exception, or partial failure between the two writes leaves the systems inconsistent, and because each write is usually retried independently (or not retried at all), the inconsistency can be a silent, permanent record loss rather than a visible outage.

Agent communication pipelines have exactly this shape:

handle_outbound_message(payload):
    result = transport.send(payload)      # write #1: external side effect
    audit_db.insert(payload, result)      # write #2: local system of record
    return result

Two things make this worse than the textbook e-commerce example:

The two writes have very different failure domains. transport.send talks to an external network service (Telegram/Slack/webhook API) and its failure is usually loud — a non-200 response, a timeout, an exception the caller has to handle to know whether to retry. audit_db.insert talks to a local resource (a SQLite file, in the observed incidents) that "should never fail," so engineers reach for defensive try/catch around it out of a reasonable-sounding instinct: don't let a logging subsystem take down the primary function of the service. That instinct is exactly what converts a loud failure into a silent one.
There is no natural retry-and-reconcile signal. In a queue-based system, a failed publish is visible as a gap in an offset or a DLQ entry. In a "send then locally log" gateway, a failed local write leaves literally nothing behind — no error surfaces to the caller, no entry exists to reconcile against, and a transport response lost before persistence may not be recoverable through any later history query.

This second point is the crux: the dual-write problem is dangerous in proportion to how invisible the failure is. A failed payment-then-order write is bad, but it usually produces a support ticket ("I was charged but got nothing") that forces reconciliation. A failed audit write produces nothing — no user complaint, no error budget burn, nothing but a hole in a table that nobody is looking at until an incident review needs it.

Failure Taxonomy

Generalizing past the two observed incidents, failures in a "deliver + record" pipeline fall into four categories:

1. Silent swallow (application-level)

The code path around the audit write catches exceptions and continues. This is nearly always introduced with good intentions — "a broken audit log shouldn't block message delivery" — and is nearly always wrong in isolation, because it removes the only signal that anything went wrong. The defensible version of this pattern keeps the non-blocking behavior for the user-facing path but makes the failure loud somewhere else (structured error log with alerting, metric increment, fallback write — see Recommendations). The indefensible version is a bare except: pass or except: log.warning(...) with no counter, no alert, and no fallback.

2. Environment drift (infrastructure-level)

The runtime environment used to execute a given call path is not the one that was tested or the one the dependency was built against. The observed case — sudo -u resolving a different Node.js version, whose ABI didn't match a compiled native addon — is a specific instance of a broader class: version managers (nvm, pyenv, rbenv), privilege-switch tools (sudo, su, runuser), containers with mismatched base images, and CI runners with a different toolchain than production all create paths where "the same code" runs against a subtly different environment and a compiled/native dependency breaks. This is particularly dangerous for native bindings (SQLite, better-sqlite3, node-gyp-built modules, Python C extensions) because the failure is an exception at import or first-call time, not a slow degradation — it is a hard break that a try/catch around "just the audit part" converts into an invisible one.

3. Partial writes

The audit write starts but doesn't complete atomically — e.g., a multi-statement insert where the first statement succeeds and the second fails, or a write that succeeds at the OS level but is not durable (not flushed/fsynced) before a crash. SQLite in WAL mode and most embedded databases guard against torn writes at the storage-engine level, but application-level partial writes (writing to two tables, updating a counter and inserting a row non-atomically) remain the caller's responsibility.

4. Ordering/atomicity failures (the core dual-write case)

Delivery succeeds, recording fails (or vice versa), and there is no transaction spanning both. This is the general case that categories 1–3 are specific instances of. The two sub-variants matter for design:

Send-then-record: message goes out, audit write fails → an unrecorded, but real, outbound message (the observed incidents' shape).
Record-then-send: the durable intent exists before the call. A definite rejection can be recorded as failed, but a timeout or crash after the call begins may leave the outcome unknown: the message could already have been accepted. An intent row must not be presented as proof of delivery.

Detection strategies

Write-then-readback: after an audit insert, immediately read the row back (or check rowcount/lastrowid) rather than trusting a non-exception return as success. Catches partial writes and silently-degraded storage.
Negative controls: periodically send a known-canary message through the full pipeline and assert both the delivery receipt and the audit row exist. This catches environment drift (category 2) proactively instead of by accident, because it runs the exact code path including any privilege switches, rather than a unit test that runs as the developer's own user.
Reconciliation sweeps: compare the audit DB with a transport's independently queryable outbound receipt history where one exists. Preserve unresolved outcomes if no such evidence is available. A proxy request log proves an attempt, not transport acceptance; only a durably captured successful response provides that evidence.
Fallback-file diffing: if a durable fallback log exists (see below), a nightly job that diffs "rows present in fallback but absent from the primary audit DB" surfaces every silent-swallow event even if no alert fired at the time.

Prior Art Survey

Transactional outbox pattern. The standard fix for the dual-write problem in service-to-service messaging: instead of writing to the database and separately publishing to a message broker, the application writes the business row and an outbox row in the same local database transaction, and a separate relay process (or CDC connector like Debezium) reads the outbox table and publishes to the broker, deleting/marking rows once confirmed. This preserves intent in one local transaction and supports an at-least-once relay, but replay can duplicate an external effect unless the receiver deduplicates it. The direct translation to agent communication: the audit write and the "intent to send" should be the atomic operation, and the actual transport send should be the at-least-once relay step, not the other way around. The AWS outbox guidance explicitly calls for idempotent consumers because duplicate delivery remains possible. Deduplicating local audit rows does not deduplicate messages at an external transport.

Write-ahead logging (WAL). Databases guarantee durability by writing every change to an append-only log before applying it to the main data structures, and only acknowledging the operation as committed once the log write is durable (flushed to disk). Recovery after a crash replays the log to determine what actually happened. The generalizable idea for audit logging: an append-only, sequentially-written log is cheap to make durable and cheap to make crash-safe, which is exactly the property a "local system of record for compliance" needs — and it is a good candidate for the fallback layer when the primary structured audit DB write fails.

At-least-once delivery + idempotent processing. Message systems that need reliability (Kafka with acks=all, SQS, most brokers) choose at-least-once delivery over exactly-once because exactly-once is either impossible or prohibitively expensive across independent systems, and instead push the burden of correctness onto idempotent consumers (dedup by producer ID + sequence number, or by a caller-supplied idempotency key). The audit-logging analogue: it is fine, and often correct, for the audit-recording step to be retried and to occasionally produce a duplicate row, as long as duplicates are cheap to detect (unique message ID) and reconciliation tooling treats "duplicate audit row" as a non-event while treating "missing audit row" as an incident.

Syslog reliability tiers. Classic syslog over UDP is best-effort. TCP improves transport reliability and TLS adds hop-by-hop security, but neither proves durable audit storage. RFC 5425 §6.3 specifies no application-layer acknowledgments: after a connection breaks, the sender may not know which messages reached the remote syslog application. Durable audit delivery needs a separate contract for persistent sender queues, replay/deduplication, receiver storage commits, and application receipts tied to those commits. The lesson that transfers directly: best-effort logging is a legitimate design choice, but only when it is an explicit, documented choice with a named consumer who has accepted the loss characteristics — not a default that falls out of an unexamined try/catch. An audit trail used for compliance, incident response, or "what did the agent say" reconstruction is never the right consumer for a best-effort transport.

OpenTelemetry / observability pipeline guarantees. Modern telemetry pipelines (OTel Collector, vendor agents) distinguish between the SDK-side export (which can buffer, retry, and drop under backpressure — usually best-effort by design, to avoid the telemetry pipeline taking down the host application) and durable sinks further downstream. The key design pattern worth borrowing is backpressure policy as an explicit, named setting (e.g., OTel's batch processor drop-vs-block behavior) rather than an accidental property of how the code happens to be written. An agent audit pipeline should make the same choice explicitly: what happens when the audit DB is unavailable — block the send, drop the audit record with alerting, or fall back to a durable file? All three are legitimate; "silently drop with a debug-level log line" is not.

Financial systems' journal-before-post. Ledger and payment systems generally journal an intent record before attempting the external effect (authorize/capture), specifically so that a crash mid-operation leaves a recoverable, reconcilable trail rather than an untraceable side effect. This is the record-before-send ordering discussed below, and it is the dominant pattern in domains where "we did something but can't prove what" is unacceptable — which describes agent audit logging as much as it describes payments.

Recommended Patterns for Agent Systems

1. Record-before-send (outbox-style), not send-then-record. Commit a pending intent row before the external call. Before a worker starts sending, durably mark its attempt in_flight; persist a successful transport response as sent with its returned message ID. Here sent means transport acceptance, not proof that a human read the message. A definite rejection can become failed. A timeout, lost response, or abandoned in_flight attempt becomes unknown, never automatically failed or sent.

This preserves the attempted message and its uncertainty even if the process dies after transport acceptance but before saving the response. Reconcile from authoritative outbound receipts if available, or retry with the same provider-supported idempotency key when that guarantee exists. Otherwise retain unknown and apply an explicit policy: hold for operator resolution, or retry while recording that a duplicate may result. Local outbox IDs alone do not make an external resend idempotent.

2. If send-then-record is unavoidable, the audit write must never silently fail. Some systems have legitimate reasons to prefer send-then-record (e.g., minimizing latency on the user-facing send path). If so, the audit write failure path must do all of the following, not just log a warning:

Append to a durable, dependency-light fallback (see #3).
Increment a metric/counter that feeds an alert (audit_write_failures_total), not just a log line that nobody greps.
Re-raise or surface the failure to a supervisory layer even if the original request to the caller still reports success — the caller succeeded, but the system is now in a degraded state that must be visible.

3. Durable append-only fallback file when the structured DB write fails. A flat, append-only file (or a WAL-mode SQLite file separate from the primary — different failure domain, e.g., no native-addon dependency, plain-text JSON lines) that the gateway writes to whenever the primary audit insert throws. This borrows the WAL principle of persisting records before treating them as safe; the fallback needs an explicit flush/commit policy and recovery checks. A remote syslog TCP/TLS fallback alone would not provide that guarantee without the separate queue, receipt, and storage contract described above. The local fallback should be as dependency-free as possible so that the failure mode that breaks the primary audit DB (a native ABI mismatch, a disk-full condition on that particular volume, a lock contention issue) is unlikely to also break the fallback. A nightly reconciliation job merges fallback entries back into the primary store and alerts on any fallback activity, since fallback activity is itself the leading indicator that something is silently broken in the primary path — this would have caught the ABI mismatch on day one instead of "by accident."

4. Alert on audit-write failure as a first-class signal, not a debug log line. The single biggest lesson from both observed incidents: the failures were technically logged, but logged in a way indistinguishable from routine noise. Audit-write failure should page or notify the same way a failed message delivery would — arguably more urgently, since a failed delivery is usually self-evident to the sender (an exception bubbles up) while a failed audit write is not.

5. Run negative controls / canary sends through the exact privileged code path. Because the ABI-mismatch incident was specifically triggered by a privilege switch (sudo -u) resolving a different toolchain, a canary test that only runs as the developer's own user would never have caught it. Health checks and canaries for agent gateways must exercise the actual invocation context — same user, same privilege elevation, same environment resolution — not a convenient proxy for it. This generalizes: any environment-drift class of bug (categories 2 above) is only caught by tests that run in the real execution environment, not adjacent to it.

6. Periodic reconciliation between transport logs and the audit DB. Sweep pending and unresolved attempts against whatever outbound evidence the transport actually provides. Telegram's getUpdates and webhooks deliver incoming updates; they are not a queryable receipt history of this bot's outbound sendMessage calls. sendMessage returns a Message on success, but its documented interface provides no general outbound-history query or caller-supplied idempotency key. A lost response can therefore remain unresolved, and retrying can send the same text twice. See Telegram incoming updates and sendMessage.

A gateway receipt ledger helps only if it durably saves the transport response and correlation ID before acknowledging its own caller. It can recover a caller crash after that save, but the gateway itself can still crash between external acceptance and saving the receipt. A generic access log or pre-send fallback entry establishes an attempt, not its outcome. Report unmatched intent, confirmed acceptance, and unknown outcome separately; do not turn absence from an incomplete log into proof of non-delivery.

7. Treat native/compiled dependencies in the audit path as a supply-chain risk, not an implementation detail. Where possible, prefer the pure-JS/pure-Python fallback of a database driver for the audit path specifically, or pin and verify the exact runtime (Node/Python version, architecture) the compiled binding was built against as part of process startup — fail loudly at boot if the resolved runtime doesn't match, rather than deferring the failure to the first audit write under a privilege-switched invocation.

Conclusion

The two incidents that motivated this note — a swallowed exception and an ABI mismatch surfaced only by a privilege switch — are different bugs with the same shape: an audit write was allowed to fail quietly because it was implemented as a second, independent step after the "real" work (message delivery) had already succeeded. The distributed-systems literature has a name and a well-tested fix for this shape of problem (the dual-write problem and the transactional outbox pattern), and adjacent fields (databases via WAL, message brokers via at-least-once + idempotency, payments via journal-before-post) have converged on the same underlying principle: make intent durable before the side effect, record confirmation when available, and keep unknown outcomes visible when the transport cannot resolve them. For agent fleets specifically, where the audit trail is often the only evidence of what an autonomous system said or did on a human's behalf, this is not a logging nicety; it is the mechanism by which "agent did X" remains a falsifiable, trustworthy claim.