Crash-Safe, Resumable Migrations Across a Dual Store: Database Metadata and Filesystem Bytes

Executive Summary

Almost every real system that stores "files" actually stores two things in two different places: a database row describing the file (path, size, checksum, owner, status) and the bytes themselves, sitting in a directory, an object-storage bucket, or a blob store. These two stores do not share a transaction manager. Postgres cannot roll back an S3 PutObject; the filesystem cannot roll back a COMMIT. Any migration that touches both — moving files to a new layout, re-encoding them, switching storage backends, backfilling a new metadata column that a background copier needs to populate — is therefore an unavoidably non-atomic, two-step operation, and a crash between the two steps is not an edge case, it is a coin flip that happens on every real deployment sooner or later. The two failure directions have asymmetric severity: "file copied, metadata not yet updated" self-heals (the migration just looks incomplete and can restart); "metadata says migrated, file missing or unverified" is silent and user-facing — a permanent 404 for a record that insists the resource exists, discovered by a customer instead of a health check. This article works through the fix: model the migration as an explicit state machine with a persisted status field (pending → copying → copied → verifying → verified → committed → swept, or a subset), gated by preconditions so every transition is idempotent and re-enterable after a crash at any point; use write-then-rename-then-fsync at the filesystem layer the way Postgres's own WAL and Lucene's prepareCommit/commit split use write-then-fsync-then-flip-the-pointer at the database and search-index layer; run a verification pass that treats "row says done" as a hypothesis to check, not a fact, and a separate reconciliation/orphan sweep that treats the two stores as sets to be diffed, not mirrors assumed to already match; and design the whole thing forward-only, because a "down" migration for bytes that were already deleted, re-encoded, or physically moved is usually a fiction nobody can actually run safely. The article closes on why this matters more, not less, for AI agents performing unattended upgrades: an agent that runs a migration, gets killed by a context rotation or an OOM kill mid-copy, and comes back with no memory of exactly which files were mid-flight needs the system, not its own recollection, to answer "where did I leave off" — which is precisely the property a persisted, idempotent state machine provides and an in-memory copy loop does not. This piece is a structural companion to Crash-Safe Incremental Persistence for Real-Time Agent Sessions (which covers keeping a single store — the DB — durable against mid-session kills) and to Making Deletion Stick (which covers keeping absence durable against a reconciler's next pass); this one is about keeping two stores mutually honest while a migration walks every row between them.

The Fundamental Problem: Two Stores, No Shared Transaction

A typical "files" feature looks like this: a table files(id, path, status, checksum, size, created_at) in Postgres or MySQL, and the actual bytes at path on a local disk, an NFS mount, or an S3-compatible bucket. Writing a new file "atomically" from the application's point of view means, at minimum: write the bytes somewhere durable, and insert or update the row that says they exist. There is no operation that does both at once. Every ordering choice has a specific failure mode if the process dies between the two steps:

Write file, then commit DB row. Crash after the file write but before the commit: an orphan file exists on disk with no metadata pointing to it — wasted space, invisible to the application, generally harmless and cleanable later.
Commit DB row, then write file. Crash after the commit but before the file write completes: the database asserts the file exists; every reader that trusts the row gets a 404 or a corrupt partial read for a resource the system itself vouched for. This is the dangerous direction, because it is silent at write time and only surfaces later, to whoever reads that row — often a customer, often much later than the crash, with no correlation back to the migration that caused it.

Migrations amplify this from "one row, one file, rare crash window" to "N rows, N files, a long-running batch job with a proportionally larger cumulative crash probability." A migration that walks 2 million rows, copying or re-encoding the associated file for each one and then flipping a storage_version or path column, has 2 million individual instances of exactly the race above, plus the meta-problem of the migration's own progress needing to survive a crash — otherwise restart means re-scanning from row 1, which is slow, and worse, unsafe if the copy step is not idempotent (partial re-copies, double-charged storage costs, or — if "copy" is destructive, like a move — data loss on the second attempt).

The academic framing for the general version of this is the dual-write problem: any time an operation must durably affect two systems that don't share a transaction coordinator (a database and a message broker, a database and a cache, a database and a filesystem/object store), there is no way to make the combined operation atomic using only the two systems' own primitives. The two standard resolutions are (a) make one side authoritative and derive the other, asynchronously, from a durable log of intent — the transactional outbox pattern: write the "I intend to do X" record in the same transaction as the row it describes, then let a separate, retryable, idempotent worker perform X and only then mark it done — or (b) accept eventual consistency and build a reconciliation pass that finds and fixes the cases where the two sides disagree, because they inevitably will. A DB+filesystem migration is a specialized, high-volume instance of the same problem, and the mature answer borrows directly from the outbox pattern: the database row is the durable log of intent, its status field is the outbox, and a background worker (the migration process itself, or its resumed successor) performs the file-side work and reports back — never inferring completion, always recording it.

Why "It Worked in Testing" Lies

A migration script tested on a laptop, on a small fixture, run start-to-finish without interruption, proves almost nothing about crash safety — it proves the happy path is correct, which was never in doubt. The property that matters is: for every one of the N discrete steps a migration performs per row (read old file, write new file, fsync, update row, verify, delete old file), if the process is killed at that exact instant and restarted, does it converge to a correct final state without operator intervention, without silent data loss, and without silently skipping the row. Standard test suites don't exercise this because a normal test run has no reason to inject a kill signal mid-write, and CI environments rarely simulate slow or partially-completed I/O. This is precisely why crash-consistency testing had to become its own subfield with its own tooling (covered later) rather than being caught by ordinary unit or integration tests.

The Framework Migration Illusion

Django, Rails/ActiveRecord, and Flyway all provide strong-feeling migration primitives — and every one of those primitives stops at the database's own transactional boundary, which does not extend to the filesystem or object store at all.

Django wraps each migration in a database transaction by default (atomic = True on the Migration class), which is genuinely useful for schema changes — if AddField fails halfway, Postgres rolls it back cleanly. But this default has known holes even within the database: it has no effect on databases that don't support DDL transactions (MySQL, Oracle — DDL there auto-commits statement by statement, so a failure mid-migration leaves partial schema changes with no rollback), and Postgres explicitly disallows some operations inside a transaction at all (CREATE INDEX CONCURRENTLY, and non-transactional operations generally), forcing atomic = False and manual atomic() blocks around the parts that can be wrapped. None of this machinery has any concept of a file on disk. A RunPython data migration that reads a FileField's path and moves the underlying bytes is doing filesystem I/O with zero durability guarantee from Django's transaction wrapper — if the process dies after the file move but before the surrounding transaction commits (or vice versa), Django's "atomic migration" did not protect that operation at all, because the operation was never inside the boundary the framework actually enforces.

Rails/ActiveRecord migrations have the same shape: ActiveRecord::Migration runs DDL and data changes, optionally transactionally per adapter, with zero built-in awareness of Active Storage blobs living in S3 or on local disk. Rails' own Active Storage subsystem independently has to solve a version of this problem for direct uploads: a client creates a blob record before the upload completes (so the DB row can exist before the bytes are confirmed durable), which means an interrupted upload — user closes the tab — leaves an orphaned blob row with no corresponding object, or (depending on ordering) an object with no attaching record. Rails' answer is not transactional cleverness, it's a scheduled reconciliation job: ActiveStorage::Blob.unattached.where(created_at: ..2.days.ago).find_each(&:purge_later), a sweep that finds blobs unattached to anything after a grace period and purges them. This is the reconciliation-pass pattern in miniature, shipped as a rake task because the framework authors correctly concluded that no amount of transactional discipline at write time eliminates the need for a periodic pass that reconciles the two stores after the fact.

Flyway versions and checksums SQL migration files and tracks applied versions in a flyway_schema_history table, which solves a real and adjacent problem — which migrations have run — with a clean, well-tested mechanism (checksum mismatch on a previously-applied migration fails validation, preventing silent drift). But Flyway's unit of atomicity is a SQL script executed against the database; a migration that shells out to move files, call a storage API, or re-encode blobs is, from Flyway's perspective, an opaque step that either the whole migration "ran" or "didn't," with no native concept of the file-side work being interruptible mid-step. Flyway's callbacks and Java-based migrations can do arbitrary file I/O, and Flyway will happily record that migration as applied once the callback returns — even if the callback's file-side effects are only partially durable at that instant, because Flyway's transactional guarantee (where the target database supports transactional DDL) covers the database session it opened, not any external system the migration code chose to touch.

The consistent lesson: every one of these tools gives you a real atomicity boundary, and that boundary is the database connection, full stop. The moment a migration's logical unit of work spans the database and a second store, the framework's transaction wrapper is decoration around the database half of the work and provides zero guarantee about the other half. Treating "wrapped in atomic()" or "flywayed" as proof the whole operation is crash-safe is the exact mistake this article is about.

The Staged Protocol: Mark → Copy → Verify → Commit → Sweep

The fix used by every mature system that actually solves this (Vitess/PlanetScale online schema changes, MongoDB's cluster-to-cluster migration verifier, Postgres's own CREATE INDEX CONCURRENTLY, cloud storage migration tooling) is the same shape: decompose the migration into small, individually-idempotent steps, and persist which step each unit of work is on, in the same database that's already the source of truth for everything else. A workable state machine for a per-row file migration:

pending    -- row identified as needing migration, no action taken yet
copying    -- file write to new location started (in-flight marker)
copied     -- new file written and fsynced; old file still authoritative
verifying  -- checksum/size comparison against source in progress
verified   -- new file confirmed byte-identical to source
committed  -- DB row now points readers at the new location
swept      -- old file deleted (or scheduled for deletion after retention window)
failed     -- terminal error state, retains enough context to retry

Example preconditions. This teaching protocol assumes immutable source bytes, or application writes stopped and drained from before copying through cutover. Each row has only one active executor; recovery starts only after the previous executor can no longer write. A status-only cutover cannot protect a live mutable source: verifying destination v1, then acknowledging source v2, can still publish v1. Overlapping takeover and live writes require additional mechanisms outside this example.

-- one durable status column drives the whole state machine
ALTER TABLE files ADD COLUMN migration_status TEXT NOT NULL DEFAULT 'pending';
ALTER TABLE files ADD COLUMN new_path TEXT;
ALTER TABLE files ADD COLUMN new_checksum TEXT;

-- claim a batch, idempotently: only rows still pending are eligible,
-- and the claim itself is a durable, visible fact before any file I/O happens
UPDATE files SET migration_status = 'copying'
WHERE id IN (
  SELECT id FROM files WHERE migration_status = 'pending' LIMIT 500 FOR UPDATE SKIP LOCKED
)
RETURNING id, path;

-- after the copy + fsync succeeds, and only then:
UPDATE files SET migration_status = 'copied', new_path = $1, new_checksum = $2
WHERE id = $3 AND migration_status = 'copying';

-- after independent verification (re-read new_path, hash it, compare):
UPDATE files SET migration_status = 'verified'
WHERE id = $1 AND migration_status = 'copied' AND new_checksum = $2;

-- the actual cutover readers observe: single-row, single-column flip
UPDATE files SET migration_status = 'committed', path = new_path
WHERE id = $1 AND migration_status = 'verified';

-- separate sweep job, run later, with a retention window
UPDATE files SET migration_status = 'swept'
WHERE migration_status = 'committed' AND updated_at < now() - interval '7 days';

Within those preconditions, WHERE guards constrain database transitions to their expected prior states. They do not make filesystem effects transactional or make arbitrary retries safe: a rejected update cannot undo an earlier rename. Recovery re-reads nonterminal rows and actual files, then repeats only operations whose idempotency has been established. Readers keep using the old path until the verified destination is durably published and the cutover updates that path. These guarantees depend on the source remaining unchanged and the prior executor being unable to write, not on the status column alone.

Vitess offers a useful comparison, but the per-file stages above are a proposed application protocol, not Vitess's lifecycle. Its documented migration states are queued, ready, running, complete, failed, and cancelled; each shard's primary tablet schedules and executes migrations independently, with status and control exposed by migration ID. This supports the narrower lesson of durable, queryable progress, not a guarantee of atomic visibility across shards. See Vitess 22: managed Online DDL.

Vitess's optional --postpone-completion flag holds cutover until an explicit completion command. An operator can use this to inspect readiness before releasing migrations, but a readiness check or a command targeting all shards does not turn their independent cutovers into a global transaction. A failure after one shard completes can leave mixed schemas. Plan schema compatibility and recovery per shard; do not assume that all readers switch at once. See postponed migrations.

Idempotency and Resume-on-Restart

The state machine above only delivers crash safety if every individual step is actually idempotent, not merely "looks retryable." Three concrete failure points and the discipline each one requires:

The copy step itself. If "copy" means "write bytes to a new key/path," writing the same bytes to the same destination twice is naturally idempotent — the second write just overwrites the first with identical content. If "copy" means "move" (delete the source as part of the operation), it is not idempotent by construction — a retry after a crash between delete-source and confirm-destination can lose data outright. The fix is structural: never model file migration as move-then-verify; always model it as copy-then-verify-then-delete-original, with the deletion pushed to the separate, later sweep phase, specifically because verification has to happen against a source that still exists.
Initial claims under concurrent workers. In the PostgreSQL example, FOR UPDATE SKIP LOCKED separates initial claims while their transactions hold row locks. Those locks end with the transaction; they do not provide exclusive filesystem access during later recovery. A nonterminal status is a recovery candidate, not permission to overlap the old executor. Establish that it can no longer write before resuming the row.
Verification as the actual proof of correctness, not the copy's exit code succeeding. A file write can return success from the application's perspective (the syscall didn't error) while still not being durable — buffered but not fsynced, or fsynced to a since-corrupted destination. The verification step should independently re-read the new location and compare a checksum against a value computed from the original source, not trust a checksum the copy step itself calculated in the same process invocation (which would not catch, for example, silent truncation on a slow network filesystem).

Content-addressable storage sidesteps a meaningful chunk of this problem by construction: when the destination key is the content hash (as in Git's object store, container registries, or a CAS-backed blob layer), "copy the same content twice" is trivially idempotent — the second write lands on the identical key with identical bytes, and a hash mismatch on read is self-detecting corruption rather than something a separate verification pass has to go looking for. This is why systems that can afford to restructure storage around content addressing during a migration (rather than path-addressing) get a large fraction of this article's guarantees close to free; systems that can't (because paths are part of a public API contract, for instance) have to build the verification discipline explicitly.

Forward-Only vs. Reversible: Why "Down" Migrations Are Often a Fiction Here

Schema migration tools (Rails, Flyway, Django to a lesser degree) popularized the idea that every migration should ship a reverse: up/down, or a Flyway "undo" script. For pure schema changes this is often achievable — DROP COLUMN reverses ADD COLUMN, modulo data loss the tool can at least warn about. For a DB+filesystem migration, a true reverse requires the old file layout to still exist, byte-for-byte, at the moment someone wants to roll back — which is precisely what the sweep phase's whole job is to eventually eliminate. Two consequences fall out of this directly:

The retention window is a rollback window, and the two are the same design decision. Rails' Active Storage cleanup task uses a grace period (2.days.ago) before purging unattached blobs precisely so a slow or resumed process still has a chance to attach them; a migration's swept transition should use an analogous window — days, not minutes — during which committed rows still have their pre-migration file physically present and rollback is a real, cheap operation (flip path back, no file I/O required). Once the sweep actually deletes the old bytes, rollback stops being "flip a pointer" and becomes "re-run the entire forward migration in reverse," which has exactly the same crash-safety requirements as the forward migration did — it is not a free undo button, it is a second migration.
Design the default as forward-only, and treat "rollback after sweep" as its own migration, not a feature of the first one. This is the same lesson Terraform's removed block and Kubernetes' cascade-deletion policies encode for a different flavor of irreversibility (covered in the companion piece on deletion persistence): once a system has converged past a point of no return, pretending the operation is still symmetrically reversible is worse than admitting it isn't, because it invites someone to trust an undo path that silently doesn't work.

Borrowing the Pattern From Systems That Already Solved It

Lucene's two-phase commit. IndexWriter exposes prepareCommit() followed by commit() specifically for the case where Lucene's index needs to stay consistent with an external resource like a database: prepareCommit() does essentially all the risky, potentially-failing work — flushing buffered documents and deletes, fsyncing new segment files, writing most of the next segments_N file — and if anything is going to fail (disk full, I/O error) it happens here, leaving the index in its previous valid state, still readable, nothing corrupted. Only commit() does the final, fast, low-risk step of finishing the segments_N write that makes the new state visible to readers. The segments_N file is the actual commit point: "the commit point is a list of segments... comprising the whole index at the point in time when the commit operation was successfully completed," and once that call returns, "even if the OS or JVM crashes or power is lost... after rebooting, the index will be intact... and will reflect the last successful commit." This is precisely the copied/verified-vs-committed split above, expressed as a library API: do all the fallible work first against a location nothing depends on yet, and make the switch to "this is now the truth" the very last, smallest, most reliable step.

PostgreSQL concurrent index operations expose partial progress. CREATE INDEX CONCURRENTLY permits normal writes while building; it does not avoid all table locks, and conflicting schema changes must wait. It registers an invalid index and scans the table across separate transactions, so it cannot run inside a transaction block. A failed build can leave the requested index INVALID (pg_index.indisvalid = false): ignored for queries but still adding update overhead, and possibly still enforcing uniqueness. After diagnosing the failure, the documented recovery is to drop that invalid index and retry creation, or rebuild it with REINDEX INDEX CONCURRENTLY. See CREATE INDEX: concurrent builds.

REINDEX CONCURRENTLY has a different recovery path because it builds a replacement for an existing index. An invalid _ccnew index is the failed replacement: drop it and retry the rebuild. An invalid _ccold index is the old index left after a successful replacement: drop that leftover. Numeric suffixes may be added to avoid name collisions. Verify the operation and catalog state before choosing which index to remove. Concurrent reindexing also holds table and index locks that prevent schema modification while allowing normal writes. See REINDEX: concurrent rebuilding.

The migration-design lesson is that partial progress must remain inspectable, with a recovery path specific to the operation and phase. An invalid catalog entry is evidence to investigate, not permission to delete an arbitrary index matching a suffix.

Git's blob-before-ref ordering. Git's object database and its use of index.lock embody the same write-then-atomically-flip principle at the version-control layer: a new object (blob, tree, commit) is content-hashed and written to .git/objects before anything references it, so a crash after writing a blob but before updating a ref leaves an unreferenced, harmless object — exactly the low-severity "orphan file, no metadata" failure direction from the introduction, not the dangerous one. Ref updates themselves use the lockfile pattern directly: create <ref>.lock with O_CREAT|O_EXCL (so a competing writer is detected, not silently overwritten), write the new value, then atomically rename() it over the real ref file to commit — with an atexit/signal handler registered specifically to clean up an abandoned lockfile if the process dies mid-operation. "A commit can never only partially record changes: the hash of a commit is computed and stored in the database only after all relevant trees, blobs and parent commits have already been stored" — objects first, pointer last, is the same ordering this article recommends for copied/verified before committed.

Write-then-fsync-then-rename-then-fsync-the-directory, at the filesystem layer itself. The correct durable file replacement recipe — independent of any of the above, and the primitive all of them ultimately rest on — is: write to a temp file in the same directory, fsync() the temp file's data, rename() the temp file over the target (POSIX guarantees this rename is atomic — a reader never observes a half-written target), then also fsync() the containing directory's file descriptor, because the rename itself is a directory-metadata change that is not guaranteed durable until the directory entry is flushed. Skipping that last step is a well-documented, easy-to-miss gap: "without directory fsync, the rename itself may be lost... application may think that rename is durable, but on accidental reboot, old file may appear." A migration's copy step should use exactly this four-step recipe for every file write it performs, not a bare write() or a naive move.

The outbox pattern, generalized. All of the above are specific instances of the general dual-write fix: make the database the durable log of intent (the row, with its status column, is the outbox message), and make every external effect (the file write) something a retryable, idempotent worker performs after reading that intent and reports back into the same row — never inferred, never assumed. Everything in the "Staged Protocol" section above is the outbox pattern applied at migration scale, one row of intent per unit of work instead of one queue message per event.

Verification and Reconciliation Passes

Two structurally different checks belong in a mature migration, and conflating them is a common mistake:

Inline verification, run once per unit of work, immediately after the copy, as part of the copied → verified transition above — a checksum comparison, a size comparison, ideally both. This catches corruption or truncation in the specific copy that just happened, before that row is ever allowed to reach committed and become visible to readers.
Periodic reconciliation, run separately, at a coarser cadence, over the entire dataset — not "did this one copy work" but "do the two stores, as sets, still agree, including rows the inline check never touched because the migration was interrupted before reaching them, and including files that exist in storage with no corresponding row at all." This is a diff operation, not a per-row check: enumerate everything storage has, enumerate everything the database claims should exist, and inspect both differences.

The naming convention used by ClickHouse's own S3-metadata consistency tooling frames this precisely: "a consistency check process generates a list of data objects as 'Set A' and metadata objects as 'Set B,' verifying whether all objects in Set A are referred to by objects in Set B, with gaps marking missing objects or orphan objects" — two independent enumerations, diffed, rather than trusting either side's self-report. MongoDB's own cluster-to-cluster migration-verifier tool applies the same idea to live database-to-database migrations: it independently re-reads both the source and destination clusters and reports document-level mismatches, rather than trusting the migration process's own "I copied N documents" counter. Stripe's internal large-table migration sequence, similarly, has correctness checking as an explicit, separate pipeline stage — after bulk import and async replication, before the traffic switch — via point-in-time snapshot comparison, precisely so the traffic cutover only happens once an independent check, not the copier's own optimism, has signed off.

Orphan Handling: The Two Kinds of Garbage

A migration — and ordinary operation of any dual-store system — produces two structurally different kinds of orphan, and the correct handling differs:

File with no metadata (an object in storage with nothing in the database pointing to it): low severity, generally safe to delete after a retention window, because nothing in the application is currently relying on it being reachable. This is what Rails' unattached-blob purge job and GitLab's gitlab:cleanup:project_uploads / gitlab:cleanup:orphan_lfs_files rake tasks clean up — GitLab's uploads cleanup task specifically "attempts to fix the file if it can find its project, otherwise moving it to a lost and found directory" rather than deleting blind, a conservative default worth copying: when a sweep finds an anomaly it can't confidently explain, quarantine it, don't destroy it.
Metadata with no file (a database row claiming a path that doesn't resolve to actual bytes): high severity — this is the exact "permanent 404" failure this whole article is about, and it needs to be surfaced as an incident-worthy finding, not silently cleaned up. The correct automated response is narrower than for the first case: flag the row (a status = 'inconsistent' or equivalent), stop serving it as if the file exists, and — if the migration is still in progress — re-queue it for another copy attempt from whatever source is still authoritative; only escalate to a human/operator decision if no authoritative source remains.

A sweep job that treats both kinds identically — "clean up anything that looks orphaned" — risks the worse failure of the two: deleting a metadata row that pointed at a file that's merely temporarily unreachable (a slow object-store replication lag, a network partition to one storage backend) converts a recoverable inconsistency into permanent data loss. The asymmetry from the introduction — orphan files are cheap, orphan metadata is dangerous — should be encoded directly into how automated cleanup is allowed to act, not just how the tables are described in a design doc.

Testing This: Deterministic Fixtures, Fault Injection, and Mutation Controls

Ordinary tests don't exercise crash safety because nothing in a normal test run kills the process at an arbitrary instruction boundary mid-write. The academic and industrial answer to "how do you actually test this" splits into a few concrete techniques, worth adopting even at small scale:

Deterministic post-crash fixtures. Rather than trying to literally kill a process at a random point (flaky, slow, hard to reproduce), construct fixtures that represent every reachable state of the migration's state machine directly — a row stuck in copying with no file at the destination, a row in copied with a file that fails checksum verification, a row in verified that never reached committed, a committed row whose old file was already swept out from under a concurrent rollback attempt — and assert the resume/reconciliation logic converges each one to a correct terminal state. This turns "did resume-on-restart work" from a timing-dependent integration test into a deterministic unit test per state.

Fault injection on the actual syscalls. For the filesystem layer specifically, academic tooling exists precisely because this is hard to get right by inspection: CrashMonkey is a record-and-replay framework that "simulat[es] power-loss crashes while [a] workload is being executed, and check[s] if the file system recovers to a correct state after each crash" at the block-I/O level, and found real crash-consistency bugs in mature, widely-used Linux filesystems (including one in a formally verified filesystem, FSCQ) by systematically enumerating crash points a manual test would never think to try. ALICE (Application-Level Intelligent Crash Explorer) takes a complementary approach specifically aimed at application update protocols rather than filesystem internals: it logs an application's actual syscalls (write, fsync, rename, unlink) during a normal run, then uses a specification of what a given filesystem's crash semantics actually permit (which is looser than most developers assume — buffered writes can be reordered, rename durability without a directory fsync is not guaranteed, and so on) to enumerate every crash state the application's actual sequence of calls could produce, not just the ones a human tester would guess to try. Applying either tool's methodology — even without the tool itself — to a migration's copy step means explicitly listing every syscall the copy performs in order (open temp, write, fsync, rename, fsync-dir, update-row) and asking, for each possible truncation of that sequence, whether the resume logic handles it.

Mutation controls: match the fault to the evidence. Introduce a known-bad variant (a positive control), such as moving instead of copying, and check that a test covering that failure detects it. Removing directory fsync needs a durability-aware model that can discard or reorder unsynced directory updates, or a suitable filesystem/VM crash harness. A normal SIGKILL and restart leaves the kernel cache alive; both variants may pass, so that result does not by itself make a process-recovery test vacuous. A call-order spy establishes only whether the call was made. Linux's fsync documentation distinguishes file persistence from directory-entry persistence; the harness must exercise that distinction before claiming to detect its loss.

A minimal checklist that covers most of the value cheaply, without adopting a research tool:

Kill the migration worker (SIGKILL, not SIGTERM) at each state-machine transition, one at a time, in a test harness that can pause execution at that exact point; restart only once the old worker cannot write; assert convergence to a correct terminal state. This establishes process recovery, not power-loss durability.
Run the same transition twice sequentially under the single-executor precondition; assert byte-identical result to running it once. This does not test overlapping takeover.
Run the reconciliation/sweep pass against a fixture containing one of each orphan type (file-no-metadata, metadata-no-file) and assert each is handled per its correct severity, not identically.
For durability testing, remove directory fsync and confirm a durability-aware model or suitable crash harness detects a permitted lost-directory-update outcome; keep the correct implementation as the baseline. Ordinary process restarts alone do not establish this.

The AI-Agent Angle: Migrations With No One Watching

Everything above assumes a human operator who notices a stuck migration, reads a dashboard, or gets paged. An AI agent performing an unattended upgrade — applying a schema change, moving a component's data files to a new layout, backfilling a new storage backend as part of a version bump — removes that backstop, and does so in a way that's structurally different from a human operator being asleep: a human operator who gets paged at 3am still has continuous memory of what they were doing when they went to bed; an agent that gets killed mid-migration by a context rotation, an OOM kill, or a session restart may come back with no recollection of the migration having started at all, unless the resume information lives somewhere more durable than its own context — precisely the finding of this platform's own incremental-persistence research on why agent memory (bounded by context, subject to compaction, explicitly designed to be discarded and reconstructed from external stores under this platform's own session-rotation model) cannot be the thing that answers "where did I leave off."

This makes the state-machine discipline in this article not a nice-to-have but close to a hard requirement for agent-driven upgrades specifically:

The resume signal must be a query, not a memory. "Which rows are still pending or copying" has to be answerable by a fresh agent session — or a different agent entirely — with zero conversational context, using durable status and verified source/destination state rather than assumptions about what the previous worker finished. An agent that tracks migration progress only in its own reasoning trace has built a system that is durable exactly until the next context rotation, which is the identical failure mode the "Crash-Safe Incremental Persistence" companion article diagnosed for in-memory conversation buffers, recurring here for migration progress instead of transcript turns.
The dangerous-direction asymmetry gets worse, not better, unattended. A human running a migration interactively who sees "row says committed, file 404" investigates immediately, because they're watching the terminal output. An agent-run migration with no verification pass will happily report success (exit code 0, "migration complete") while having silently produced exactly that state for some subset of rows — and the first sign anyone gets is a downstream 404, potentially reported by a user, potentially much later, with no obvious link back to which agent run caused it. The verification and reconciliation passes in this article are not optional polish for an unattended agent — they are the only mechanism that can catch this class of error at all, since there is no human in the loop watching for it in real time.
Idempotent, small, checkpointed steps are also what makes agent-driven migrations reviewable and interruptible by policy, not just by accident. An agent bound by this platform's own rule to confirm destructive operations before proceeding (irreversible deletes, the sweep phase specifically) needs the migration to actually be pausable at a safe, well-defined boundary — verified, waiting for confirmation before committed/sweep — rather than a monolithic script that either runs to completion uninterrupted or leaves an unknown, unlabeled partial state if stopped. The state machine that makes a migration crash-safe against an accidental kill is the same state machine that makes it safely interruptible for a deliberate human-in-the-loop gate — durability and controllability turn out to be the same design requirement, approached from two different directions.

The practical takeaway for any agent performing this kind of upgrade: treat the migration's own progress as data with exactly the same durability requirements as the data it's migrating — persist it in the same database, gate every transition with a precondition, verify independently before committing, sweep only after a retention window, and never let "did the migration finish" be a question only answerable by the transcript of the session that ran it.

Practical Checklist

Question	If yes →
Does a migration write to both a database and a separate file/object store?	Needs the full staged state machine, not a bare script
Can the copy step be re-run against a destination that already has partial content?	Needs write-temp / fsync / rename / fsync-dir, not in-place write
Does the migration's "done" status live only in a log line, exit code, or agent conversation?	Needs a persisted, queryable status column
Is "copy" implemented as move (source deleted as part of the copy)?	Change to copy-then-verify-then-separately-delete
Does anything read `path`/location fields before independent verification has run?	Move the reader-visible cutover to strictly after `verified`
Is there a scheduled job that diffs storage-as-set against metadata-as-set?	Add one; inline per-row checks alone miss rows the migration never reached
Are file-orphans and metadata-orphans handled by the same cleanup code path?	Split them — the severity and correct action differ
Can the same migration worker process crash and resume without operator input?	Verify with a kill-at-each-transition test, not inspection
Does the test exercise the failure introduced by each mutation?	Use transition tests for state guards; use a durability-aware model or suitable system-crash harness for omitted directory fsync
Does an agent running this migration unattended have any way to answer "where did I leave off" other than its own context?	It needs one — this is the load-bearing requirement for unattended runs

Sources:

"Non-atomic migrations in Django?," Django developers mailing list, groups.google.com/g/django-developers/c/aAYiyAqTlUc
"#25833 (Add support for non-atomic migrations)," Django Trac, code.djangoproject.com/ticket/25833
"Writing database migrations," Django documentation, docs.djangoproject.com/en/5.2/howto/writing-migrations
"Migrations," Django documentation, docs.djangoproject.com/en/4.2/topics/migrations
"Active Storage Overview," Ruby on Rails Guides, edgeguides.rubyonrails.org/active_storage_overview.html
"Rails Active Storage S3: Direct Uploads, Variants and Production Configuration," TTB Software, ttb.software/2026/04/15/rails-active-storage-s3-direct-upload
"Migrating from Paperclip to Active Storage in Rails 6.1.3," Zoe Walker, Medium
"Understanding the Dual-Write Problem and Its Solutions," Confluent, confluent.io/blog/dual-write-problem
"The Dual-Write Problem," AuthZed, authzed.com/blog/the-dual-write-problem
"Transactional outbox pattern," AWS Prescriptive Guidance, docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html
"Handling the Dual-Write Problem in Distributed Systems," Auth0, auth0.com/blog/handling-the-dual-write-problem-in-distributed-systems
Vitess 22: Managed, Online Schema Changes
Vitess 22: Postponed migrations
"IndexWriter (Lucene API)" and prepareCommit/commit documentation, lucene.apache.org
"Testing Lucene's index durability after crash or power loss," Mike McCandless, Changing Bits, blog.mikemccandless.com/2014/04/testing-lucenes-index-durability-after.html
"Transactional Lucene," Mike McCandless, Changing Bits, blog.mikemccandless.com/2012/03/transactional-lucene.html
CREATE INDEX, PostgreSQL documentation
REINDEX, PostgreSQL documentation
"Pg_Upgrade Bug with Invalid Concurrently-Created Indexes," EnterpriseDB, enterprisedb.com/blog/pgupgrade-bug-invalid-concurrently-created-indexes
BUG #9148, "generate invalid index (create index CONCURRENTLY)," PostgreSQL mailing list, postgresql.org/message-id/20140208062848.26700.46932@wrigleys.postgresql.org
"Git - api-lockfile Documentation," git-scm.com/docs/api-lockfile
"Recovering the index after a git crash?," git.vger.kernel.narkive.com/tL8qRvyP/recovering-the-index-after-a-crash
"A Detailed Explanation of the Underlying Data Structures and Principles of Git," Alibaba Cloud Community
"CrashMonkey: A Framework to Systematically Test File-System Crash Consistency," USENIX HotStorage '17, usenix.org/conference/hotstorage17/program/presentation/martinez
"CrashMonkey and ACE: Systematically Testing File-System Crash Consistency," ACM Transactions on Storage, dl.acm.org/doi/10.1145/3320275
"How to Durably Write a File on POSIX Systems," Calvin Loncaric, calvin.loncaric.us/articles/CreateFile.html
"Crash Consistency: fsync(), rename(), and Durability," 0xKiire, 0xkiire.com/crash-consistency-fsync-rename
"Rename atomicity is not enough," npm/write-file-atomic#64, github.com/npm/write-file-atomic/issues/64
"Clean up orphaned objects on s3," Altinity Knowledge Base, kb.altinity.com/altinity-kb-setup-and-maintenance/altinity-kb-s3-object-storage/clean-up-orphaned-objects-on-s3.md
"ClickHouse MergeTree on S3 - Keeping Storage Healthy and Future Work," Altinity, altinity.com/blog/clickhouse-mergetree-on-s3-keeping-storage-healthy-and-future-work
"Cleanup," GitLab Raketasks documentation, docs.gitlab.com (gitlab.org/gitlab/lib/tasks/gitlab/cleanup.rake)
GitHub - mongodb-labs/migration-verifier, github.com/mongodb-labs/migration-verifier
"Designing robust and predictable APIs with idempotency," Stripe Blog, stripe.com/blog/idempotency
"Implementing Stripe-like Idempotency Keys in Postgres," Brandur Leach, brandur.org/idempotency-keys
"Content Addressable Storage (CAS)," Abilian Innovation Lab, lab.abilian.com
"Idempotent Database Migrations: Safe to Run Twice," CI/CD Delivery Guide, cicd.ariefw.com/articles/22-2-writing-database-migrations-that-wont-break-when-run-twice
Zylos Research, "Crash-Safe Incremental Persistence for Real-Time Agent Sessions," zylos.ai/research/2026-07-22-crash-safe-incremental-persistence-realtime-agent-sessions
Zylos Research, "Making Deletion Stick: Negative Desired State in Self-Healing Agent Infrastructure," zylos.ai/research/2026-07-13-negative-desired-state-deletion-persistence