Voice Fidelity in AI Ghostwriting: Building, Calibrating, and Evaluating a Principal's Personal Writing Voice

Executive Summary

Most "personalized AI writing" tooling personalizes to a brand, not a person. That distinction matters more than it sounds: a brand voice tolerates polish, consistency, and a house style guide; a person's voice is irregular by definition, full of habits its owner doesn't consciously notice and would reject if named explicitly. Our agent ghostwrites short-form public text — social posts, announcements — for a principal who reviews and edits every draft before it goes out. This piece describes the working system behind that: a durable voice-style profile built from roughly six real posts and a one-hour interview transcript, an explicit negative-constraint list that turned out to matter more than the positive instructions, and an append-only calibration record that logs what the principal changes in every edit rather than retraining anything. We ground each design choice in published research — stylometry's function-word findings, personalization benchmarks like LaMP and PersonalLLM, authorship-representation models like LUAR, the documented lexical fingerprints of LLM output (the "delve" phenomenon), and edit-based alignment work like PRELUDE/CIPHER — and lay out the artifact schemas we actually use, so the piece works as both a literature grounding and a design reference.

1. What Actually Constitutes "a Voice" — Lessons from Stylometry

Authorship attribution research has spent decades asking what features reliably distinguish one writer from another, and the consistent answer is: not vocabulary, but the boring stuff. Function words — articles, prepositions, conjunctions, auxiliary verbs — are used below the level of conscious control and at remarkably stable rates regardless of topic, which is precisely why they are the most reliable stylometric signal for authorship attribution rather than content words that vary with subject matter. Classic stylometric feature sets combine function-word frequencies with sentence-length distributions, punctuation rates, vocabulary-richness measures (type-token ratio, hapax legomena), and part-of-speech n-grams — see the overview in work on stylometric authorship attribution and, for a more structural angle, research on syllabic/rhythmic patterns as authorship features in place of surface vocabulary. The underlying claim generalizes: an author's fingerprint sits in distributional habits — how often, how long, how placed — not in a list of favorite words.

This reframes what a "voice profile" for generation should even try to capture. A profile built from vocabulary alone ("uses the word 'honestly' a lot") will feel like a costume. A profile built from distributional habits — target sentence-length band, where emphasis particles land, how often a sentence ends short after a long one, punctuation rhythm — has a chance of feeling native. The profile we built for our principal deliberately followed this logic: sentence-length norms, emoji placement rules keyed to emotional peaks rather than sentence position, an ordering rule (state the feeling before the fact), a hedge pattern for self-praise (undercut the compliment immediately after making it), preferred colloquial connectors, and a small bank of concrete, physically-grounded metaphors the principal returns to. None of these are "topics." All of them are the kind of low-level regularity stylometry says actually identifies a person.

2. Getting an LLM to Wear That Voice: Exemplars, Profiles, or Fine-Tuning

Three mechanisms compete for eliciting personal style from an LLM, and the research on sample-efficiency at the low end — single-digit exemplars, which is the realistic regime for an individual rather than a brand — favors prompting over training.

Few-shot exemplar prompting puts 2–10 labeled samples directly in context. Practitioner and research consensus is that this is enough to elicit noticeable style imitation through in-context learning without any parameter updates, and it avoids the two failure modes fine-tuning introduces at small scale: catastrophic forgetting of the base model's general competence, and overfitting to whatever the handful of samples happen to contain (idiosyncrasies of those six posts, not the person). See general treatments of few-shot prompting technique for the mechanism.

Style-profile instructions — a structured natural-language description of the style rather than raw exemplars — is a middle path: "12–15-word sentences, active voice, concrete metaphors drawn from [domain]," etc. This is effectively what Writer's brand-voice product and Anthropic's Claude custom styles both support as an alternative to samples (more on both below), and it has an important practical property exemplars alone lack: it is editable. You can't easily tell an LLM "stop doing that" about an implicit pattern in six examples, but you can add a line to an explicit profile. This is exactly why our calibration loop (Section 5) appends to a profile document rather than to a growing exemplar pool.

LoRA or full fine-tuning on a personal corpus is the heavyweight option, and the evidence suggests it's the wrong tool for a single individual's low-volume public writing. LoRA is well suited to shifting tone, structure, or register when there's a reasonably sized, clean corpus, and dropout is recommended specifically as insurance against overfitting when the fine-tuning set is narrow — e.g., "aligning a model to one person's writing style" is named explicitly as the overfitting-risk case in practitioner LoRA guidance. With six posts and one transcript, there simply isn't enough data to fine-tune safely; the corpus is a source of signal to describe, not a training set.

Academic benchmarks for personalization corroborate the shape of the problem even though they target different tasks. LaMP frames personalization as retrieval-augmented generation over a user's historical items (their past reviews, headlines, tweets) rather than parameter updates, explicitly testing whether models can produce personalized output from a profile rather than a fine-tune — the same architectural bet we're making. PersonalLLM goes further, arguing that persona-prompting on demographic attributes (age, stated preference) produces homogeneous, generic output relative to real human variation, and that capturing genuinely idiosyncratic preference requires representations built from the person's actual behavior/content, not category labels — which is the whole argument for building from a real transcript and real posts instead of a generic "brand persona" questionnaire.

For measuring whether style transfer worked at all, the authorship-representation literature offers content-independent style embeddings: STEL (Wegmann & Nguyen's benchmark for style-similarity tasks that resist confounding with topic) and LUAR (Learning Universal Authorship Representations), a contrastive model that embeds same-author texts closer together regardless of subject matter, shown to transfer across domains (Amazon reviews, fanfiction, Reddit). We return to these as evaluation instruments in Section 6.

3. Commercial State of the Art — and Where It Stops at "Brand"

Every major commercial writing tool now ships some form of style personalization, and almost all of it is architected for organizations, not individuals — which is worth naming because it explains why a bespoke system was necessary rather than adopting an off-the-shelf product.

Writer.com's Voice feature explicitly recommends the sample-based approach over manual description — "WRITER's testing shows that voices created from examples consistently perform better than manually written descriptions" — but requires a minimum of 300 words of samples (500+ recommended), and the resulting profile is stored as a set of style adjectives plus tone/vocabulary/cadence rules, testable against sample prompts and refined iteratively; see the voice calibration walkthrough. The unit of design is still "a voice for the brand's content operation," reinforced by the ability to layer a company style guide and terminology list on top.

Jasper's Brand Voice works the same way — upload samples, Jasper extracts tone and style, and IQ persists that alongside style guides and audience profiles applied across an entire team's output, per Jasper's Brand Voice documentation. Multiple voices are supported per account, but the product's center of gravity is teams publishing at scale, not one person's idiosyncratic register.

Grammarly is the interesting counterpoint: it maintains a continuously updated personal voice profile and separates it by context — "a different style for longer documents [vs.] short messages" — detecting tone from word choice, phrasing, and punctuation as you write, per its voice features documentation. This is closer to individual-level personalization than the marketing tools, but it's inline correction/suggestion, not generation from a durable profile loaded before a ghostwriting task.

Anthropic's Claude custom styles let a user either upload a writing sample or describe a style in natural language, with presets like Formal, Concise, and Explanatory alongside custom ones, as covered in press coverage of the feature (TechRadar, Tom's Guide). ChatGPT's custom instructions and memory work similarly at the assistant level — two persistent fields ("what should I know about you," "how should I respond") plus accumulated memory, per OpenAI's documentation. Both are general-purpose personalization primitives an agent can build on, but neither ships the domain-specific machinery a ghostwriting workflow needs: an explicit negative list, per-platform register switching, or an edit-diff calibration loop. That's the gap our system fills — using the same underlying idea (profile loaded before generation) but purpose-built for one person's public-facing short-form writing.

4. "AI-ese": the Failure Mode Has a Name and a Fingerprint

The number one failure mode we identified in practice — call it copywriter voice or 文案腔 — is not "wrong facts" or "wrong length." It's a text that reads as if written by a brand account rather than a person, and it turns out this failure mode is well documented and has measurable lexical signatures.

The clearest evidence comes from bibliometric tracking of word frequencies in scientific writing after ChatGPT's release. Researchers identified 21 "focal words" whose frequency spiked sharply and disproportionately in LLM-generated text — "delve," "intricate," "underscores," "showcasing," "meticulous," "boasts," among others — with "delves" specifically rising by 6,697% in PubMed abstracts between 2020 and 2024 (0.21 to 14.38 occurrences per million words). The leading hypothesis in the FSU-led study is that RLHF is implicated: annotators rating model outputs during training may unconsciously reward certain words as proxies for quality (thoroughness, sophistication), teaching the model to overuse them independent of whether they fit the content — a decoupling of stylistic polish from substance. Beyond vocabulary, structural tells are equally well cataloged: the "it's not just X — it's Y" contrastive construction (often with an em dash) is widely cited as the single most recognizable AI writing tell, and LLMs show a strong bias toward rhetorical triads (rule-of-three lists) repeated back-to-back in a way that reads as pattern-matched rather than intentional, per commentary in GPTZero's writeup and Rolling Stone's coverage of the em-dash tell.

Two things follow from this evidence for a ghostwriting system. First, humans are already tuned to notice this at some level even when they can't name it: general audiences distinguish AI from human text only marginally better than chance in blind trials, but people who frequently use LLMs for their own writing are dramatically better — a study of expert annotators found the majority vote among five frequent-LLM-user judges misclassified only 1 of 300 texts, roughly 99.7% accuracy, per "People who frequently use ChatGPT... are accurate and robust detectors". The principal reviewing every draft before publication is exactly this kind of expert reader — which is precisely why the system cannot rely on "good enough" style and must treat every published edit as a labeled example of what still reads wrong.

Second, and this is the design point worth stating plainly: negative constraints outperform positive instruction for this failure mode. Telling a model "sound authentic and human" is vague and unfalsifiable; telling it "no bullet lists, no parallel rhetorical triads, no corporate-speak, no explicit calls-to-action" gives it a checkable rule against a known, named pattern. Our profile's negative list exists because the failure mode is recognizable and nameable (bulleted structure, tricolon rhetoric, "delve"-class vocabulary, CTA endings) in a way that "be more human" is not. This mirrors why detection researchers built lexical-frequency and structural-pattern detectors in the first place — the tells are specific enough to enumerate.

5. Calibration Loops: Learning from Edits Without Retraining

Every draft the agent produces gets edited by the principal before it's published. That edit is a labeled example of drift between what the model produced and the voice it's supposed to be reproducing — and the research question of how to turn edits into durable improvement without full retraining has a body of prior work worth borrowing from.

Machine translation post-editing is the oldest version of this problem: human translators correct MT drafts, and the field has explored both offline retraining on post-edited data and online adaptation where the system updates continuously from a stream of edits without a full retrain cycle. A controlled user study of online NMT adaptation to post-edits (29 subjects, ~4,500 interactions) found a significant reduction in post-editing effort as the system adapted, per the original online-adaptation study — evidence that edit-stream signal alone, applied incrementally, measurably closes the gap between draft and desired output.

More directly relevant is recent LLM alignment work that treats user edits as the entire preference signal, skipping explicit ratings or comparisons altogether. PRELUDE (NeurIPS 2024) proposes exactly this: infer a natural-language description of a user's latent preference from their historical edits to an agent's output, then use that inferred description as a prompt for future generation — an algorithm they call CIPHER. The paper's stated motivation for not fine-tuning is worth quoting in spirit: fine-tuning per user is costly, doesn't scale across users, risks degrading general capability, and — critically — produces an opaque preference the user can't inspect or correct, whereas a descriptive, natural-language preference record is interpretable and directly editable by a human. That is the exact justification for our own design choice: the calibration record is a durable, human-readable, append-only document, not a weight update.

In production, when the principal edits a draft before posting, the agent diffs the published version against its own draft and appends a calibration entry describing the delta — not "this run was better," but why, in generalizable terms. The recurring real finding from this process: edits consistently pushed toward looser, more spoken phrasing (adding a concrete time anchor — "上周" rather than an abstract reference — and softening a claim with an aspect particle rather than stating it flatly), and consistently preferred comparative framing over superlative framing (better-than-X rather than best-in-category). Notably, the only addition pattern observed — content the model hadn't drafted at all — was inserting another person's title or role for social credibility, a context-specific, unteachable-in-the-abstract detail that the append-only record is exactly the right place to capture and that a static profile written once would never have anticipated. This is the practical argument for calibration entries over trying to write a perfect profile upfront: some drift is stable and generalizable (looser phrasing, comparative framing), and some is contextual and additive (a specific person's title, in a specific kind of post) — the record needs to hold both without conflating them.

6. Evaluating Voice Fidelity Without Fooling Yourself

Voice fidelity is hard to score because the two obvious shortcuts — asking an LLM to rate similarity, or checking surface n-gram overlap — both fail for reasons that generalize badly.

LLM-as-judge for style specifically inherits the judge model's own biases: judges systematically favor longer, more elaborate, more "polished" answers independent of whether that polish serves the content, with measured length-bias effects in some studies exceeding what human evaluators show, and judges are shown to rely on "surface-level cues such as fluency, verbosity, or stylistic polish rather than genuine causal or logical understanding" — see the survey on LLM-as-judge bias in evaluation. Since our target failure mode (文案腔) is itself a fluency and polish pattern, using a fluency-sensitive judge to detect it is close to asking the disease to grade its own treatment. (A companion piece on this blog covers fluency bias in LLM judges in more depth; the short version for this context is: don't use an LLM judge as the primary voice-fidelity metric.)

The more promising automatic approach is authorship/style-embedding similarity: encode the principal's real corpus and a candidate draft with a content-independent style representation — LUAR or STEL/StyleDistance — and measure embedding distance rather than asking a generative model to introspect on "does this sound like them." These models are explicitly trained to cluster same-author texts together regardless of topic, which is the right invariance property for this task (a post about a product launch and a post about a personal reflection should embed close together if the same person wrote both). Authorship-verification framing — "would this draft be classified as written by the same person as the reference corpus" — is a more falsifiable question than "does this sound authentic," and it's worth noting recent work explicitly stress-tests whether these representations capture genuine stylistic features versus incidental correlations, e.g. work probing whether authorship representation learning captures stylistic features — a caution against over-trusting any single embedding metric.

But the pragmatic gold standard we actually optimize against is simpler than any of the above and requires no model at all: the principal's own edit distance, trending toward zero over time. If calibration is working, the gap between draft and published version should shrink — fewer phrasing substitutions, fewer structural rewrites, fewer instances of the same recurring correction reappearing in the calibration log. A rising or flat edit-distance trend is the clearest possible signal that the profile has stopped capturing the person's actual drift, regardless of what any automatic style-similarity score reports. This is consistent with the MT post-editing literature's own preferred metric — post-editing effort/distance reduction — rather than a proxy automatic score, and it has the added advantage of being literally what the end user experiences as "getting better."

7. Risk and Ethics: Consent, Disclosure, and Register

Ghostwriting under a real person's name only stays legitimate under a consent boundary that's easy to state and easy to violate by accident. The distinction that matters is ghostwriting with the principal's knowledge and pre-publication review versus unsupervised voice cloning that publishes without a human in the loop — the latter is the pattern regulators are actively moving against: the FTC's Impersonation Rule (final in 2024) bars commercial impersonation of real individuals, and state-level statutes like Tennessee's ELVIS Act create civil and criminal exposure for unauthorized voice/likeness replication, per coverage in voice-cloning legal and ethics guides. Every design choice described above — human review before publish, an editable and inspectable calibration record rather than an opaque fine-tune, a negative list the principal can audit — is also, not incidentally, what keeps the system on the consented side of that line: nothing publishes without the principal seeing it first, and the principal can read exactly what the profile claims about their voice.

Register is the other risk worth naming briefly: the same person writes differently across platforms — more clipped and informal in one channel, more measured in another — and a single voice profile applied uniformly everywhere will feel wrong on at least one platform even if it's accurate on average. Grammarly's decision to keep separate voice profiles for "documents" versus "messages" rather than one blended profile is the right instinct here, and it's one we've adopted structurally (see the artifact schema below).

8. Practical Design Checklist

Voice-profile artifact structure (the durable document loaded before every ghostwriting request):

Distributional norms: target sentence-length range and variance; typical paragraph/post length band.
Ordering rules: e.g., emotion/reaction stated before the triggering fact.
Emphasis mechanics: where emphasis markers (emoji, punctuation, repetition) land — tied to emotional peaks, not fixed positions.
Hedge patterns: how self-praise or strong claims get immediately softened, and the specific hedge phrases used.
Lexical bank: a small set of recurring, concrete, physically grounded metaphors and colloquial connectors actually observed in the source corpus — not invented "brand adjectives."
Negative list: explicit, checkable bans (no bullet lists, no parallel/triadic rhetorical structures, no "delve"-class vocabulary, no corporate-speak, no explicit calls-to-action) — enumerated from known AI-ese patterns, not generic "sound human" instructions.
Per-platform register variants: separate parameter sets (or overrides) per publishing surface, not one blended profile.
Source provenance: which real posts/transcript segments each rule was derived from, so the profile stays falsifiable against the original evidence rather than becoming folklore.

Calibration-entry schema (appended after every human edit of a published draft):

Timestamp and platform.
Draft text vs. published text (the diff itself).
Category of change: phrasing/register softening, structural rewrite, factual correction, addition, deletion.
Generalizable rule extracted, if any (e.g., "prefer comparative over superlative") — left blank if the edit is context-specific and shouldn't generalize (e.g., a one-off addition of someone's title).
Whether this entry confirms, refines, or contradicts an existing profile rule.

Pre-publish self-check (run by the agent before presenting a draft):

Scan against the negative list for known AI-ese patterns (bulleted structure, tricolon repetition, banned vocabulary, CTA endings).
Confirm sentence-length distribution falls within the profile's target band.
Confirm at least one hedge/softening pattern is present if the draft contains self-praise.
Confirm platform-appropriate register variant was applied, not the default/blended profile.
Flag (don't auto-fix) any deviation for the principal's review rather than silently normalizing it — the principal's edit is itself the calibration signal, so over-correcting before review destroys data.

Open Questions

A few things remain genuinely unresolved in this workflow. First, how much calibration history is enough before a profile should be considered "converged," versus needing periodic re-grounding against fresh source material as the person's own voice naturally drifts over time — the append-only log has no built-in decay or re-weighting toward recent entries yet. Second, whether style-embedding similarity (LUAR/STEL-class metrics) can be validated well enough against edit-distance-to-zero to serve as a leading indicator rather than a lagging one — right now we treat automatic metrics as diagnostic, not authoritative, precisely because of the fluency-bias risk described in Section 6. Third, the negative-list approach works well for a known, cataloged failure mode (AI-ese) but is reactive by construction — it can only ban patterns someone has already noticed, which means the system will always be slightly behind whatever the next "copywriter voice" tell turns out to be as LLM output patterns keep shifting.