Zylos LogoZylos
2026-07-20

Default, Override, or Append: Prompt-Template Layering in Multi-Tenant AI Agent Products

researchprompt-engineeringmulti-tenantconfiguration-managementai-agentsrounds

Executive Summary

Every AI agent product that lets a customer edit "how the agent behaves" — a system prompt, a probing script, a digest template — eventually has to answer a question that looks small but is architecturally load-bearing: when a tenant customizes a field, does their text replace the product default, or does it append to it? The two answers produce genuinely different products. Override semantics give a tenant a WYSIWYG snapshot: what they see in the field is exactly what the agent gets, full stop. Append semantics give the tenant a delta: the product default is always injected first, and their text rides on top, which means every future improvement the product team ships to the default reaches every tenant automatically, without anyone re-editing a field.

This distinction is not cosmetic. It determines whether a product's prompt quality improves monotonically over its install base, or whether it fragments into thousands of frozen snapshots the moment a tenant clicks "customize." Zylos's Rounds component hit exactly this fork in the road on 2026-07-20: its global probing_guidance field had shipped with a seeded default text specific to daily standups, but that global field feeds every communication task, not just the daily one — so the daily-specific guidance was leaking into unrelated tasks. The fix (commit ee86632, v0.20.0) emptied the global default entirely and moved the daily-specific probe into a code-level default carried by the built-in daily task, with a rule that a tenant's custom probe_instruction field appends after the code default rather than replacing it. The result: teams who have never touched the field get the maintained default; teams who added "chase cross-team blockers immediately" get the maintained default plus their delta; and the next time the product team improves the standard probing script, every team's calls pick it up without anyone touching a setting.

This article works through the design space that decision sits in: the override-vs-append trade-off, layering models for where defaults live, propagation and versioning strategies, and prior art from configuration management (Kustomize, Helm, Terraform), styling (CSS cascade layers), and templating (Jinja2/Django) that agent-prompt systems can borrow wholesale rather than reinvent.

The Core Distinction: Override vs. Append

Override semantics: a tenant-editable field, when non-empty, entirely replaces the product default for that scope. When empty, the runtime falls back to the default. The composed prompt at any moment is either 100% product-authored or 100% tenant-authored for that field — never both.

Append (layered) semantics: the product default is always injected, unconditionally, and the tenant's field (if non-empty) is concatenated after it. The composed prompt is always at least the default, plus an optional delta.

DimensionOverrideAppend
What the tenant sees in the field == what the agent getsYes (WYSIWYG)No — field is a delta, not the whole prompt
Default improvements reach existing customizersNo — frozen at the point of customizationYes — automatically, every deploy
Risk of silently losing built-in behaviorHigh — clearing/editing the field can drop guidance with no fallbackLow — the code default is not editable away
Discoverability of "what's actually running"High — read the field, that's itLower — must know the default exists and where it lives
Risk of duplicated/contradictory guidanceLowHigher — tenant may unknowingly restate what the default already says
Best fitWhole-document templates (report layout, output schema)Incremental behavioral guidance (follow-up rules, tone nudges)

The Rounds case is a clean illustration of the second row. Before the fix, a team that had customized probing_guidance was permanently on whatever version of the probing script existed when they typed it in — the classic "copied-the-default-into-my-field" anti-pattern, where the seed default is presented as a starting point to edit, and editing it silently opts the tenant out of every future improvement. This is worse than it sounds because it's invisible: nothing in the UI tells the tenant they've forked. AWS's SaaS Lens design guidance names the same failure mode in a different vocabulary: it recommends "a universal mechanism that is available to all tenants, eliminating the need to deploy and manage separate environments or versions for each tenant" (AWS SaaS Lens, OPS_4) — precisely because per-tenant forks are an operational and quality liability, not just an engineering inconvenience.

Where Each Model Belongs: A Decision Heuristic

The heuristic that falls out of the Rounds case and the prior art below is roughly: override for whole-document replacement, append for incremental behavioral guidance.

  • Override is right when the field is the artifact — a digest template, an output schema, a report layout. There is no sensible way to "append" two competing document structures; the tenant either wants the product's shape or their own, and mixing halves of two templates produces garbage. Here, WYSIWYG transparency matters more than auto-propagation, because a template redesign is a deliberate, opt-in choice a tenant should consciously accept, not something that silently rewrites their report layout underneath them.
  • Append is right when the field is guidance layered on top of a baseline behavior — probing rules, tone constraints, escalation triggers. These are naturally incremental: a tenant wants "also do X," not "replace the whole reasoning process with X." Losing the baseline (e.g., "confirm completion status," "chase blockers") because a tenant added one custom line would be a regression nobody intended.

A second, orthogonal test: does an empty field mean "use nothing" or "use the default"? If clearing a field should silently disable a whole behavior, override-with-empty-fallback is fine. If clearing a field should never disable core behavior — only remove the tenant's addition to it — append is the only semantics that doesn't have a silent-behavior-loss failure mode. Rounds' own empty-string-vs-fallback discipline is visible in its test suite: assert.equal(s.getContext('probing_guidance'), '') — an explicitly empty global default is a deliberate value, not a null that triggers some hidden fallback, while the daily task's code default is injected unconditionally regardless of what the custom field holds.

Layering Models: Scope and the "Empty Global" Pattern

A second axis, independent of override-vs-append, is how many scopes stack, and what lives at each. The Rounds redesign is really a two-axis fix: it changed the composition operator (override → append) and the scope where the daily-specific content lives (global container → task-level code default).

The bug being fixed was a scope leak: a container labeled "global" (applies to every task) was carrying content that was actually task-specific (daily standup only). Any non-daily task inherited daily-specific instructions it had no use for. The fix follows a pattern visible across config-layering systems generally: keep cross-cutting defaults at the broad scope, and push scope-specific defaults down into the scope they actually belong to, emptying the broad scope of anything narrower than its stated reach. Concretely: DEFAULT_PROBING_GUIDANCE (global) is now ''; the previously-global daily probing script became dailyProbeDefault, a constant scoped to the daily task and injected only when !generic (i.e., only for the daily flow), with non-daily tasks getting neither the old leaked content nor any daily-flavored replacement.

This is precedence-by-scope, and it has well-known analogues:

  • Kustomize separates a base (shared resource definitions) from overlays (environment-specific patches), using strategic merge patches that understand Kubernetes field semantics — merging maps and list-by-key rather than blindly overwriting, with an explicit $patch: replace escape hatch when a field genuinely needs full replacement rather than merge (Kustomize strategic merge patches; Kustomize inlinePatch.md). The parallel to Rounds: the daily task is the "base," the tenant's probe_instruction is the "overlay," and the composition is a merge (append) by default, not a replace.
  • Helm resolves values in explicit precedence order — chart values.yaml default, overridden by a parent chart's values, overridden by a user-supplied values file, overridden by --set flags — and gives operators --reuse-values / --reset-then-reuse-values / --reset-values flags specifically to control whether an upgrade keeps prior overrides, replays them onto new defaults, or discards them in favor of the chart's new baseline (Helm values files docs; helm upgrade docs). This is the clearest prior art for "how do you decide, at upgrade time, whether the tenant's prior customization should ride on top of the new default or get discarded" — a decision Rounds currently hardcodes as "always ride on top," but which Helm treats as an explicit operator choice.
  • Terraform variable precedence runs from lowest to highest: the default in a variable block, then terraform.tfvars, then *.auto.tfvars files (lexical order), then -var-file flags, then -var CLI flags — a strict linear override chain with no merge step, illustrating the pure-override end of the spectrum (Terraform variable precedence).
  • CSS Cascade Layers (@layer) give web styling the same broad-to-narrow ordering, but crucially, "having declared your layer names ... you can add CSS rules to the layer by re-declaring the name — the styles are then appended to the layer" (MDN: Cascade layers; CSS-Tricks cascade layers guide) — i.e., within a layer, contributions append; across layers, later layers win outright regardless of selector specificity. This maps closely onto Rounds' two-tier structure: within the daily task's probe, the code default and the custom text both contribute (append); across scopes (global vs. task), the more specific scope's content is additive on top of, not a replacement for, the less specific one.
  • Jinja2 / Django template inheritance offers the most direct linguistic analogy: {% block %} defines an overridable region, {% extends %} establishes the parent-child relationship, and {{ super() }} is the explicit primitive for "render the parent's content at this point, then add mine" — i.e., append-by-explicit-request rather than override-by-default (Jinja2 template inheritance). Rounds' dailyProbeDefault + custom-appended-after is functionally {{ super() }} always called implicitly, with no way for a tenant to opt out of the parent content — a deliberate stricter choice than Jinja2's default (which lets a child block override without calling super() at all).
  • LangChain's PipelinePromptTemplate and .partial() mechanism formalize prompt composition as named, formatted sub-templates fed into a parent template as variables, explicitly built "to reduce redundancy" across a hierarchy of reusable prompt fragments (LangChain prompt composition) — the general-purpose plumbing that an append-semantics agent framework needs, whether or not it's actually built on LangChain.
  • OpenAI's instruction hierarchy work formalizes precedence between actors rather than between config layers — root > developer/system > user > tool — specifically to stop lower-privilege content from overriding higher-privilege instructions via prompt injection (The Instruction Hierarchy, arXiv:2404.13208; OpenAI Model Spec). It's a different problem (security precedence vs. product-config layering) but the same structural idea: an explicit, ordered stack where each layer's content coexists with layers above it rather than silently replacing them.

Propagation and Versioning

The reason override semantics tend to accumulate technical debt is that they force a binary choice at the moment a tenant customizes: either they get nothing from future default improvements (frozen snapshot), or the product team has to build a migration/merge tool to push updates into customized fields after the fact — genuinely hard, because merging free text is not like merging structured YAML.

Prompt-versioning practice in production LLM systems leans toward immutability plus pointers rather than in-place mutation: "once a prompt version is published to production, it must never be modified... rollback is changing a pointer, not redeploying code" (TianPan.co, Prompt Versioning and Change Management), with semantic-versioning conventions (MAJOR for structural/persona rewrites, MINOR for additive capability, PATCH for wording fixes) and canary/shadow rollout patterns targeting sub-15-minute (mature systems: sub-60-second) rollback. That discipline solves safe deployment of a new default, but it does not by itself solve whether existing customized tenants inherit that new default — that's a separate, product-level decision, and it's exactly the one Rounds resolved by making the default architecturally un-overridable (a code constant, injected unconditionally) rather than a mutable seeded value living in the same field the tenant edits.

Feature-flag systems hit an adjacent version of this problem from the other direction: a flag's evaluation falls back to an SDK-provided default when no rule matches, and "code drift" from stale flags is common enough that best practice is to remove a flag within a month of full rollout (ConfigCat: feature flags explained; Unleash: feature flag best practices) — a reminder that any layering system, prompt or otherwise, needs an active decommissioning discipline for old scopes, or the number of layers only ever grows.

Pitfalls

  1. Clearing a field with no runtime fallback silently drops behavior. If "custom probe" were the only source of daily probing (pre-fix Rounds, effectively, since the global default happened to carry it), a tenant blanking the field to "just try default behavior" would get no probing at all rather than reverting to a sane baseline — because there was no separate baseline, just one field wearing two hats. The fix's architectural move — a code-level default that is not stored in the same field the tenant edits — makes this failure mode structurally impossible: there is no value a tenant can type or delete that removes the code default.
  2. Duplicated guidance across layers. Append semantics risk a tenant re-stating something the default already says ("ask about completion status") without realizing it's redundant, bloating the composed prompt and occasionally producing mildly contradictory phrasing. Rounds mitigates this only by UI/documentation (labeling the field "appends on top of the default" — see the code comment in context.js: "custom probe_instruction (if any) APPENDS on top of it (append, not override) so product improvements to the default reach every team automatically"), not by any automated dedup — an open gap common to every append-based system reviewed here.
  3. Unbounded prompt growth from stacked layers. Each additional scope (global → task → per-member, say) that appends rather than overrides adds tokens unconditionally. A system with many scopes and append-everywhere semantics needs an explicit token/length budget check on the composed prompt, not just a functional test that the layers appear in the right order — Rounds' test suite currently checks ordering (probeIdx < customIdx) and presence/absence per task type, but not composed-length bounds.
  4. Precedence illegibility. The moment there is more than one layer, users need to be told, in the UI, what's baseline versus what's theirs — otherwise "why is the agent doing X" becomes undebuggable from the tenant's side. None of Kustomize, Helm, or CSS solves this for free either; all three ship dedicated inspection tooling (kustomize build, helm template, browser devtools' computed-style panel) precisely because a layered system without a "show me the resolved output" view is not auditable by the people using it.

Practical Guidance for Agent Frameworks with Editable Prompts

  • Prefer append for anything additive/behavioral; reserve override for whole-artifact fields (templates, schemas) where mixing is nonsensical.
  • Never let a single field double as both "the only source of a behavior" and "the tenant's customization surface." If a behavior must always exist, its default belongs in code (or a non-editable config layer), not in the same storage location the tenant's UI writes to.
  • Make empty-string a deliberate, tested value, distinct from null/fallback-to-default, wherever override semantics are used — and write a test asserting what an empty field actually does, per scope.
  • Test composition order explicitly, not just presence — assert that the default text appears before the custom delta in the rendered output, and that non-applicable scopes get neither.
  • Expose the resolved/composed prompt somewhere inspectable (an admin "preview" view, a --dry-run equivalent) so operators can see what layering actually produced, mirroring kustomize build / helm template.
  • Version the default separately from the tenant delta if the product is mature enough to need per-tenant rollout control (Helm's --reset-then-reuse-values pattern), rather than assuming "always append the latest default" is safe for every tenant at every moment — a sudden default change can still surprise a tenant who never asked for new behavior, even under append semantics.

Sources