Repository Archaeology: Mining Git History for Honest Engineering Retrospectives

Executive Summary

Every engineering organization eventually faces the same quarterly ritual: someone has to explain what the team actually did. Increasingly, that someone is starting from git log instead of memory — and increasingly, that someone is an AI agent, pointed at an org's repositories and asked to produce a narrative. This is repository archaeology: excavating commits, tags, releases, and PR/issue metadata to reconstruct what happened, at what pace, and (the hard part) whether it mattered.

The practice sits on genuinely contested ground. A cottage industry of engineering-intelligence vendors (LinearB, Swarmia, DX, Sleuth, GitClear, Code Climate Velocity, Jellyfish) has built dashboards on top of exactly this data. The DORA research program legitimized four (now five) metrics derived substantially from commit and deploy events. And a parallel, louder literature — Kent Beck and Gergely Orosz's rebuttal of McKinsey, GitClear's own code-churn research, a decade of "Goodhart's law in software" essays — argues that most of what's easy to count from git is a poor proxy for value, and gets worse the moment it becomes a target.

The honest position is that VCS data is real evidence of activity, weak evidence of effort, and almost no evidence of impact on its own — but it becomes genuinely useful when mined with discipline: proper windowing, bot and import discounting, release-density literacy, and every quantitative claim tied to a qualitative check. AI agents make this both more necessary and more dangerous: necessary because they can do the tedious cross-repo sweep a human never would, dangerous because a fluent narrative generated from commit messages reads as authoritative regardless of whether it's true. The rest of this piece works through the landscape, the evidentiary limits, a concrete methodology, the AI-specific failure modes, how real projects already do this well (Kubernetes, Rust, "brag documents"), and the governance questions — gaming and privacy — that follow the moment this becomes routine.

The Practice Landscape: Who Sells Git-Derived Truth

A distinct product category — sometimes called "engineering intelligence" or "software engineering intelligence" (SEI) — exists specifically to convert VCS and issue-tracker exhaust into management-facing dashboards. LinearB leans on deep Git integration, cycle-time breakdowns (coding, review, deploy) and "investment allocation" reporting that buckets commits against ticket categories. Swarmia pairs DORA metrics with developer-experience signals and "working agreements" about PR size and review turnaround. Sleuth and Code Climate Velocity focus tightly on deployment-event tracking and code-level analytics respectively. DX, founded by Abi Noda with Laura Tacho and DORA/SPACE authors Nicole Forsgren and Margaret-Anne Storey as advisors, publishes the DX Core 4 framework — speed, effectiveness, quality, and business impact, 16 metrics total, explicitly framed as a synthesis of DORA, SPACE, and DevEx research, benchmarked across 300+ organizations (see the DX Core 4 announcement and benchmarks). Notably, a recurring criticism of this whole vendor category is that none of them can cleanly separate AI-generated code from human-written code in their commit-derived metrics, which undercuts exactly the AI-ROI story customers now want from them.

Underneath all of it sits DORA (DevOps Research and Assessment), the Google-sponsored research program that gave the industry its four canonical delivery metrics: deployment frequency, lead time for changes, change failure rate, and time to restore service — all derivable, in principle, from commit timestamps, deploy events, and incident/rollback tags. In 2024 DORA added a fifth, deployment rework rate (the percentage of deployments that are unplanned remediation of a prior incident), explicitly because change failure rate alone was a noisy proxy for how much work is corrective rather than new (see DORA's own history of the metrics). The 2025 DORA report — the first to look hard at AI's effect on these metrics — found that AI adoption tends to raise throughput while simultaneously increasing delivery instability, and frames AI's effect as an amplifier of whatever organizational strengths or dysfunctions already exist rather than a fix in itself.

The SPACE framework (Forsgren, Storey, and Microsoft Research colleagues, 2021) is the standing counterweight inside the same research lineage: Satisfaction, Performance, Activity, Communication, Efficiency — five dimensions explicitly designed to stop teams from reducing "productivity" to a single activity count. Its own authors state plainly that productivity "cannot be measured by a single metric or dimension," and that lines-of-code or speed-only measurement is misleading (see the Microsoft Research SPACE paper). In practice, the framework is applied narrowly: reviews of AI-productivity studies find roughly 90% invoke at least two SPACE dimensions but only 15% touch more than three, and the Communication and well-being dimensions are the most consistently skipped — precisely because Activity (commits, PRs, LOC) is what's cheap to instrument from git.

What VCS Data Can and Cannot Evidence

Git history is unusually good at answering when and how much surface area — timestamps are trustworthy, diffs are exact, tags mark real release boundaries. It is close to useless, on its own, at answering why or how much it mattered.

Commit counts vs. impact. GitClear — a vendor that has built its research reputation on being skeptical of its own industry's favorite metric — states plainly that "commits and lines of code don't convey much about the habits of individual developers. Some developers spend hours planning and make a single commit for a large feature. Others make many small commits" (GitClear, Code Analysis: Beyond Lines of Code). Their peer-reviewed-adjacent research argues a "Diff Delta" measure correlates better with actual effort than raw commit or LOC counts.

Release cadence as a delivery signal — with a catch. More releases can mean either a healthier, smaller-batch delivery pipeline (the DORA-endorsed direction) or a symptom of scope thrash and incident-driven hotfixing (the deployment-rework-rate problem DORA added in 2024 specifically to catch). Reading release density requires knowing why releases happened, not just how many there were — a distinction no tag list encodes.

The code-churn evidence. GitClear's longitudinal analysis of 211 million lines of code changes (their 2025 AI Copilot Code Quality report) is the sharpest empirical case that raw commit/line volume can move in the wrong direction relative to quality: cloned ("copy/pasted") code rose from 8.3% to 12.3% of lines between 2021 and 2024, with an eightfold jump in duplicated-block frequency in 2024 alone; code revised within two weeks of being written ("churn") rose from 3.1% to 5.7% over the same window; and refactored ("moved") lines — the signature of intentional cleanup — fell from 24.1% to 9.5%. All three trends coincide with rising AI-assisted commit volume. If a retrospective mining pipeline reports "commits per engineer up 18% this quarter" without this context, it is very plausibly reporting a decline dressed as a gain.

The McKinsey debate. In August 2023, McKinsey published "Yes, you can measure software developer productivity," proposing an org-wide productivity index built substantially from output measures like PR throughput. Kent Beck and Gergely Orosz's two-part rebuttal in The Pragmatic Engineer became the industry's reference takedown. Their core argument is an effort → output → outcome → impact chain: effort (planning, coding) and output (commits, PRs, features shipped) are cheap and tempting to measure but easiest to game and least correlated with what the business actually cares about; outcome (customer response) and impact (revenue, retention) are harder to measure but are what should drive decisions. Beck's blunter line — "the report is so absurd and naive that it makes no sense to critique it in detail" — and his account of Facebook's internal survey scores getting rolled up into director-level pressure, which then cascaded into managers negotiating with ICs over their own self-reported numbers, is the canonical case study of what happens when effort/output metrics get institutional teeth. Their prescription: measure the way sales teams do — closed deals, not calls dialed — i.e., track outcomes even though outcomes are harder to attribute to git activity.

Goodhart's law, concretely. "When a measure becomes a target, it ceases to be a good measure" is not an abstract warning in this domain — it is observed behavior. Once commit count is watched, commits get split into smaller pieces; once story points are watched, estimates inflate; once code coverage is a gate, tests get written to hit lines rather than catch bugs. The mechanism is structural, not a matter of bad actors: any metric derived from an activity that the people being measured also control will be optimized toward the metric once stakes attach to it (see Jellyfish's writeup and Hillel Wayne's essay). This is precisely why retrospective mining — a backward-looking, low-stakes narrative exercise — is a much safer use of VCS data than forward-looking individual scorecards.

Methodology for Honest Retrospective Mining

If VCS data is going to be the evidence base for a retrospective, the mining has to be defensive by design. A working methodology:

1. Window discipline. Define the reporting window against merge/release timestamps on the default branch, not commit-authored dates — rebases, cherry-picks, and long-lived feature branches routinely backdate authored timestamps well outside the window they actually landed in. State explicitly whether the sweep is "commits merged to main between dates X and Y" or something looser; these produce materially different counts.

2. Default-branch vs. all-branch counting. All-branch commit counts double- or triple-count work that gets rebased, squashed, or abandoned before merge; they also pick up long-dead experimental branches that never shipped. Default-branch-only counting undercounts genuine collaborative effort (pairing commits, WIP snapshots) but is far more defensible as a "what shipped" signal. Pick one, and say which.

3. Bot and content-commit discounting. Dependency-bump bots (Renovate, Dependabot) and doc/content pipelines can dominate raw commit counts without representing engineering judgment at all — Renovate deliberately keeps to one commit per update branch, while less-configured Dependabot setups can generate one PR per dependency across dozens of repos (Renovate vs. Dependabot comparison). A repo that is mostly markdown/content (docs sites, changelogs, translation repos) will show commit velocity indistinguishable from a shipped-feature repo unless author identity and file-path patterns are used to separate bot/content commits from engineering commits before aggregating across an org.

4. Imported-history pitfalls. A repository created mid-window that inherits a prior project's full history (via git subtree, history-preserving import, or a monorepo merge) will show a commit graph stretching years before the repo "existed" in the org's system of record. Any org-wide sweep needs to detect and exclude or flag these — either by repo-creation-date vs. first-commit-date mismatches, or by explicit allowlists of "this repo's real start is commit X."

5. Release-density interpretation, not just counting. Many small releases and few large releases can both look like "N releases this quarter" — the number is meaningless without breaking down release size (files/lines changed), whether releases were scheduled or hotfix-triggered, and change-failure-rate context per release. This is the direct git-mining analogue of DORA's 2024 deployment-rework-rate addition: raw deployment frequency without a rework/hotfix split misrepresents stability as velocity.

6. Detecting abandoned or paused work streams. Activity cliffs — a branch, module, or repo that had steady commits and then goes silent — are a real and useful retrospective signal (a project was deprioritized, a person left, a dependency blocked progress), but only if distinguished from normal completion (the work shipped and moved to maintenance mode). Academic work on OSS abandonment prediction is directly relevant here: studies modeling repository survival find raw repo-level activity metrics are unreliable abandonment proxies on their own, and combine timeline-based behavioral features (contributor turnover, issue response latency, dependency health) rather than commit-count alone (see the 2025 abandonment-prediction paper and the empirical survival-rate study from MSR 2022, which notes many OSS projects die within their first year and survival odds decline yearly thereafter).

7. New-repo birth/death rates as a portfolio signal. At the org level, counting repos created vs. archived per quarter is a legitimate proxy for experimentation rate and technical-debt cleanup discipline — but it inherits every problem above (imported history inflating "new" repo commit counts, bot-generated scaffold repos, forks miscounted as new projects) and should always be reported alongside repo survival to the next window, not just creation counts.

The AI-Agent-Specific Angle

The practice this article is named for — an agent pointed at an org's repositories and asked to draft a quarterly retrospective — is a genuinely new capability, not just automation of an old spreadsheet. It changes two things: the scale of the sweep (an agent can walk every repo an org owns, something no human analyst does by hand every quarter) and the authorship of the narrative (a fluent paragraph is generated, not just a number).

The authorship shift is where the risk concentrates. LLMs narrating commit intent from commit messages and diffs are prone to a specific hallucination pattern: inferring a coherent, purposeful story ("the team focused on hardening the auth layer this quarter") from what may actually be a scattershot of unrelated small fixes, because commit messages are terse and diffs alone under-specify motivation. Current practitioner and research work on this converges on a few concrete mitigations:

Compute, don't guess. Where a fact is computable from the repo (files changed, lines added/removed, merge date, PR review count), the agent should compute and pass it explicitly rather than let the model infer it from a truncated diff — directly analogous to a finding in recent work on LLM-based commit summarization, that feeding raw git-blame data straight to a model caused hallucination and information loss, versus a structured intermediate representation (see arXiv 2505.17710, LLM Contribution Summarization and arXiv 2603.14619 on LLM-augmented release intelligence).
Evidence-linking as a hard requirement. Every narrative claim in the output — "the team shipped three major features," "release cadence doubled" — should carry an inline citation to the specific commit SHA(s), PR number, or release tag it's based on. This is the single highest-leverage mitigation: it converts an unfalsifiable summary into a checkable one, and it is exactly the discipline good technical writing already demands of human authors.
Pair quantitative sweeps with qualitative checks. A git-mining sweep should be treated as a first draft that a human (or a second-pass verification agent) reconciles against what the team actually remembers happened — the same "bottom-up retro" instinct that makes stand-up-style retrospectives useful, applied to whether the quantitative story matches lived experience. An agent-generated retro that nobody on the team reviewed before publication is a narrative with no error-correction step.
State the counting method up front. Given the methodology section above (branch scope, bot discounting, window boundaries), an AI-drafted retrospective should open with its own methodology note — what was counted, what was excluded, and why — so a reader can independently judge whether a "quiet quarter" claim reflects real quiet or a counting artifact (a repo migration, an import, a maintenance-mode transition).

Recent academic framing of "agentic repository mining" (arXiv 2605.04845) treats this as a multi-task evaluation problem precisely because summarization, impact attribution, and timeline reconstruction are different tasks with different failure modes — collapsing them into one undifferentiated "summarize the quarter" prompt is where hallucination risk concentrates.

Real-World Practices Worth Copying

Some of the best retrospective practices predate the AI-agent framing entirely and are worth studying as ground truth for what "good" looks like.

Kubernetes release retrospectives. SIG Release runs a structured, multi-part retrospective for every release cycle (documented per-cycle in the sig-release repo) — an initial retro early in the burndown period, with follow-up retro sessions in subsequent weeks, facilitated by a designated Emeritus Adviser role. Release notes themselves are produced by a dedicated Release Notes sub-team continuously drafting through the cycle, separate from the Documentation sub-team that publishes final copy to kubernetes.io. The structural lesson: the qualitative retro (what went wrong in the release process) and the quantitative changelog (what shipped) are deliberately separate workstreams with separate owners, not one document trying to do both jobs.

Rust's annual survey plus git data. The Rust language's annual community survey (9,450 participants invited, 7,310 completions in the 2024 edition) is explicitly a qualitative instrument — sentiment, adoption, pain points — that the project runs alongside its git/GitHub activity data rather than instead of it, and separate academic work has specifically studied Rust's contributor demographics and collaboration patterns using the commit graph (see arXiv 2503.22066). The combination — survey for outcome/satisfaction signal, git mining for activity/collaboration-structure signal — is a working instance of the SPACE framework's insistence that no single data source suffices.

Brag documents vs. automated evidence gathering. Julia Evans' popularization of the "brag document" — a running personal log of accomplishments kept by the individual, precisely because "managers do not know what stuff you did, actually" — is the individual-level, human-curated counterpart to org-level git mining. It is instructive that the brag-document culture explicitly favors self-reported narrative over automated commit-count extraction: an individual knows which of their 40 commits mattered and which were noise, in a way no external miner can reconstruct from the graph alone. Staff-plus promotion packets (staffeng.com's guide) formalize this further — engineers are advised to "leave breadcrumbs" (design docs, project timelines, code contributions) deliberately, because the promotion committee's evidence base is explicitly not just the commit graph; it's curated artifacts the engineer chose to preserve. Both practices point the same direction as the AI-agent methodology above: automated mining is a good first pass, but the subject's own account is the correction mechanism, not an afterthought.

Failure Modes and Governance

Two failure modes recur once repository archaeology becomes routine rather than occasional.

Gaming once mining becomes a target. The moment engineers know a quarterly retrospective — visible to leadership, tied loosely to review cycles — is generated from commit/PR counts, Goodhart's law activates: commits get split, PRs get inflated with trivial diffs, releases get cut more frequently but thinner. The mitigation isn't better counting; it's institutional. Retrospectives mined from git should be explicitly scoped as team-level, backward-looking, narrative artifacts — closer to Kubernetes' retro model than to a scorecard — with no direct line from an individual's commit graph to their compensation or ranking. The moment that line exists, the data quality of the mining degrades on contact.

Privacy and surveillance at the individual level. Individual-level mining of commit timestamps, PR turnaround, and review latency is, functionally, workplace behavioral monitoring, and in several jurisdictions that triggers real legal machinery before it can even be deployed. In Germany, Section 87(1) no. 6 of the Betriebsverfassungsgesetz (BetrVG) gives works councils co-determination rights over any technical system capable of monitoring employee behavior or performance — a threshold broad enough to catch engineering-analytics dashboards, and case law treats skipping works-council agreement as grounds for injunctions and fines (see the works-council/data-protection overview and BetrVG/GDPR compliance guide). Under GDPR/BDSG, behavioral data about identifiable individuals is personal data requiring a lawful basis and, in Germany, typically a works agreement; German DPAs generally recommend routine monitoring data retention capped at 3–6 months. The practical governance answer that most of the credible literature converges on (Jellyfish, DX, the Goodhart's-law essays alike) is the same one Beck and Orosz reach from a pure productivity-measurement angle: aggregate at the team level, never surface individual-level git activity in a retrospective artifact, and treat any tool that defaults to per-engineer leaderboards as a governance red flag independent of whether the metrics are technically accurate.

The throughline across both failure modes is the same: git history is an excellent record of what happened to the code and a poor, easily-corrupted record of who should get credit or blame. Retrospective mining works when it stays disciplined about which of those two questions it's actually answering.