Created on 2025-09-27 13:15
Published on 2025-09-27 13:19
If you’ve been anywhere near a deployment pipeline lately, you’ve heard the dueling philosophies. One camp swears by trunk-based development and continuous delivery: smaller changes, shipped more often, with automated checks catching issues before they graduate to “call the on-call”. The other camp counters with a hard stare and a badge that reads “regulated domain”, insisting that human approvals, formal change windows, and an honest-to-goodness Change Advisory Board are the thin line between business continuity and headline-grabbing outages—or worse, audits that end in footnotes and fines.
Both sides are right about the same thing: change is where risk lives. SREs and DevOps folks trade in risk economics: push risk left with automation, absorb residual risk with guardrails, and put just enough brakes on to keep production from catching fire. The trick is picking brakes that slow you down the least.
Data helps. DORA’s research, now across a decade, has consistently found that heavyweight, external change approvals correlate with worse delivery outcomes, and there’s no evidence these ceremonies reduce change failure rates. That doesn’t mean “no approvals ever”; it does mean the type and timing of approvals matter more than the ritual.
Of course, if your service lives under a regulator’s umbrella, you don’t get to “no ritual”. Many control frameworks expect documented change control, sometimes via boards and formal authority. NIST SP 800-53’s CM-3 even name-checks configuration control boards and formal approval records. PCI DSS requires documented approval for certain production changes. In FDA-regulated worlds, 21 CFR Part 11 pushes for auditable electronic records and signatures that map decisions to humans with privileges. You can’t automate your way out of evidence. You can, however, automate the evidence itself.
When a release goes sideways, we rarely wish for a bigger meeting. We wish for smaller changes, faster rollbacks, and a crisp error budget policy that stops risky changes before pagers start singing. Google’s SRE playbook is explicit: if you’re burning error budget, you freeze changes, redirect engineering energy to reliability, and only lift the freeze when the service is back within SLO. That’s governance that bites, not barks.
Trunk-based development doubles down on this idea by shrinking the batch size of change. At truly large scale—think Google’s monorepo world—working close to trunk with code review and heavy automation reduces integration hell and keeps changes reviewable. There is nothing magical about “trunk”; the magic is in quick integration, peer review, and guardrail automation that promotes or blocks changes based on signals, not vibes.
Side A sounds like your favorite platform engineer: “Automate checks, ship tiny changes, and let the pipeline decide. Humans are great at context, terrible at consistency.” DORA backs them: external approvals slow delivery and don’t correlate with fewer failed changes. Side A’s secret weapon is progressive delivery—canaries, feature flags, targeted rollouts—and automated rollback when health metrics go sour. That last bit is the on-call gift you buy yourself.
Side B is the person who has actually sat across from an auditor. They’ll quote chapter and verse about change authorities, recorded approvals, segregation of duties, and maintenance windows that align with business risk. They are not wrong. Frameworks like NIST SP 800-53 and standards like PCI DSS explicitly call for documented approvals and retained change records. The question isn’t “approval or not?” It’s “what risk needs which approval, and how do we make the happy path self-service?”
At 02:07, FinCo’s core payments API spikes 5xx errors. Their model is trunk-based CD with canaries. The canary sees a 1.4% error delta, trips an automatic pause, and the pipeline rolls back in under five minutes while paging the team. Error budget policy toggles a release freeze until reliability returns to SLO. Postmortem points to a config migration; a safer, incremental rollout plan is templated for next time. No one calls a CAB; the guardrails were the approval.
At 03:33, HealthCo’s device telemetry platform—subject to FDA audits—hits a data ingestion delay. Their change freeze window is active, so the only allowed change is a P0 fix through the emergency path. The emergency change authority, defined in policy, is a small on-call quorum with electronic signature capture and an automatic evidence package: Git commit with reviewers, CI logs, test artifacts, SBOM for updated libraries, and a signed provenance attestation. The team applies a safe config rollback, documents the approval, and ships a corrective CAPA record for the auditor. Release speed is, by design, a little slower; the evidence trail is, by design, a lot stronger.
Both are, as long as you connect governance to risk and turn policy into automation. ITIL 4 itself reframed “Change Management” to “Change Enablement”—help changes happen safely, not stop them. SRE takes that spirit further: use error budgets and SLOs to dynamically dial velocity. The goal isn’t to eliminate humans; it’s to ensure human judgment is saved for ambiguous, high-risk change—while the pipeline proves low-risk changes safe at speed.
Everyone can list metrics; fewer teams wire them up end-to-end. Start with the industry’s shared language, then add the ones your auditors care about.
Change failure rate is the percentage of production deployments that result in an incident or require rollback/hotfix. It’s best calculated by joining deployment events with incidents, not by guessing from alert volume. DORA and Google’s Four Keys patterns lay out the join logic clearly—your CI/CD system becomes the source of truth, your incident tracker supplies outcomes, and the pipeline links them by deployment ID.
Mean lead time for changes is the stopwatch from commit to running in production. Track it at the artifact level so you can graph each step—build, test, deploy—and see where time burns. The point isn’t shaming anyone; it’s making the bottleneck visible enough to fix. DORA formalized this in the four keys a long time ago, and it remains the most useful “speed” indicator that doesn’t reward risky shortcuts.
Rollback frequency is not one of the classic four, but SREs treat it as a vital “early warning” of release quality. If your canary or flags roll back often, you’re catching risk late. If they don’t roll back but incidents climb, your gates are too lenient. Track the ratio of rollbacks to total deployments and tag the cause: metrics regression, error rate spike, user behavior change, or security signal. Use progressive delivery tooling to automate both the decision and the record.
Audit findings are the uncomfortable metric that matters in regulated spaces. Count not just total findings, but repeatfindings and the time-to-remediate. Map each finding to a control family and a pipeline step—did the issue occur because approvals lived in Slack screenshots instead of tamper-evident logs? Did you lack SBOMs or build provenance to satisfy your software supply chain obligations? That mapping turns “audit” into a backlog you can burn down with code.
Here’s the fun part: you can combine the speed of trunk-based CD with the certainty of gated releases by moving gates into code and making them risk-aware.
First, adopt a risk-based change policy that classifies changes automatically. Low-risk, reversible changes with proven tests and blast radius controls should auto-approve through your pipeline. Normal changes require peer review, security scans, and maybe a canary bake time. High-risk or safety-critical changes escalate to a human authority with clear criteria. This is not hand-wavy process art; NIST’s CM-3 and SSDF are comfortable with risk-tiered controls, recorded approvals, and retained evidence—as long as you keep the receipts.
Second, codify guardrails in the pipeline. Use policy-as-code to check for code review, test coverage thresholds, vulnerability budgets, SBOM presence, and signed build provenance before promotion. When something fails, the pipeline says “no” and explains why, creating an audit trail as a side effect of doing the right thing. That’s governance that scales with your repo, not with your meeting calendar.
Third, ship with progressive delivery by default. Canaries and feature flags aren’t just safety nets; they’re levers for learning. If the canary’s error rate deviates from control, roll back automatically and open an incident linked to the deployment. If a flagged feature hurts latency or user engagement, dial back exposure instead of redeploying. Your rollback frequency becomes a learning loop rather than a scarlet letter.
Fourth, enforce release freezes with error budgets, not with the Gregorian calendar. Maintenance windows are fine when tied to real business constraints, but nothing is as honest as “we overspent the budget—stop shipping.” This keeps on-call humans out of philosophy debates and puts them into reliability work when it’s needed most. If you must have blackout periods, measure their impact on error budgets and shorten them over time.
Finally, make evidence generation boring. Every promotion should attach its attestations: who reviewed, what tests ran, what version was built, which SBOM covers it, which SLSA provenance proves the build’s integrity, which SLOs were green at deploy time, and which authority (human or automated) approved it. When the auditor shows up, you export a bundle rather than compile a novella.
If a CAB approves a change no one understands, is the risk reduced or just redistributed onto the on-call?
At what rollback rate would you halt all feature work for a sprint—and would your product partner thank you or fire you?
If your error budget policy never triggers a freeze, do you have fantastic reliability or dangerously forgiving SLOs?
Where should the “change authority” live for infra-as-code that reconfigures a regulated data flow—the platform team, the service owner, or a cross-functional quorum?
If your trunk-based pipeline produces perfect evidence automatically, what’s the narrowest set of changes that still deserve a live CAB slot?
A team once invited me to their weekly CAB. They dutifully presented a ten-page change record to explain a one-line config tweak that would be deployed by an anonymous “batch process.” We replaced that ritual with a pipeline step that refused to ship unless the change ticket linked to a commit with an approving review and the deployment referenced the ticket ID. Two weeks later, the same folks were laughing about how the audit practically wrote itself because the artifacts were attached to each release. The ritual wasn’t wrong; it just belonged in code.
Another shop had heroic trunk-based CD but a habit of “let’s just see” canaries. After three noisy rollbacks in a week, they finally defined concrete success/failure thresholds for latency and error rates, and wired rollback into the gate. Rollback frequency dropped, and—more importantly—so did incident volume. When the canary said no, it meant no, not “let’s keep our fingers crossed.”
Release governance is not about picking a side; it’s about choosing a feedback loop. Trunk-based CD says “shrink changes and learn quickly.” Gated releases say “prove it’s safe before you touch prod.” The SRE answer is “both”—prove safety with automation, turn humans into escalation points for ambiguity, and let error budgets and SLOs steer when to speed up and when to stop. If your governance makes the right thing the easiest thing, it will keep working long after the CAB meeting ends and the on-call falls back asleep.
Streamlining Change Approval (DORA Capabilities) — https://dora.dev/capabilities/streamlining-change-approval/
Why Google Stores Billions of Lines of Code in a Single Repository — https://research.google/pubs/why-google-stores-billions-of-lines-of-code-in-a-single-repository/
ITIL 4 Practitioner: Change Enablement (Axelos) — https://www.axelos.com/resource-hub/blog/itil_4_practitioner_change_enablement
Secure Software Development Framework (SSDF) v1.1 — NIST SP 800-218 — https://nvlpubs.nist.gov/nistpubs/specialpublications/nist.sp.800-218.pdf
NIST SP 800-53 Rev. 5 — CM-3: Configuration Change Control — https://csf.tools/reference/nist-sp-800-53/r5/cm/cm-3/
#SRE #SiteReliability #DEVOPS #DevOps #ChangeManagement #ITIL #ContinuousDelivery #TrunkBasedDevelopment #ProgressiveDelivery #ErrorBudgets #DORA #Compliance #PCI #NIST #SBOM