Rolling Back Cache Keys After a Bad Deploy

A bad asset ships. Users report a broken page, a white screen, or a JavaScript error that did not exist in the previous release. The deployed HTML now references app.deadbeef.js — a fingerprinted file that is live on the CDN and in user browsers. The safest recovery is not a CDN purge of that file; it is re-pointing the HTML entry points to the previous hashed filename so that the old, known-good asset resumes being requested.

This page covers how to execute that rollback, what CDN behaviour to expect on Cloudflare, CloudFront, and Nginx, how to verify the recovery, and when to roll forward instead. For CI/CD-level rollback automation see the rollback guide for fingerprinted assets in CI/CD.

Symptom and Mental Model

The symptom is always visible to users before it is visible in monitoring: the HTML served to browsers references a broken asset. The cache key architecture of fingerprinted deployments means:

  1. The broken app.deadbeef.js has its own unique URL and its own CDN cache entry.
  2. The previous app.a1b2c3d4.js also has its own cache entry — it was never purged because the filename changed on deploy, making the old URL simply unreferenced rather than invalid.
  3. Users who already have app.a1b2c3d4.js in their browser cache are not affected. Users who have the new HTML (referencing deadbeef) are broken.

The rollback therefore has two components: (a) stop serving the broken HTML, and (b) confirm the old asset is still warm in the CDN.

Clarification: Why There Is No Hash Collision to Worry About

A fingerprinted filename encodes the content at the moment of the build. Rolling back means re-deploying the previous HTML, which already contains the correct hash (a1b2c3d4) for the previously built asset. There is no risk of collision with the broken asset (deadbeef) because both URLs exist independently in the CDN.

The only scenario requiring active cache management is if you must serve the broken HTML to no one — meaning you need the stale-HTML-serving CDN edge nodes to stop returning the broken document. That requires purging the HTML, not the assets.

Decision Matrix: Roll Back vs Roll Forward

Factor Roll Back (redeploy previous release) Roll Forward (hotfix in new deploy)
Time to recovery Seconds to minutes — no new build needed Minutes to tens of minutes — requires a build
Risk Low — known-good state Medium — hotfix may introduce regressions
CDN asset state Old asset already cached; HTML swap is instant New asset must propagate to all edge nodes
Broken asset cache entry Becomes unreferenced; expires via TTL Coexists harmlessly
When to choose Any incident where root cause is unclear Bug is trivially identified and fix is certain
CI/CD prerequisite Previous build artifacts still accessible Normal build pipeline availability

Default to roll back. A roll forward during an active incident requires confidence in the fix that is rarely available under pressure.

Roll back or roll forward From an alert, the single question is whether the previous release is known good. If it is, swapping the HTML restores service in about a minute; if it is not, a hotfix build is the only path and takes roughly ten minutes. INCIDENT PATH: ROLL BACK OR ROLL FORWARD Alert fires users report break Decision Previous release known good? ROLL BACK swap HTML only about 60 s to recovery ROLL FORWARD build + deploy fix about 10 min to recovery Verify curl HTML + asset Verify watch error rate When the root cause is unclear, roll back first and diagnose against a healthy production.
The branch is decided by one question only — whether a known-good previous release exists — and the recovery times differ by an order of magnitude.

Step-by-Step Rollback Procedure

Step 0: Bound the Blast Radius

Two minutes spent measuring who is affected saves a wasted rollback. Fingerprinted deployments fail asymmetrically: only sessions that fetched the new HTML are broken, and that population grows at the rate your HTML TTL allows. Read the actual number off the entry point rather than guessing:

# How stale can a served document be, and how far has the new one propagated?
curl -sI https://example.com/ | grep -iE "^(cache-control|age|cf-cache-status|etag):"

# Which asset hashes are live documents currently referencing?
for pop in 1 2 3 4 5; do
  curl -s -H "Cache-Control: no-cache" https://example.com/ \
    | grep -o 'main-[a-f0-9]\{8\}\.js' | sort -u
done | sort | uniq -c

If Cache-Control on the HTML is max-age=60 and Age is already past it, every new visitor is getting the broken document and the incident is fully rolled out. If the HTML TTL is 300 seconds and the deploy is 40 seconds old, most traffic is still on the previous document and you have a narrow window in which a fast roll-forward genuinely competes with a rollback.

The second command samples the hash actually embedded in the served HTML. Seeing two distinct hashes in the sample means the edge has not converged — some nodes still serve the old document — which changes the purge step from optional to mandatory.

Step 1: Identify the Previous Hashed Filenames

Locate your previous release’s asset manifest. Depending on your build tool:

# Vite: manifest lives at .vite/manifest.json inside the dist directory
# If you store build artifacts in CI, retrieve the previous run's artifact:
cat dist-previous/.vite/manifest.json | python3 -m json.tool | grep "file"

# Webpack: retrieve the previous webpack-manifest.json or stats.json
cat dist-previous/asset-manifest.json

# Generic: list files by modification time to confirm the previous hashes
ls -lt dist-previous/assets/*.js | head -20

If you use a deployment platform that stores artifact history (GitHub Actions artifacts, S3 versioning, Netlify deploy history), retrieve the previous release’s dist/ output directly.

Step 2: Re-deploy the Previous HTML

The critical file to replace is the HTML entry point (usually index.html) that contains <script src="..."> and <link rel="stylesheet" href="..."> tags referencing the broken hashes.

# Example: restore previous index.html from a tagged Git release
git show v2.4.0:dist/index.html > dist/index.html

# Then re-deploy only the HTML to your CDN origin or static host
# (the assets themselves do not need to be re-uploaded — they are already cached)

# Cloudflare Pages: push the previous commit or redeploy via API
curl -X POST "https://api.cloudflare.com/client/v4/accounts/$CF_ACCOUNT_ID/pages/projects/$PROJECT_NAME/deployments" \
  -H "Authorization: Bearer $CF_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"branch": "main"}'

# Netlify: redeploy a previous deploy ID
curl -X POST "https://api.netlify.com/api/v1/sites/$SITE_ID/deploys/$PREVIOUS_DEPLOY_ID/restore" \
  -H "Authorization: Bearer $NETLIFY_TOKEN"

For server-rendered or self-hosted deployments, replace the symlink that points to the active release:

# Atomic symlink swap — no downtime
ln -sfn /var/www/releases/v2.4.0 /var/www/current
# Reload the web server to pick up the symlinked change
systemctl reload nginx

Step 3: Purge the HTML Cache Entry

The assets do not need purging — only the HTML does. Purge the minimum set of URLs that serve the entry point HTML.

Cloudflare — purge specific HTML URLs:

curl -X POST "https://api.cloudflare.com/client/v4/zones/$CF_ZONE_ID/purge_cache" \
  -H "Authorization: Bearer $CF_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"files": ["https://example.com/", "https://example.com/index.html"]}'

CloudFront — create an invalidation for the HTML path only:

aws cloudfront create-invalidation \
  --distribution-id "$CF_DISTRIBUTION_ID" \
  --paths "/" "/index.html"

CloudFront invalidations cost $0.005 per path after the first 1,000 per month. Targeting only the HTML (one or two paths) keeps cost negligible. Do not invalidate /assets/* — those objects are immutable and have not changed.

Nginx proxy cache — purge the HTML entry point only:

# If using nginx-cache-purge module (ngx_cache_purge):
curl -X PURGE "https://example.com/"
curl -X PURGE "https://example.com/index.html"

# Without the purge module, move the cache file and reload:
find /var/cache/nginx -name "*.cache" -newer /var/www/current/index.html -delete
nginx -s reload

Step 4: Confirm the Old Asset Is Still Warm

Before the rollback completes, verify the previous good asset is still accessible at its original URL:

GOOD_HASH="a1b2c3d4"
ASSET_PATH="assets/main-${GOOD_HASH}.js"

curl -sI "https://example.com/${ASSET_PATH}" \
  | grep -E "^(http|cf-cache-status|x-cache|cache-control|age):" -i

Expected output:

HTTP/2 200
cache-control: public, max-age=31536000, immutable
cf-cache-status: HIT
age: 172800

A HIT with a positive Age: confirms the old asset is still warm in the CDN. A MISS means the TTL expired (unusual within 24 hours of a deploy) or the asset was never deployed to this CDN edge. In the rare MISS case, the origin will re-serve the file if you kept the old asset in origin storage — which you should always do for at least one release cycle.

Fingerprinted asset rollback states Before the rollback the HTML entry point references the broken hash and browsers receive the broken asset. After redeploying the previous HTML and purging it, the same CDN serves the previous good hash, which was never evicted. BEFORE ROLLBACK — broken state index.html refs app.deadbeef.js CDN Cache app.deadbeef.js — live app.a1b2c3d4.js — warm Browser executes the broken bundle ROLLBACK ACTION redeploy + purge HTML AFTER ROLLBACK — restored state index.html refs app.a1b2c3d4.js CDN Cache app.deadbeef.js — unreferenced app.a1b2c3d4.js — HIT Browser executes the previous bundle
A rollback replaces only the HTML entry point and purges its cache entry; the previous good asset was never evicted and resumes serving from the existing CDN object.

Verification Command

After completing Steps 1–4, run a single end-to-end check:

#!/bin/bash
# rollback_verify.sh
# Usage: DOMAIN=example.com GOOD_HASH=a1b2c3d4 ASSET_NAME=main bash rollback_verify.sh

set -euo pipefail

DOMAIN="${DOMAIN:?set DOMAIN}"
GOOD_HASH="${GOOD_HASH:?set GOOD_HASH}"
ASSET_NAME="${ASSET_NAME:?set ASSET_NAME}"

ASSET_URL="https://${DOMAIN}/assets/${ASSET_NAME}-${GOOD_HASH}.js"
HTML_URL="https://${DOMAIN}/"

echo "=== Verifying HTML references prior good hash ==="
curl -s "$HTML_URL" | grep -o "${ASSET_NAME}-[a-f0-9]\{8\}\.js" | sort -u

echo ""
echo "=== Verifying asset is still cached ==="
curl -sI "$ASSET_URL" | grep -E "^(http|cf-cache-status|x-cache|cache-control|age):" -i

echo ""
echo "=== Verifying broken asset no longer referenced ==="
BROKEN_COUNT=$(curl -s "$HTML_URL" | grep -c "deadbeef" || true)
echo "References to broken hash in HTML: $BROKEN_COUNT (expect 0)"

Running this script should show the HTML pointing to the good hash, the CDN returning a HIT for that asset, and zero references to the broken hash.

How Long a Release Stays Rollback-Ready

The rollback works because nothing ever deleted the previous asset. That guarantee has three independent expiry clocks, and the shortest one sets the real window.

Rollback readiness window The edge cache object survives for the full one-year TTL, the origin copy is deleted by a thirty-day cleanup job, and the stale HTML document expires within minutes. The shortest clock that matters for recovery is the HTML TTL. HOW LONG THE PREVIOUS ASSET STAYS ROLLBACK-READY incident detected Edge object alive — TTL 365 d, no purge issued Origin copy retained — 30-day cleanup job Broken HTML still cached at the edge — 5 min TTL deploy 5 min 30 d 365 d Recovery needs only the edge object; the origin copy is the safety net for cold edge nodes.
Three clocks run in parallel after a deploy — edge TTL, origin retention, and the HTML TTL — and only the last one is short enough to matter during an incident.

The edge object. With max-age=31536000, immutable and no purge, the previous asset stays resident at every edge node that served it, subject only to least-recently-used eviction. Eviction is realistic on a low-traffic PoP for an asset nobody has requested in hours, which is precisely the situation during a rollback: for the duration of the bad deploy, nobody was requesting the old hash.

The origin copy. This is the clock people get wrong. Build pipelines that aws s3 sync --delete or that write into a clean directory remove the previous release’s files at deploy time, so an edge miss on the old hash becomes a 404 instead of a re-fetch. Sync without --delete for asset prefixes, and let a scheduled lifecycle rule reclaim objects after 30 days:

# Deploy assets additively — never remove the previous release's hashed files
aws s3 sync dist/assets/ "s3://$S3_BUCKET/assets/" \
  --cache-control "public,max-age=31536000,immutable" \
  --metadata-directive REPLACE

# Reclaim space on a schedule instead, so rollback targets survive an incident
aws s3api put-bucket-lifecycle-configuration \
  --bucket "$S3_BUCKET" \
  --lifecycle-configuration '{
    "Rules": [{
      "ID": "expire-old-fingerprinted-assets",
      "Filter": { "Prefix": "assets/" },
      "Status": "Enabled",
      "Expiration": { "Days": 30 }
    }]
  }'

The HTML object. The shortest clock, and the only one you actively manage during recovery. Everything in the rollback procedure exists to shorten it.

When to Reconsider (Roll Forward Instead)

Roll forward — fixing the bug and deploying a new release — is appropriate when:

  • The bug is a single-line fix with zero risk of regression and the build pipeline is fast (under two minutes). Rolling forward resolves the incident cleanly rather than leaving an unreferenced broken asset in CDN storage.
  • The previous release also had a bug that you are rolling back to. A rollback to a previously known-good state is safe; a rollback to a state that also had problems trades one incident for another.
  • The breakage is in the HTML template itself rather than the asset content. A new deploy with the corrected template is simpler than re-deploying an old HTML version that may also contain the template error.
  • You are mid-migration between query-parameter versioning and filename hashing. In that case, the old filenames may not exist on the CDN because the previous strategy used different cache keys. Confirm old assets are accessible before committing to a rollback path.

CDN-Specific Considerations

Cloudflare: HTML purge is near-instant globally via the API. Old fingerprinted assets cached at the edge remain available until their max-age TTL expires — typically one year. No action needed for the assets.

CloudFront: Invalidation propagation takes 10–60 seconds globally. Until the invalidation completes, some edge nodes may still serve the broken HTML. To minimise exposure, include both / and /index.html in the invalidation. The old asset remains cached at every edge node where it was requested — it is never actively evicted during normal operation.

Nginx (self-hosted): Proxy cache purge requires either the ngx_cache_purge module or a manual cache-file deletion. The most reliable approach is an atomic symlink swap (ln -sfn) for the serving directory followed by nginx -s reload, which clears the active configuration without downtime. Old asset files on disk at the origin remain accessible for the duration of their CDN TTL.

Frequently Asked Questions

Should I purge the broken asset URL as part of the rollback?

No, and doing so can make the incident worse. Purging app.deadbeef.js does not un-break the browsers that already downloaded it, and it turns every request from a still-stale document into an origin fetch — adding origin load at the exact moment you least want it. The broken object is harmless once no HTML references it; leave it to expire. The only URLs worth purging are the HTML entry points.

What if the previous asset returns a MISS instead of a HIT?

A MISS means that edge node no longer holds the object, usually because it was evicted while nobody was requesting it during the bad deploy. This is recoverable as long as the file still exists in origin storage: the first request repopulates the edge and every subsequent user gets a HIT. It is only fatal if your deploy step deleted the previous release’s files, which is why additive sync and a lifecycle rule matter more than any purge command.

Does a rollback break users who already loaded the broken page?

Their current tab keeps running the broken bundle until they reload — a rollback changes what the server sends, not what a running page has already executed. If the failure is severe enough that you cannot wait for organic reloads, ship a tiny always-revalidated script from the HTML that compares a build identifier against a known-good value and forces location.reload(). Keep that script on Cache-Control: no-store so it is never itself stuck at a stale version.

How does this differ when assets are versioned by query parameter?

Substantially, and for the worse. Under query-parameter versioning the previous asset shares a base path with the broken one, so restoring the old ?v= value may resolve to whichever bytes the edge happens to hold for that path. Recovery then requires an asset purge as well as an HTML purge, and the window between the two is a window of undefined behaviour.