Rolling Back a Bad Asset Deploy on CloudFront
A broken JavaScript bundle shipped. Users see a blank page or a runtime error. The fix is already staged but the current release is live and CloudFront is serving it globally. This guide walks through the fastest safe rollback path: restore the prior index.html from S3, invalidate that one path, and rely on the fact that all prior hashed assets are still in S3 and still cached at the edge.
Why CloudFront Rollbacks Are Simpler With Fingerprinted Assets
The key insight behind a surgical rollback is that content hashing makes each release’s assets independent objects in S3. When you deployed main.a1b2c3d4.js in the bad release and main.9e2f1a3b.js was the prior good release, both files exist in S3 and both may still be cached at CloudFront edge POPs. Rolling back means pointing index.html back to the old hash — the file it references is already present.
Contrast this with non-fingerprinted deployments, where main.js is overwritten in-place. There, rolling back requires re-uploading the old binary and then invalidating /* to evict all stale copies — a much noisier operation that triggers a cache-miss storm.
Fingerprinting also protects against the most common rollback hazard: a mismatch between HTML and assets. When index.html references main.9e2f1a3b.js, that exact file is served from S3 (or from an edge POP that still has it cached). There is no window where users get new HTML with old files or old HTML with new files.
Review cache key architecture for the underlying reason: each unique filename is a distinct cache entry. The old hash URL and the new hash URL coexist in the edge cache simultaneously. Only index.html — which is not fingerprinted — needs any attention at all.
Choosing a Rollback Strategy
| Dimension | Nuclear cache wipe (/*) |
Surgical HTML rollback (/index.html only) |
|---|---|---|
| What gets invalidated | Every object in the distribution | HTML entry points only |
| Cache-miss storm risk | High — all POPs must refetch everything | Minimal — only HTML POPs refetch |
| Invalidation path cost | 1 path (/* counts as one) |
1–3 paths |
| Time to user recovery | 5–30 s invalidation + cache warm-up | 5–30 s invalidation; assets already warm |
| Works with fingerprinted assets | Yes, but wastes edge cache | Yes, optimal |
| Works with unhashed assets | Required if assets were overwritten | No — same URL, old content still cached |
| Risk of partial-state exposure | Moderate during warm-up | Very low — old hashed assets serve immediately |
Use the surgical approach whenever your assets are fingerprinted. Use the nuclear approach only when assets are not fingerprinted (same URL, new content) or when a security-sensitive payload must be purged immediately from all edge caches.
Triage Before You Roll Back
Not every post-deploy incident is a rollback. Spend the first sixty seconds establishing which layer actually broke, because the wrong remedy costs more time than the diagnosis.
If the browser console shows a JavaScript exception thrown from your own bundle, the code is bad and the rollback below applies. If it shows a 403 or 404 on a hashed bundle URL, the code is probably fine and the upload ordering broke: the HTML reached S3 before the assets it names, and the edge cached the error. That is fixed by finishing the upload and invalidating the failing asset path, not by reverting the release. If it shows a mixed state — new HTML, old chunk names that no longer exist — a service worker is serving a stale document from its own cache and neither S3 nor CloudFront is involved at all.
Check the response headers before deciding. x-cache: Error from cloudfront points at a cached origin failure. An age value larger than your HTML max-age points at a MinTTL above zero holding the document past its intended lifetime. A correct-looking etag with broken content points at the deploy itself. Only the last of those calls for the procedure below.
Step-by-Step Rollback Procedure
The complete rollback script below handles all five steps: identifying the prior S3 version, restoring it, re-uploading with correct cache headers, invalidating index.html, and polling for completion.
#!/usr/bin/env bash
set -euo pipefail
# -----------------------------------------------------------------------
# CloudFront rollback: restore prior index.html and invalidate the path.
# Prerequisites: AWS CLI v2, correct IAM permissions, DIST_ID and BUCKET set.
# -----------------------------------------------------------------------
DIST_ID="${DIST_ID:?Set DIST_ID to your CloudFront distribution ID}"
BUCKET="${BUCKET:?Set BUCKET to your S3 bucket name}"
HTML_KEY="index.html"
PRIOR_HTML="/tmp/prior-index.html"
echo "=== Step 1: List recent S3 versions of ${HTML_KEY} ==="
aws s3api list-object-versions \
--bucket "$BUCKET" \
--prefix "$HTML_KEY" \
--query "Versions[*].{VersionId:VersionId,LastModified:LastModified,IsLatest:IsLatest}" \
--output table
# Set PRIOR_VERSION_ID to the version you want to restore.
# Example: export PRIOR_VERSION_ID="abc123XYZ..."
PRIOR_VERSION_ID="${PRIOR_VERSION_ID:?Export PRIOR_VERSION_ID after reviewing the table above}"
echo "=== Step 2: Download prior version to local disk ==="
aws s3api get-object \
--bucket "$BUCKET" \
--key "$HTML_KEY" \
--version-id "$PRIOR_VERSION_ID" \
"$PRIOR_HTML"
echo "=== Step 3: Re-upload with no-cache headers ==="
aws s3 cp "$PRIOR_HTML" "s3://${BUCKET}/${HTML_KEY}" \
--cache-control "no-cache, must-revalidate" \
--content-type "text/html" \
--metadata-directive REPLACE
echo "=== Step 4: Create invalidation for /index.html only ==="
INVALIDATION_ID=$(aws cloudfront create-invalidation \
--distribution-id "$DIST_ID" \
--paths "/index.html" \
--query "Invalidation.Id" \
--output text)
echo "Invalidation ID: $INVALIDATION_ID"
echo "=== Step 5: Wait for invalidation to complete ==="
while true; do
STATUS=$(aws cloudfront get-invalidation \
--distribution-id "$DIST_ID" \
--id "$INVALIDATION_ID" \
--query "Invalidation.Status" \
--output text)
echo "Status: $STATUS"
if [ "$STATUS" = "Completed" ]; then
echo "Rollback complete. Invalidation propagated to all edge POPs."
break
fi
sleep 5
done
echo "=== Verify ==="
echo "Run: curl -sI https://your-distribution.cloudfront.net/index.html | grep -i 'x-cache'"
echo "Expected: x-cache: Miss from cloudfront (first hit), then Hit from cloudfront"
Using S3 Versioning
S3 object versioning must be enabled on the bucket before you deploy. If you have not enabled it, you cannot use list-object-versions to restore a prior HTML file. Enable it with:
aws s3api put-bucket-versioning \
--bucket "$BUCKET" \
--versioning-configuration Status=Enabled
Once enabled, every s3 cp or s3 sync creates a new version. To restore without downloading locally:
# Restore a specific version in-place (no local file needed)
aws s3api copy-object \
--copy-source "${BUCKET}/${HTML_KEY}?versionId=${PRIOR_VERSION_ID}" \
--bucket "$BUCKET" \
--key "$HTML_KEY" \
--cache-control "no-cache, must-revalidate" \
--content-type "text/html" \
--metadata-directive REPLACE
If S3 versioning is not enabled, roll back from your Git repository instead — check out the prior index.html from the previous release tag and re-upload it.
The mental model that makes this fast under pressure: with fingerprinted assets, a release is not a directory of files that gets replaced. It is a pointer. Every bundle any release ever produced is still sitting in the bucket under its own content-addressed name, costing a few cents of storage and harming nothing. index.html is the only mutable object, and it is nothing more than the name of the bundle set that is currently live. Rolling back is re-pointing it.
This also explains why you should never add a lifecycle rule that expires noncurrent versions of index.html after a day or two, and why deleting “old” hashed bundles from the bucket to save storage is a false economy. A viewer who loaded the previous release five minutes ago still holds HTML naming those bundles; if a chunk is lazily imported on a route they have not visited yet, that fetch happens after your cleanup ran. Keep at least the last several releases’ assets, and expire noncurrent HTML versions on a horizon measured in weeks, not hours.
Verification
After the invalidation completes, confirm that users receive the prior HTML:
# Check x-cache header — should be Miss on first request after invalidation
curl -sI "https://your-distribution.cloudfront.net/index.html" \
| grep -iE "x-cache|age|cache-control|etag"
# Expected on first request:
# x-cache: Miss from cloudfront
# age: 0
# cache-control: no-cache, must-revalidate
# Confirm the HTML references the old asset hash (not the broken release hash)
curl -s "https://your-distribution.cloudfront.net/index.html" \
| grep -o 'main\.[a-f0-9]\{8\}\.js'
# Expected: main.9e2f1a3b.js (the prior-release hash)
Verify the old hashed asset itself is still reachable (it was never invalidated):
curl -sI "https://your-distribution.cloudfront.net/assets/main.9e2f1a3b.js" \
| grep -iE "x-cache|age"
# Expected: x-cache: Hit from cloudfront (still warm in edge cache)
# age: some positive number — it never left the cache
If the old asset is not in the edge cache (age: 0, x-cache: Miss), CloudFront will fetch it from S3. As long as the file still exists in S3, the rollback is safe — the edge simply refills its cache from the origin on the next request.
The Recovery-Time Budget
The reason to care about which rollback path you take is wall-clock time with a broken site in production. The two approaches differ by roughly a factor of four, and almost all of the difference is work that fingerprinting makes unnecessary.
The numbers are representative rather than guaranteed, but the shape holds. The S3 write is a few kilobytes either way when only HTML changes, and invalidation propagation is the same 30–60 seconds for one path as for a wildcard. What the nuclear path adds is a full asset re-upload — proportional to bundle size and count — followed by a period where every POP is cold and viewers pay origin latency on their first request for each object. On a distribution normally running above 95% hit ratio, that refill window is visible in your latency percentiles long after the invalidation reports Completed.
Two operational points follow. Rehearse the surgical procedure before you need it: the slowest part of a real incident is usually a human deciding which VersionId to restore, not any API call. And keep the rollback script in the repository next to the deploy script, parameterised by DIST_ID and BUCKET, so the incident does not begin with someone reconstructing CLI flags from memory. The broader pipeline-level version of this is covered in the CI/CD rollback reference.
When to Reconsider the Surgical Approach
Non-fingerprinted assets were overwritten in S3. If you deployed new content to main.js (no hash in the filename), the old file is gone from S3. A surgical rollback cannot work because the URL you are rolling back to points to the overwritten file. You must re-upload the old binary to main.js and then invalidate /* to evict stale copies. This is a strong argument for switching to content-addressed filenames — the content-hashing vs semantic-versioning reference explains the trade-offs.
A security-sensitive payload was exposed. If a release accidentally shipped credentials, tokens, or private data in a JavaScript bundle, a surgical rollback that only changes index.html leaves the compromised bundle cached at edge POPs until its TTL expires. In this case, you must also invalidate the specific hashed asset path to evict it from the edge, even though hashed assets normally never need invalidation.
S3 versioning was not enabled. If you do not have a prior version of index.html in S3, you need to reconstruct it from Git or an artifact store. The CI/CD rollback guide covers how to re-run the build from a prior commit and redeploy from the artifact.
Your HTML files are themselves fingerprinted. Some frameworks (Next.js App Router, for example) fingerprint even the HTML file names. In that case, the deployment places the new HTML at a new URL and the root route (/) redirects to it. The rollback procedure is the same conceptually — restore the redirect to point at the prior HTML URL — but the mechanism differs by framework.
The rollback is for a Cloudflare-hosted site. The procedure described here is specific to S3 + CloudFront. For Cloudflare, see rolling back a bad asset deploy on Cloudflare, which uses cache-tag purge rather than path invalidation.
FAQ
Do I need to invalidate the broken hashed bundle as well?
Normally no. Once index.html no longer names main.bad0cafe.js, nothing requests it, and a cached object nobody asks for is harmless — it ages out of the edge LRU on its own. Invalidate it only when the bundle’s content is the problem rather than its behaviour: leaked credentials, an exposed internal endpoint, or a supply-chain compromise. In those cases add the exact hashed path to the same create-invalidation call so it costs one extra path and no extra propagation time.
Why re-upload index.html with no-cache instead of just invalidating it?
Because the invalidation only reaches CloudFront. If the bad index.html went out with a max-age of 3600, browsers that already fetched it will keep using their local copy for the rest of the hour no matter what you purge at the edge. Setting Cache-Control: no-cache, must-revalidate on the restored document guarantees the next deploy’s viewers revalidate, and it is the header entry documents should have been carrying all along. The header split for each asset type is covered under Cache-Control immutable and TTL tuning.
How long should I wait before rolling forward instead?
Roll back first, diagnose second — the rollback is 45 seconds and reversible, while a forward fix requires a full build, deploy, and a fresh chance to be wrong. The exception is a bad release whose damage is already written somewhere durable, such as a migration or a bad client-side write that reached your API. Reverting the bundle stops new damage but does not undo what shipped, so pair the rollback with whatever data remediation the incident needs.
Does a rollback change the hashes in the next forward deploy?
No. Content hashes are derived from bundle content, so reverting the source and rebuilding reproduces exactly the hashes the good release had — assuming the build is deterministic. If a rebuild of an unchanged commit produces different hashes, the build is picking up a timestamp, an absolute path, or an unpinned dependency, and that is a separate problem worth fixing before the next incident. The deterministic build outputs guide covers the usual culprits.
Can I automate this entirely?
You can automate everything except choosing the version, and you should. A rollback job that takes a git tag, fetches that tag’s index.html from your artifact store, uploads it with the right headers, invalidates one path and polls for completion is around forty lines of shell. Keeping a human on the version choice is deliberate: the failure mode of a fully automatic rollback is rolling back to a release that was also broken.
Related
- AWS CloudFront invalidation guide — full invalidation reference: CLI, API, cache behaviors, TTL, cost model
- CDN purge strategies — overview of purge mechanisms across CloudFront, Cloudflare, Fastly, and Nginx
- Rolling back fingerprinted assets in CI/CD — pipeline-level rollback: re-running from a prior artifact without touching the CDN directly
- Rolling back a content-hashed release — the general rollback pattern independent of CDN provider
- Rolling back a bad asset deploy on Cloudflare — equivalent procedure for Cloudflare-hosted sites