SEV-2ResolvedINC-2291Blameless

Postmortem: checkout outage, September 19

For 38 minutes (14:02–14:40 UTC) about 30% of checkout attempts failed with a price-mismatch error. A release rollback skipped our cache purge, so checkout validated carts against stale prices for 212 products. A manual purge fixed it; rollbacks now purge automatically.

Failed checkouts1,184of 3,950 attempts
Revenue at risk$63.9k1,184 × $54 avg order
Time to detect4 min14:02 → 14:06
Recovered71%re-ordered within 24 h
$

timeline --utc

TriggerUndetectedRespondingRecovering
  1. Release 2026.09.19-3 rolled backdeploy

    A feature flag misbehaved in the product page; the release was rolled back from the deploy console.

  2. Rollback completes, cache purge does not run

    The purge hook only listens for deploy.succeeded. Rollbacks emit deploy.rolled_back.

  3. Checkout starts rejecting cartsimpact starts

    The price cache expires on its normal schedule for most products, but 212 changed products keep stale prices and fail validation.

  4. Alert fires, on-call paged

    Checkout error rate above 5% for 3 minutes. Priya is paged.

  5. SEV-2 declared

    Incident channel opened; status page updated at 14:15.

  6. Rollback identified as trigger

    Errors started ten minutes after the rollback; every failing cart contained a recently repriced product.

  7. Pricing cache purged manuallymitigated

    Using the existing runbook. Error rate drops within two minutes.

  8. Error rate back to baselineresolved

    Monitoring continued for an hour before closing the incident.

$

root-cause --chain

TriggerRelease rolled back after a feature-flag problem
Root causePurge hook ignores rollbacks. It only runs on deploy.succeeded
EffectPrice cache keeps 212 stale products
EffectCheckout validation rejects the cart on price mismatch
Impact1,184 failed checkouts over 38 minutes
Contributing: customers saw a generic "Something went wrong", so many retried the same cart instead of refreshing.
$

review

What went well

  • The alert fired four minutes after impact began.
  • The cache-purge runbook was current and worked first time.
  • Status page was updated within six minutes of the page.

What didn't

  • Rollbacks take a different code path than deploys, and nobody had tested it with the purge hook.
  • The checkout error message gave customers nothing to act on.
  • Our dashboards don't show deploys and cache purges next to error rates.
$

action-items

ActionOwnerDuePriorityStatus
Purge the pricing cache on every deploy state change, including rollbacksMMarcoSep 26P1Done
Run a synthetic checkout after every deploy and rollbackPPriyaOct 3P1In progress
Replace the generic error with "Prices changed, please review your cart"LLenaOct 10P2In progress
Annotate deploys and cache purges on the checkout dashboardSSamOct 17P3Not started

Example from the MintPage showcase · fictional company, sample incident · any agent can publish a page like this with one publish_page call