AWS Was Down for 20 Minutes. One Customer Was Down for Nine Hours

On 24 July 2026, AWS lost the networking hardware that carried routes between us-west-2 and the Seattle Metro. Traffic that began and ended inside the region kept flowing normally. Anything crossing the region boundary timed out. Within minutes, outage trackers filled with Apple Pay, DoorDash, Reddit, Hulu and PlayStation Network.

Main impact ran from 10:55 to 11:15 UTC. Twenty minutes. Route reconvergence caused a second wobble between 11:47 and 11:59, and the final Direct Connect path came back at 12:12.

NinjaOne, an endpoint management vendor sitting downstream, opened an incident at 11:38 UTC and did not resolve it until 21:28. Nine hours and twenty-nine minutes after AWS had finished restoring routing.

That gap is not Amazon’s. It belongs to the vendor, and the equivalent gap in your architecture belongs to you.

The tail is always longer than the outage#

Nine cascade incidents were confirmed across seven downstream providers. The spread in how long each took to come back is the whole story:

Provider	Opened (UTC)	Resolved (UTC)	Time after AWS restored
Cube Cloud	11:03	11:12	Recovered before AWS did
Bambu Lab	11:12	13:18	1h 19m
FortiSASE	11:20	13:31	1h 32m
StatusHub	10:59	14:10	2h 11m
SparkPost	11:14	18:44	6h 45m
SendGrid	12:06	19:31	7h 32m
NinjaOne	11:38	21:28	9h 29m

Same upstream fault. Same twenty minutes of lost routing. Roughly a 65-fold spread in customer-visible duration. Nothing about the AWS incident explains that spread — every difference lives in the downstream architectures.

NinjaOne’s own explanation is the most instructive engineering artifact to come out of the day. Their agents tried to reach the backend during the outage, failed, and dropped into a backoff and retry path. When routing came back, over 150,000 devices reconnected within an hour. The backend did not fall over, but device status processing could not keep up, and full recovery required scaling that tier while the herd was already arriving.

Read that again: the failure was not the outage. The failure was the reconnection.

Queue drain is arithmetic, and the arithmetic is brutal#

Most engineers estimate recovery time as roughly equal to outage time. That intuition is wrong by an order of magnitude, and you can prove it on the back of an envelope.

Take a Data Platform ingesting 5,000 events per second with a pipeline provisioned to handle 6,000 per second — a comfortable 20% headroom that would pass any capacity review. Twenty minutes of blocked egress produces a backlog of six million events. Drain capacity is the spare 1,000 per second. Six million divided by 1,000 is 6,000 seconds.

Twenty minutes of outage, one hundred minutes of catch-up. A 5x multiplier, from a system that is not even under stress.

Now assume the more typical 10% headroom. Spare drain capacity drops to 500 per second, and drain time goes to 12,000 seconds — three hours and twenty minutes. The outage did not change. Your headroom did.

SparkPost’s disclosure matches the model almost exactly. They blocked outbound delivery from the US West region by 11:53 UTC and kept queueing internally, which was the correct call. Delivering the accumulated backlog after restoration still took another 80 to 90 minutes.

The lesson is not “add headroom” — headroom is expensive and mostly idle. The lesson is that drain rate is a design parameter you should know the value of, the same way you know your p99. If you cannot state your drain multiplier for a 20-minute stall, you do not know your recovery time objective. You have an aspiration.

Reconnect is a load test you did not schedule#

The reconnect path is where most of the nine hours went, and it is governed by four decisions that are usually made by default rather than on purpose.

Backoff without jitter is synchronisation. If every client backs off 1s, 2s, 4s, 8s from the moment the upstream failed, they all failed at roughly the same instant, so they all retry at the same instants. You have built a distributed metronome. Full jitter — sleeping a random duration between zero and the current backoff ceiling — spreads the same traffic across the window and is a two-line change.

Circuit breakers must decide fail-open or fail-closed per dependency, not globally. A breaker that fails open on your recommendation service is correct: degrade the ranking, serve the page. A breaker that fails open on an authorisation check is a security incident. Teams pick one behaviour, apply it everywhere through a shared library, and discover the mismatch during an outage. Write the choice down next to each dependency, with the degraded behaviour spelled out.

Half-open probing needs a rate limit. Breakers that admit every waiting caller the moment the timer expires reproduce the thundering herd inside your own process. Admit one probe, and close the circuit only after a run of successes.

Idempotency is what makes replay safe. A backlog drain is by definition a replay of work whose outcome you are unsure about. If the receiving side cannot deduplicate on a client-supplied key, you cannot safely retry. Idempotency keys are cheap to add before an incident and impossible to add during one.

Cube Cloud recovered before AWS did. SendGrid pulled the degraded region out of load balancing by 11:21 UTC — six minutes after AWS began restoring — and reported zero customer impact for the rest of the day. Those teams did not have better luck. They had shorter dependency chains and a failover path someone had actually run.

Which brings up the uncomfortable part: most teams test failure and never test recovery. Chaos engineering practice is dominated by kill-the-instance, sever-the-link, inject-the-latency. Almost nobody runs the other half of the experiment — restore the dependency and watch what the fleet does in the following ten minutes. That is where the thundering herd lives, and it is invisible until you look.

The other July incident: new features are not battle-tested#

Eight days earlier, on 16 July 2026, CloudFront went down globally from 07:45 to 11:18 UTC — three hours and thirty-three minutes. An internal connection-management limit stopped routing configuration from loading onto the network processors.

Only distributions using VPC Origins were affected. Every other origin type served traffic normally. Hugging Face, Ubiquiti, Frontegg, TigerData, Doxy, Coda, Instructure Canvas and Blackboard all took the hit.

VPC Origins is a good feature — it lets CloudFront reach private origins without exposing them publicly. It is also newer and less exercised than the origin types it replaces, and on 16 July it became a global single point of failure while its older sibling kept running.

My position: treat any new managed-service feature as un-battle-tested for its first year in production, and keep a fallback path you can switch to under load. Not a rollback plan on a wiki page — a configuration you can flip. Traffic volume is what finds the limits in a control plane, and the first year is when that volume arrives.

Hugging Face being unreachable for three and a half hours deserves its own note. Plenty of AI implementation work pulls model weights or tokenisers from a single host at build time, and that dependency rarely appears on any architecture diagram.

The classroom is a production system#

Canvas and Blackboard both went down. That means the outage landed in classrooms — lectures, submissions, assessment deadlines — during a window that is mid-morning in Europe and the school day across South Asia.

Education software has an availability profile people consistently underrate. A School ERP or a Hospital Management System has hard, externally imposed time windows: attendance is taken at a fixed hour, an exam starts when it starts, a medication round happens on schedule. You cannot shift that load to 3am. A three-hour outage during the wrong three hours is not a degraded day, it is a lost one, and the recovery tail matters more than in almost any consumer product because the backlog has a deadline attached to it.

If you build in that space, the design consequence is specific: offline-tolerant clients, local write buffering with idempotent sync, and a drain path that prioritises time-sensitive records over bulk ones.

The checklist#

• Write down your drain multiplier. For a 20-minute stall, how long until every queue is empty? If nobody knows, measure it in staging this quarter.
• Add full jitter to every retry path, including the ones inside SDKs you did not configure. Check the defaults; several are worse than you assume.
• Rate-limit circuit-breaker half-open probes to a fixed small number of in-flight attempts.
• Record fail-open or fail-closed per dependency, with the degraded behaviour written next to it.
• Make every retryable operation idempotent on a client-supplied key. This is the prerequisite for safe replay.
• Prioritise your drain. Deadline-bound records first, bulk analytics last. Two queues, not one.
• Run a recovery game day, not just a failure one. Sever a dependency, restore it, and instrument the following ten minutes.
• Inventory single-vendor features under twelve months old and confirm each has a switchable fallback.
• Cap fleet-wide reconnect concurrency at the server side too, so a client bug cannot become your incident.
• Alert on backlog depth and drain rate, not only on error rate. Error rate returns to normal the instant the upstream recovers; the backlog does not.

The recovery tail is where Operational Automation earns its keep — not in the failover, which is well-trodden, but in the ordered, rate-limited, observable return to normal that nobody wants to hand-run at midnight.

Twenty minutes of AWS is not your problem. Twenty minutes of AWS becoming nine hours of you is.

---

If you can’t state how long your system takes to drain after a 20-minute stall, that number is your real RTO — and you don’t know it yet. We help engineering teams measure recovery tails, design reconnect paths that survive their own fleet, and build platforms that come back as cleanly as they go down. Start a conversation.