OA
All projects Cloud & Ops · Kubernetes

Rush Hour

The shop survives the Black Friday night and I can prove it.

How the shop handles the rush

The proof

0oversold: 500 buyers, 10 sneakers, exactly 10 orders
2 → 6Catalog servers, 0.00 % errors, even when one fails
43suntil a payment problem is detected automatically
3checksper commit: tested, scanned, signed

The problem

Everyone arrives at once. Plus a risky update and a slow payment service.

  1. Banks on payday
  2. Ticket sales at a concert drop
  3. Logistics before Christmas

Try it yourself

You are the sales night. Send the rush, push a faulty update, take a server down. The shop holds.

Try it: follow the golden hints.

Time23:58:00Time lapse, 10 ×
● Orders per minute● Error rate
Shop versionv1.4 stable
Payment serviceworking
Running servers
Sneakers in stock10 of 10 pairs
0 sold · 0 oversold
120Orders per minute
0.1 %Error rate now
100.00 %Successful orders since start, target 99.5 %
0paid after the wait

0 orders this night

Rebuilt in the browser with the values from the real test run: update taken back after 45 s, payment disrupted for 65 s, peak of 696 orders per minute. The real code with Kubernetes, Argo Rollouts and Grafana is on GitHub.

When something goes wrong

  1. Emergency brake for payments: payment hangs, the order waits instead of failing
  2. Refund: orders paid too late get their money back
  3. Clean up: remove stuck orders, release the stock again

Game Day

6 minutes of sales night, simulated. With a deliberately broken update and a payment outage in the middle.

45suntil the faulty update was taken back automatically
93.1%of checkouts went through even during that time
696orders per minute at peak
  1. Midnight: rush on the shop
  2. Faulty update is pushed out
  3. The system takes it back automatically
  4. Payment slow, orders wait for 65 seconds
  5. All waiting orders paid afterwards

The checkout missed its availability target (98.1 % instead of 99.5 %). That is exactly what the error budget is for. The catalog stayed at 100 %.

omar.dev@web.de