Engineering
Rate Limiting Done Right with Token Buckets
Sarah ParkNovember 28, 20246 min read
Most rate limiters punish legitimate burst traffic. We built a token-bucket system that's fair, predictable, and lets your application burst when it needs to without hitting artificial walls.
The problem with fixed windows
Fixed-window rate limiters create a thundering herd problem at window boundaries. If your limit resets at the top of every minute, clients will stampede to send requests the moment the window rolls over.
Token buckets to the rescue
Token buckets refill continuously rather than resetting at fixed intervals. This smooths out traffic patterns naturally and allows short bursts without penalizing the client.