Held together by assumptions: a cascading failure at login

Most systems are vulnerable to cascading failures. They're held together by a glue called operating conditions, or assumptions.

One of my gripes with the cloud is that we often solve logic or programming issues by adding more infrastructure. It works in 99% of cases, for the time being, but it skips the step of understanding the underlying issue.

The incident

I was about to board a plane when the Slack messages started coming in. Users couldn't get past login, and the failure count kept climbing as the US woke up and started its day. One of the databases was getting bogged down with traffic, and the first instinct in the incident channel, a reasonable one, was to increase the database cluster size.

The flow touched about five services, and I happened to be working on a data migration that involved it, so I joined in. I didn't have my laptop, but I had a team supporting the effort, and I asked them for the changes that had recently reached production.

The change

An engineer had noticed high load on the read replicas while the write primary of our SQL database sat nearly idle. To balance it, the change spread read queries across both the primary and the replicas. On its face, that's a sensible optimization.

The cause became clear quickly. Rolling the change back, instead of scaling up the cluster, brought the system back to normal in about 30 minutes.

The flow

• Check for a user attribute in the cache first, then in the database.
• If it's present, cache it. Otherwise, write it to the database.
• Repeat in cycles until the attribute is there. All of these requests are async.

It seems normal, but the devil is in the details and the scale. The whole thing works only if replication lag between the databases is shorter than the time it takes for the user's next read after a write.

Before the change, the primary handled only writes, so replication lag stayed small and lock contention wasn't bad. Once read traffic for the same users also went to the primary, those reads and writes competed for the same locks. The primary slowed, replication lag grew, reads on the replicas missed values that had just been written, and every miss triggered another write. Each retry made the problem worse.

Adding database capacity would have fed the loop, not broken it.

Why this matters

The change itself was reasonable. It broke an assumption that nobody had written down, which is exactly how these failures happen. Catching them takes a mix of imagination and experience, and it's a real gap in many systems today: few people get to understand both the product and the system end to end.

So why bring this up? Because these are the guiding principles and experiences I'm applying while building our identity provider 😀