A Cloudflare spokesperson told Ars that the cloud services provider saw “a spike in unusual traffic to one of Cloudflare’s services,” which “caused some traffic passing through Cloudflare’s network to experience errors.”
“We do not yet know the cause of the spike in unusual traffic,” the spokesperson said. “We are all hands on deck to make sure all traffic is served without errors. After that, we will turn our attention to investigating the cause of the unusual spike in traffic.”
That was based on anecdotal evidence, at the time. The real cause was Cloudflare shooting themselves in foot again because of a bad config file.
The bad config file is somewhere in the middle of the chain of causality.
They changed database permissions, revealing a dormant bug in a database query, leading to config files being generated badly with duplicate lines, making them too large for intake by the bot detection service, which didn’t have good input validation and made the process panic instead, ruining the service.