Cloudflare parser bug

So once again we have another major security leak. You can read about it, here and below is the email CEO Matthew Prince wrote to customers:

Dear Cloudflare Customer:

Thursday afternoon, we published a blog post describing a memory leak caused by a serious bug that impacted Cloudflare’s systems. If you haven’t yet, I encourage you to read that post on the bug:

https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/

While we resolved the bug within hours of it being reported to us, there was an ongoing risk that some of our customers’ sensitive information could still be available through third party caches, such as the Google search cache.

Over the last week, we’ve worked with these caches to discover what customers may have had sensitive information exposed and ensure that the caches are purged. We waited to disclose the bug publicly until after these caches could be cleared in order to mitigate the ability of malicious individuals to exploit any exposed data.

In our review of these third party caches, we discovered data that had been exposed from approximately 150 of Cloudflare’s customers across our Free, Pro, Business, and Enterprise plans. We have reached out to these customers directly to provide them with a copy of the data that was exposed, help them understand its impact, and help them mitigate that impact.

Fortunately, your domain is not one of the domains where we have discovered exposed data in any third party caches. The bug has been patched so it is no longer leaking data. However, we continue to work with these caches to review their records and help them purge any exposed data we find. If we discover any data leaked about your domains during this search, we will reach out to you directly and provide you full details of what we have found.

To date, we have yet to find any instance of the bug being exploited, but we recommend if you are concerned that you invalidate and reissue any persistent secrets, such as long lived session identifiers, tokens or keys. Due to the nature of the bug, customer SSL keys were not exposed and do not need to be rotated.

Again, if we discover new information that impacts you, we will reach out to you directly. In the meantime, if you have any questions or concerns, please don’t hesitate to reach out.

Matthew Prince
Cloudflare, Inc.
Co-founder and CEO

So lets be clear

  • …the greatest period of impact was from February 13 and February 18 with around 1 in every 3,300,000 HTTP requests through Cloudflare potentially resulting in memory leakage (that’s about 0.00003% of requests).
  • Only customers who use Automatic HTTPS Rewrites, Server-Side Excludes and Email Obfuscation were affected.
  • …data that had been exposed from approximately 150 of Cloudflare’s customers across Free, Pro, Business, and Enterprise plans
  • CloudFlare is SaaS
  • Security hole was completely closed in 7hrs 11mins from being Tweeted about an issue
  • Security hole was mostly closed off in 1 hr 8mins
  • Production fix and service restored in 3 days 10 hrs 9mins
  • People are jumping on the problem making it sound worse then it was (don’t get me wrong it was bad, but no where as bad as Heartbleed, Heartbleed, still IS a problem)
  • CloudFlare have been very transparent

…And this is why I review code I take on-board, regardless if it works and advise others to review my code