By: Walker Rowe, March 01, 2017 (09:04 AM)

CloudBleed Problem with CloudFlare Exposes Private User Data

Cloudflare suffered a memory leak problem that caused private data to be exposed. It was discovered in February by a Google security researcher. It was quickly labeled “CloudBleed,” because of its resemblance to the OpenSSL Heartbleed problem. Cloudflare has since fixed the code bug.

What Google found was that its search engine was caching web page responses that included memory from the Cloudflare reverse proxy servers that contained private user data like authentication tokens, cookies, and passwords. A reverse proxy server is a server that receives web traffic and then sends it out again. That is basically what Cloudflare does.

The exposed data also include private conversations from the OKCupid dating website and some chat services. Of course the revelation of any of that could be highly embarrassing to lovers and would-be lovers.

And the loss of encryption keys and authentication tokens could let a hacker spoof a user, meaning take over their user session and pretend to be them. And the loss of a password is a dire problem that needs no explanation. Even if they were not in clear text passwords are encoded and not encrypted. So it is possible to decode those.

Cloudflare says the web server SSL private keys were not exposed because CloudFlare terminates that traffic on Nginx servers that did not have their problem. However there were encryption keys there, according to Travis Ormandy, the Google researcher who discovered the problem. Those are generated from the private SSL keys.

Cloudflare is a Contribution Distribution Network (CDN) that provides hosting, load balancing, caching, and DNS services for many companies around the world. What they do is reduce latency by using their global network to locate web pages closer geographically to users. They are among the largest companies in this business with some 13 trillion web page views per month. Uber, for example, is a customer.

So Cloudflare customers around the world—at least those who understand such things and follow this type of news—were highly concerned. But this memory leak only occurs in 1 out of 300,000 page views, says Cloudflare.

The memory leak was the same problem that we see time and again when people make mistakes writing C code. This is a memory address problem that lets a program point to a location in memory that has not been initialized. In other words, it is a buffer overflow. Java, Python, and other programming languages do not have this problem because they are not allowed to address memory addresses directly. People use C because it is a low-level language that runs very fast without the need for an interpreter, such as Java and Python require.

CloudFlare says they fixed their code in 12 hours. But there was a period of time that no one knew about this problem until Google reported it. Cloudflare says they no of no user who has been hacked because of this. But that claim on its face is doubtful as Cloudflare cannot know, for example, whether someone used one of those stolen passwords.

The problem was in Cloudflare’s implementation of a tool called Ragel. Ragel is a parser such as one might use when they are writing a code compiler. CloudFlare uses it to parse web pages.

The problem occurs when there was a certain combination of unbalanced HTML tags in a web page. In other words something like <script>, <div>, <href>, <table>, or other without the corresponding ending </div>, </href>, </table>, </script>. Cloudflare rewrites HTML pages for its customer to do things this change http:// to https://, insert the Google Analytics tag, and hide email addresses.

When the problem was discovered CloudFlare turned off the affected services: email obfuscation, Server-side Excludes, and Automatic HTTPS Rewrites, until they could fix the problem.

CloudFlare said that their HTML parser caused a C code buffer overrun error because of the way they were using Ragel. Ragel compiles its instructions into C.

In particular this is the offending C code:

if ( ++p == pe )
goto _test_eof;

Which should have read:

if ( ++p >= pe )
goto _test_eof;

For those not familiar with C, p in this case is a memory address. ++p means increment that address by the size of whatever it is pointing too. == means is equal to and >= is greater than or equal to. The pointer ending up pointing to random memory.

In other words p is supposed to be pointing to some initialized value like “abc.” But errors in coding cause it to point elsewhere in memory where other stuff, like passwords, also processed by the server, are stored. This is what makes C dangerous: it can point to the memory or other programs and processes. Java and Python are walled off and cannot do that.

Buffer overflow problems in C code are what hackers look for, as those are exploitable. There are tools programmers can use to check their C code for such errors. The compiler will check for that at compile time, but it cannot check code at runtime. So programmers use a technique called fuzzing to try to crash programs or cause errors, but that is not 100% effective since it requires manually thinking up scenarios that could cause such errors.

So there is nothing for users to do except know that Cloudflare has fixed this problem.  If you happen to use a site that has been affected by CloudBleed, change your password.

For a list of sites potentially affected by the leak, visit https://github.com/pirate/sites-using-cloudflare

Walker Rowe

Walker Rowe is an American freelance tech writer and programmer living in Chile. He specializes in big data analytics, cybersecurity, and IoT and publishes the website SouthernPacificReview.com.

Notice: The views expressed here are those of the authors and do not necessarily represent or reflect the views of Cursive Security.

Be Informed. Stay One Step Ahead.

Sign up for our newsletter and stay up to date with the latest industry news, trends, and technologies