How info-beamer hosted handled two big internet incidents

Recently two major incidents made it to the news. This blog post describes both those incidents as well as how info-beamer was built to minimize the impact of those incidents.

The first one was CloudBleed. The second one was the S3 outage.

CloudBleed

What happened?

Let's start with this one. If you haven't read about it yet, here is a small summary: CloudFlare is bascially a company offering a smart reverse proxy: Customers of CloudFlare configure their web setup so HTTP(S) requests first end up at one of many CloudFlare servers which then forward (aka proxy) the request to the original server. That way the CloudFlare servers can peek into the request and (for example) block requests that look like they are part of a DDOS attack. Similarly those servers can modify the response sent to the users: They might, for example, optimize images or compress HTML pages.

This means that CloudFlares servers can look into the complete communication between the users and the web server. As a customer of CloudFlare you entrust them with keeping that communication secure. This trust was broken for several days in February. Due to a bug in CloudFlares web server, a number of web pages sent out to users (and search engines) included information from pages previously sent to unrelated users. This leaked information included API keys, cookies and other private information. Any customer of CloudFlares servers was potentially affected by this. According to CloudFlare, 1 in every 3,300,000 request was leaking data from third parties.

Impact on info-beamer hosted

info-beamer hosted uses CloudFlare for some of its services. The most visible one is the delivery and caching of images and other static web content. The image saying 'blog' on this page was sent to you through CloudFlare. Similarly all those small thumbnails on your asset page are handled by CloudFlare. info-beamer hosted uses CloudFlare for faster delivery of those images as CloudFlare has their servers all over the world.

The impact on info-beamer hosted customers is minimal: info-beamer hosted uses two different domains to provide its services: info-beamer.com and infobeamer.com (notice the missing dash). The first one if the secure domain which is directly handled by the info-beamer hosted servers. As it uses HTTPS, no third party can accidentally leak information sent or received to this domain. Session cookies and login passwords are directly sent to the info-beamer hosted servers.

The second one (infobeamer.com) is the utility domains which (among other things) handles the delivery of images and thumbnails. As all that content is on a different domain, none of the important information (like passwords) ever end up in requests to that domain. Your browser treats both domains as two completely different entities. So it was impossible for CloudFlare to leak important information of info-beamer hosted customers as that information was never sent through their servers.

The only information that might have leaked are thumbnails of uploaded assets. Unless their content is revealing, they can't be correlated to an info-beamer account. And a single thumbnail url can't be used to learn about other asset urls for the same account.

The info-beamer website also uses CloudFlare to deliver JS and CSS to your browser. Following the principle of least privilege it makes sense to make sure that content delivered through the CDN can be trusted regardless of what the CDN is doing. info-beamer pages use subresource integrity to prevent this using strong hashes. If a modified JS ends up being sent to a browser, the browser will reject it. You can see this in action if you view the source of an info-beamer page and look for 'integrity'. If another CloudFlare bug for some reason sends out corrupted JS files, info-beamer won't be affected.

S3 Outage

What happened?

S3 is a data storage service provided by Amazon. info-beamer hosted uses S3 to store assets and package data. Devices always download their content from the S3 servers, never from info-beamer.com directly. Usually S3 is pretty reliable. Except when it isn't: On late February, S3 went down for a couple of hours. Fortunately only in a single region (US-EAST-1). During this time, no downloads or uploads where possible. Luckily info-beamer hosted uses S3 in another region (EU-WEST-1), so it wasn't affected.

It makes sense to store data on S3 as they offer unlimited available space, fair prices and an easy to use API. Running a custom file storage service is out of scope for info-beamer hosted and would be a waste of time and resources. Outsourcing this aspect to another provider makes sense for both economical and reliability reasons. Of course including an external dependency must always be a conscious decision and outages must be considered when designing a system.

Impact on info-beamer hosted

If S3 goes down in the region that info-beamer hosted uses, no more package updates or asset uploads are possible. The reason is that file content for those is stored on S3. Additionally devices that run an old version of the info-beamer hosted operating system can't update to the current version. The latter problem is almost never critical: The devices will just update later.

Whether or not package update or asset uploads are critical depends on how you use the info-beamer hosted service: If your devices play a mostly static setup or a setup that only interacts with either local or third-party services (like the twitter api) for dynamic content are not affected at all: They will run as usual and you can reboot them without problems as all content required to show something on screens is already cached on the device from a previous sync.

If the content of your screen is based on a dynamic package, things might be different. During the outage the package sync call will fail as the S3 storage package isn't available. So syncing a package won't work and no new content is delivered to your devices.

None or the packages offered through the package gallery would have been affected as they all are "install-once-run-forever" type packages that have to be downloaded only once to a device.

Usually an outage is fixed within a few hours. In case of a longer outage, it might be possible for info-beamer hosted to switch S3 regions on the fly with the help of cloudfront and cross-region replication. Right now this would be a manual process. But it would be possible to do so without a lot of work.

Summary

While info-beamer hosted isn't immune to problems, careful design allows some resistence against them. Every aspect of the info-beamer hosted service has been carefully build in a way to make sure you get the most reliable service available for the Pi - even if things break from time to time.