Cloud Concern: Security: Heroku: What Went Wrong?

David Chen, founder of Duostack, posted the details of a security leak on heroku.com. Heroku has since fixed the bug.

Heroku.com is a multi-tenant PaaS provider. That means that different applications owned by different people may run on the same Linux virtual machine, but in different processes. This not as secure as having each application run on its own Linux virtual machines, but they can be isolated from each other. (Unix/Linux has done this for ages). 

David Chen's security exploit was to access the slug file of somebody else's application server and use it to get access to source code, security credentials, etc for that application owner. Ouch!

So what happened?

1. Over time, heroku.com has provided more and more functionality via the "heroku console", which is a command line program used to manage a heroku instance. The heroku console allows the user  to execute Linux shell commands. That allowed David Chen to run shell commands to see the files used by others.
2. Also, at some point, the folks at heroku.com unintentionally relaxed user privilege restrictions such that if somebody knew about files written by other people, they could get code, credentials, etc. 

David Chen was able to combine these two factors to uncover the user credentials, source code, etc from some other applications. He did the honourable thing and notified Heroku.

So what went wrong? In my opinion, Heroku did the right thing by creating a multi-tenant platform tailored for Ruby on Rails web applications. They abstracted away the operating system, and even developed new terms such as "slugs" and "dynos" for common concepts such as "application image" and "application server". Where they went wrong  was to allow the abstraction to leak by letting users run Linux shell commands via the heroku command line program. That leaky abstraction provided the ability for a smart hacker to see what was under the covers, and how to break it. When that combined with the relaxation of user privileges, it became possible to deduce the security credentials and source code of other applications.

Has Heroku fixed the problem? They fixed the specific problem, and they are well on their way preventing future security problems and/or detecting them quickly. They fixed the problem, introduced regression tests to ensure that the problem stays fixed, introduced security checks into their development process, and are conducting more independent security audits. See here for details.

What's the lesson here? Ask your cloud provider what they are doing for security. Ask yourself if that is enough to meet your security needs.

Heroku definitely let their guard down. But their response to this incident was as good because they fixed the bug, put in preventative measures, and put in a process to audit and catch security defects. That is as good as it gets, and that is what will help them become a trustworthy cloud computing platform.