Managing For Heartbleeds
So you had your corporate web servers updated with the patch, pow-wowed with your vendors and other partners to make sure everything was contained, and when you caught a breath you changed your online banking passwords (I mean, you did, didn’t you?); but now the adrenaline level has returned to normal and it’s time to think about avoiding this kind of fire drill in the future.
Before we do, let’s reflect that not everyone is as conscientious as you are, and until the OpenSSL software is patched everywhere you could be logging in to an insecure server. Remember that any password you used in the last two years could be compromised – and there are plenty of servers out there that are still vulnerable. Nor is it just web sites. Routers and all sorts of equipment you may not think of has being any kind of web server use OpenSSL, including appliances that are not even be connected to the net. If it’s disconnected it’s unlikely to have been fixed, although that reduces the risk too. The point is: there are going to be unpatched systems popping up for years. So we’re not just thinking about some abstract future vulnerability. The danger from this one hasn’t yet passed.
Let’s touch lightly on the near term tasks. I say “lightly” because this is just a re-cap of what you’ve already been thinking about.
The obvious next step is to look at how your policies and procedures held up. Were you able to react quickly? Did you know where your vulnerabilities lay or did you have to check each server? Did you have to respond with or through external partners and how was their speed, efficiency and professionalism in the crisis? Security is burdensome to most users, including admins, so it’s essential that your security policy be absorbed by the organization’s culture. If that’s not the case, you will likely have seen evidence of this in the reaction to Heartbleed. Changing the culture is tough at the best of times, and security procedures are never sexy.
Some have argued that such a crisis is best dealt with through centralized data center management running the best possible management tools (see for instance Paul Vanezia’s article on Infoworld). I don’t think anyone would argue with that, but it’s not entirely realistic in the age of cloud computing. Centralization is ideal; but good communications is always key. That means people answering the phone (or email, or chat box, or on the desk) who are capable of reacting or quickly finding the person who is; and it means ensuring information is propagated rapidly, so everyone knows what needs to be done, what has been done and who is supposed to do it.
But you took care of all that so now let’s look out a little further. Was it smart to be using OpenSSL in the first place? It’s arguable that Heartbleed wouldn’t have happened if it had had the same kind of scrutiny that a commercial product receives. Few will have had their minds changed about open source tools by all this. The arguments pro and con haven’t changed, and the next vulnerability is as likely to support one side as the other. One thing to consider is that following too closely to the leading edge has risks; it can pay to let others hit the hidden trip wires. Those who hadn’t bothered to upgrade their OpenSSL since 2012 are looking pretty clever right now. At the same time, one reason for upgrading is an expectation that a new release will plug more holes then it opens. Maybe you don’t want to be the first one over the parapet, but there are risks in hanging back too long.
Maybe the lesson of Heartbleed is that open source isn’t as free as we thought it was. If you do remain an open source booster, consider putting your money where your mouth is. Your company donates to political parties and candidates to further its interests; maybe a contribution to open source projects, standards bodies and other technical foundations are in its interest too. Perhaps you thought that was the government’s job, but we hear more about their cybersecurity exploits than their support for our digital defenses (and it has been credibly suggested the U.S. NSA took advantage of Heartbleed while saying not a word to the public). I don’t like to get too political here, but maybe if you do contribute to the OpenSSL foundation, explaining to your congresswoman that that’s where her donation went this year would leverage the investment. How ever we go about it, somebody needs to support the OpenSSL Project.
