A review by a security engineer would have prevented a false sense of security
Everyone involved in this, from the people who wrote the heartbeat code, the people who committed the code, the people at Akami who wrote this patch, this poster ... all would describe themselves as security engineers. Is everyone but Willem Pinckars incompetent?
I think the one lesson to learn is to treat crypto just like you do cloud providers. Make sure you have an exit strategy from day one; the ability to push a button to roll keys, switch algorithms, switch cert providers, etc.
There's been a lot of overly-confident statements about Heartbleed not being a big deal from apparently competent people these last few days. I'm not sure what it is about it that causes security professionals to spout off without testing their assumptions, but...
I don't think I've worked anywhere where there was a simple way to switch cloud providers with the push of a button. Even migrating to different locations within a cloud provider isn't trivial. You'd have to invest a lot of engineering time from the beginning to get this kind of capability, and you may never realize any value for all the pain.
Yep; not easy at all ... if it were easy everyone would do it :) I think with tools like puppet and orchestration tools like heat, etc it is getting closer to being easier.
I did got to a Netflix talk once where they said that while moving from AWS would huge, they had a plan and I think from memory they said it would take about a week.
But the whole episode, and this post in particular, highlights the issue that enough review is never enough. When is the the code secure? When it is written by someone with a good track record? When it gets reviewed and committed by knowledgeable parties? When it's been running for several years without incident? When it passes coverity? When Willem has time to review it?
When I just think about the embedded ssh keys I've got in my few toy systems; if I discovered ssh was broken (and it's happened; look at the Debian bug) I'd be sinking a lot of time figuring out just how to change the keys everywhere. I should have designed for this from the start; lesson learnt. I bet there's lots of admins out there who wish they had better ways to update certs on a moments notice over the past week...
I think it's reasonable to disagree with the part he quoted. "A review by a security engineer ..." Well, it was reviewed by security engineers, and the false sense of security remained. The rest seems fine though.
My experience in large companies is more like this: A (security) engineer reviews, objects and management says: "Go ahead anyway you don't have the whole picture".
It's not exactly a proof of concept if it doesn't protect the keys, is it?
It strongly hints that even if this is 'only a POC' that their actual implementation is still vulnerable since their POC failed to protect against the very attack is was written to protect against.
We're still evaluating some of his arguments. I still believe some of them are true in the general case, but do not apply to our specific embedding of this code. I say that aware that I was mistaken 12 hours ago, and so very well could be mistaken now.
But I am reasonably convinced that the CRT values are loaded into the normal heap, where they're available to a normal Heartbleed attack. Pinckaers doesn't have to be right about all his points to be right---just once---and I'm pretty sure he's right at least that once.
If the tone of my message was a bit harsh, it was mostly to reflect his, and thankfully I do not have a life boring enough that this is "my moment".
Still, I stand by my point of view that at the time of writing, akamai's POC was presented neither as an absolute final fix nor as their own production version of it, and judging it as such was misplaced.
That akamai realized the flaws he noted in their patch also applies to the real world code doesn't change that.
I never said the flaws he pointed were not real flaws nor unimportant ones, I merely disliked the ridiculous and unjustified tone he used to to destroy a proposal and used the same against him.
Everyone involved in this, from the people who wrote the heartbeat code, the people who committed the code, the people at Akami who wrote this patch, this poster ... all would describe themselves as security engineers. Is everyone but Willem Pinckars incompetent?
I think the one lesson to learn is to treat crypto just like you do cloud providers. Make sure you have an exit strategy from day one; the ability to push a button to roll keys, switch algorithms, switch cert providers, etc.