Like most IT teams on April 7th, our day began with a frantic race to upgrade our servers against the now infamous Heartbleed bug. The problem, however, is that this vulnerability could occur in a way that left you with no idea of whether you'd been compromised. You had no server logs, no changed files on disk, and no idea what data was pilfered if anyone attacked you.
However, we were able to definitively say we had not been attacked during the two-year window of vulnerability. This audit had several implications for our IT team:
- We did not need to revoke our SSL certificates.
- We did not need to send emails to our users asking them to change their passwords.
- We did not need to wonder if our SSL-protected resources had been accessed by unauthorized parties.
- We could sleep soundly at night, confident in the safety of our users and our information.
We Eat Our Own Dog Food, and It Is Delicious!We have many ExtraHop virtual appliances monitoring our internal traffic, many of them for various engineering purposes. Together, these appliances monitored every single SSL transaction in our environment going back even before March 2012. Because the ExtraHop platform parses transactions by content type, I was able to verify that our SSL servers had not received any heartbeat messages between the time the bug was created and the time we patched our servers.
After the public announcement was made and software updates started being released, we upgraded our Apache web servers both externally and internally. Our primary ExtraHop appliance showed that our SSL servers started to receive heartbeat messages soon after the public announcement, but after we had patched our servers. Looking at the client devices that sent these messages, I recognized some of the endpoints as my tests, other security-conscious ExtraHop employees' machines, and later some third-party testing tools.
ExtraHop Is an IT Admin's Best FriendOur Heartbleed audit is hardly the first time we've saved hours of work by using the ExtraHop platform. When we rolled out a new version of our website last year, we used ExtraHop to help our web agency improve the load time of several content-heavy pages, optimizing away extraneous SQL queries they didn't even know about due to the frameworks and plugins they'd utilized.
Before upgrades, planned outages, or design changes, we consult an ExtraHop appliance. Documentation, design docs, and even log files are often out-of-date due to development and infrastructure changes, but ExtraHop always shows you a current picture of what is really going on. Wire data is the ultimate source of the truth.
Most people can only build rules and monitors for situations you've already imagined. But ExtraHop gives you the ability to answer questions you had not known to ask before. You can try it out for yourself by checking out our free, interactive online demo.
This is a companion discussion topic for the original entry at http://www.extrahop.com/post/blog/how-extrahops-it-team-performed-a-heartbleed-audit-going-back-years/