Compromised? Where are the logs?
Posted by Tate Hansen Fri, 10 Nov 2006 05:55:00 GMT
Full compromises are crazy – it happens, but nearly every one I touch still sends a little shiver down my spine mixed with a “holy shit” moment.
Case in point: we got a call last week to help out with an incident and to do an investigation of some Linux based security devices. Financial industry, lots of customers.
Their engineers were working feverishly to scope the incident and to learn all the details then slam – they hit a wall. Their firewall logs everything. Their only-way-out-is-via-a-proxy-server logs everything.
And their security point solution in the DMZ? Oops, it only logs locally (it also happens to be a party to the fully compromised club inside this trusted net). And what about their desktops which appear to have offered free remote admin VTYs? Well, this all has a kicker for an answer.
It happened a year ago. Ouch. Talk about sensing a cold sweat and suddenly not trusting anything. By the way, the desktops exhibiting the worst behavior traveled the evolutionary path and have been wiped clean and rebuilt, upgraded, or replaced since the break-in. No substantive logging or backups of mischievous desktops; no way to reconstruct the perpetrator’s methods. Security point solution in DMZ had standard local logging (i.e. full rotation in weeks), therefore it is unlikely things we're looking for are there (still investigation). No network device logging. There are lots of unanswered questions.
The details of this should strengthen your neuron paths connecting logging to your “holy shit” moments. That is to say…
- If you got important data to protect, log everything you reasonably can. All the “security” in this scenario failed and failed to help reconstruct events.
- Do you have one of those semi hands-off security appliances that you presume is working fine because you can connect to the web admin portal? Make it forward logs to somewhere.
- Do you have workstations which touch sensitive data anytime? Yes. Then boost the priority to configure central logging, stop procrastinating, then take comfort you’ll be in better shape than the poor souls at this company fighting to salvage their pride, and maybe their jobs.
Thanks for this post! Despite all the NDAs and companies not wanting to become case studies of insecurity, more sterilized reports from people on the front lines are needed so that we can reinforce reality in more ways than just the big incidents reported in the media. I hope you are able to share more details as the investigation unfolds.
A financial institution compromised a year ago… Can you imagine how much fun you can have when you have a whole year with which to play with your pwned little company? For something that long, pretty much nothing is sacred.
Awesome! I am about to write my own story on that same thing …
And I love your conclusions too