building a better security events system

Posted by Tate Hansen Sat, 30 Jun 2007 21:56:00 GMT

It’s hard to build a decision support system based on partial views of the world.

My goal is to identify interesting events on a network and to prioritize those events based on sets of attributes. Yes, there are lots of products that do this. But most focus on a slice of the world (e.g. an IDS fires an alert based on a regex match on a single packet). And that is boring.

Doing it for the whole world is where the action is at.

Capture an alert fired from an IDS, check netflow for a session, note a “first-time” event recorded in a syslog message, mix in statistical data mining and learning techniques – and do it all in near real time. This is how things get interesting.

Unfortunately it’s hard to get complete visibility (i.e. get all syslog, all netflow, all application logs, etc.). There must be a point though where I can get enough information to successfully prioritize interesting events. I’m not sure exactly where that’s at, but it’s a fun problem to work on.

The picture is of the inside of IT-Universitetet in Copenhagen where I’m working for a few weeks. The meeting rooms all jet out into the open space in the middle – a pretty cool design.

Tags , , , , , , ,  | 5 comments

Comments

  1. Roland Dobbins said about 12 hours later:

    Narus, Q1, Lancope, and Mazu already do this. There’s nothing new about it. All can be improved, but this functionality has been available in COTS products for quite some time.

    And you’re right, ‘complete’ visibility isn’t the answer - it’s a problem. Contextually-complete visibility on a per-incident basis, along with trending, is the answer. These products provide it.

  2. Richard Bejtlich said about 15 hours later:

    Why not work on adding this to Sguil?

  3. Tate Hansen said 2 days later:

    Yes, I didn’t mean to imply any of this is new. But there are lots of ways to win in business.

  4. Andrew Hay said 3 days later:

    Hey Tate,

    I’ve reread your post a few times and I’m still confused as to what you’re looking for :)

    Any chance you could expand on your “wish list” to help clear it up for me? I’m just having a really hard time trying to figure out what it is you’re looking for. It almost sounds to me like you’re looking for a centralized repository for all network flows and device events discovered by all users touching the internet? Am I wrong (or maybe close)?

    Sounds like Skynet to me :)

  5. Tate Hansen said 4 days later:

    Hi Andrew,
    The products developed by the companies listed in Roland’s comments above run, more or less, in the same direction. Perusing their sites provides a good context.

    Skipping details, the game is about how to find interesting (or important, or security) events by observing everything realistically possible and presenting the results in way that allows someone to understand and deal with complex events (http://www.complexevents.com is a great site to understand complex event processing).

    For example, I’m not looking to understand every raw syslog message from every syslog-enabled device in the world nor am I really that interested in how to store it all. There are emerging tools (not new, but maybe better?) that allows for anyone to creatively build a software stack to do crazy things like analyze millions of events per second in real time and present “meta-events” far more valuable than observing a raw event. Cayuga is one. http://www.cs.cornell.edu/database/cayuga/

RSS feed for this post

Comments are disabled