http://blog.clearnetsec.com/articles/tag/real-time en-us 40 <p> I just finished a workshop covering the use of <a href="http://www.jsda.unina2.it/WDSA/home.html">Data Stream Analysis</a>. Its necessity is driven by the need to analyze massive volumes of data (e.g. system and network events) in near real time – essential given how fast you will hit your head on the insertion rate ceiling using standard relational databases. </p> <p> Off the shelf DBs (PostgreSQL, MySQL, Oracle, etc.) are unable to simultaneously commit thousands of events per second while performing complex queries. To have a chance of analyzing events in reasonable amounts of time you must analyze the incoming streams of data before inserting the data into a database. </p> <p> I ran into this scenario last year building a central log server using off the shelf components. Even a few dozen servers can stream events fast enough where you realize pretty quickly all the typical open source based how-to’s on building a system that can store, correlate, and alert are inadequate. Data stream processing is required when things get big. </p> Sun, 18 Mar 2007 11:17:00 -0600 urn:uuid:d8348f22-f34e-4c0c-a7eb-6b42c22695c8 tate@ClearNetSec.com (Tate Hansen) http://blog.clearnetsec.com/articles/2007/03/18/real-time-event-analysis ClearNet ClearNet Security Tate Hansen Real-time syslog