ClearNet Security

Today I was talking with a colleague from a partner company about the PCI certification - I think he's up for recertification.

The interesting thing is he was talking to a Qualys representative recently whom, affably speaking, offered tips on how to tune the Qualys scans based on new modifications made at Mastercard's test lab. The representative also said he could review the report Qualys automatically builds. My colleague exclaimed to me "It sounded like they already have the answers".

Of course they do. Qualys pays PCI to verify their ability to discover what PCI wants them to discover. People pay and use Qualys so they can become PCI certified. Anybody willing to click "start scan" has the ability to be an Approved Scanning Vendor.

What's my problem with all this? For one, the certification process is rotten:

http://blog.clearnetsec.com/articles/2007/05/16/pci-not-our-problem.
http://blog.clearnetsec.com/articles/2007/05/04/pci-misleading-racket

On top of that, it's costly and does little to vet engineers claiming competency. Its design is to weed out small security firms, which is probably why it fires me up in the first place and turns me into a cynical punk all day.

I've had this idea for a book for most of this new century. I left IBM because I didn't feel as if my contributions mattered very much, I proceeded to go to less and less capable software shops where my contributions mattered more and more but not exactly how I wanted and regardless of what I controlled it seemed like more and more stupid things were done. There have been a few shining moments where the places turned out to be collectively more capable than I had thought and there have also been some disasters. After thinking about it, I came up with a plan for a book. My plan has been to record the foolish things I've witnessed over the years, and believe me, I've seen a lot of really stupid stuff, and then produce a tome on the subject and somehow contribute to the cannon of software engineering or computer science by suggesting ways to avoid the mistakes I've seen and even made.

I guess I believe, perhaps foolishly, that if we know our mistakes and why they were made then somehow we can do a better job in the future. The more I look at my log, the more it's looking like my book will just be a bunch of Dilbert-like case studies which isn't what I want. There are just so many really stupid ways to screw up software... While I reformulate my plan, I'm going to blog up some of the lessons I've learned, distilled down and without naming names. Email me if you have some stories, or post them here.

Here are some thoughts on staffing. One thing I've also noticed is that some of the most stupid mistakes are made with honest and sincere good intentions. This might just be a general rule for life, I've never heard someone talking about how they were intentionally doing something stupid, there is always a logic no matter how twisted and screwed up it might be. For example, it's never a good time to fire somebody, there is always a holiday or a release or something coming up, it's usually not a personal problem and you want their kids to have a good Christmas and all that cliche stuff. It just sucks. So I've witnessed the fear of firing turn in to a real joke, I saw a completely incompetent employee kept on the job until "the time is right" and work was fabricated for them. This isn't as easy as you might think, you can't just pull assignments out of a textbook when you do that, it has to apply to the job in some way. At the same time, you've decided to terminate this person and you probably don't want them further contaminating the code base. Once I saw a place hire a contractor to do a job and then put the person who was sitting on the plank on the exact same assignment in parallel with the expectation that they are throwing one copy of the work out. Then some half-brained decision was made to some how incorporate both of the projects because "dead man walking" was upset that we weren't going to use his code. His complaint was unexpected, it was as if he saw through the crafty ruse. So instead of just doing the dirty work, there was a debate on who had better code between an hourly contractor and a soon to be ex-employee who's team didn't want his code. As if Salomon himself decreed it, they cut the baby in half and used both code bases. Even better, the team that knew of the "weak link" had their morale drop because it started looking like they weren't going to replace the guy after all. And the project didn't work quite right either.

The lesson? I've come around and become much more cold blooded on this one. The best day to fire someone is today. If you have firing to do, then get it on. No good ever comes from putting it off. I'm not advocating just firing people because you're having a bad day or something but if you decide to part ways with an employee and you have made a good decision that is well reasoned and thought through, then once you've made that decision you should act. I don't think the people that work at that particular company are stupid, I know many of them are anything but that. Emotion got in the way though, it took a simple but somewhat dirty chore and turned it into an engineering problem.

We’ve hit a new high. We’ve soaked ourselves in a bandwidth bath on behalf of a client whom would like us to discover active services across a range of six public /16 blocks plus some scattered /17s, /24s, etc. The range is close to a total of 400,000 IPs.

We started out with five dual Xeon systems running 20 to 40 instances of nmap, each tuned, and each instance targeting 64 IPs. This client wants the job completed in weeks, so we decided it was a good time to get more experience with unicornscan.

By luck, we tapped into a Danish provider that is allowing us to push 55Mbits/s. I have no idea how much that amount of bandwidth would normally cost, especially if sustaining it 24x7 for a few weeks, but I’m guessing it is way over $10,000. Our client would allow us to go up to 100Mbits/s, alas, our luck doesn’t go that far.

Anyway, we now have faster dual-core systems each pushing ~25 Mbits/s via unicornscan like so:

sudo nohup /usr/local/bin/unicornscan -mT -p –r25000 -vv xxx.zz.0.0/16:a -w unicorn.output.for.xxx.zz..0.0.fullTCP > unicorn.output.fullTCP &

We have lots of results from nmap; so far unicornscan is matching the nmap results. Having the ability to specify packets per second with unicornscan is super nice.

We’ll create a follow up post on how all our scanning worked out on this gig when we’re finished (sometime in late November).

I can’t resist rehashing this topic.

Why not, after performing a penetration test, ask the customer what they observed? And when doing so, using commensurate effort to clearly surface the importance and point from asking that question.

So many are ordering up vulnerability and penetration tests like it’s a quick immunization shot against being hacked. They may get a wee bit nauseous after viewing their vulnerability report, but once they mitigate and recover they feel invulnerable, however temporary.

Penetration testing proves nothing of invulnerability, and it naturally follows, ad nauseam, to ask, again, what does it really prove then? Marcus Ranum summarizes Gary McGraw’s answer to that:

Gary McGraw likes to refer to pen testing as the "badness-o-meter" - it's a test that registers, at one end of the dial "your network sucks" and at the other end, "we don't know."

If the “We don’t know” club doesn’t already list as members all penetration testers, it surely should. No one has the knowledge or possession of everything needed to break all systems, so everyone is playing their own part in the “don’t know”.

Even if you are named Mr. Exploiter and pay cash to subscribe to every 0day exploit factory, the value of showing your customer they get owned when you call an 0day blitz is not only from proving you can sprint past their defense. Anyone with an exploit can do that.

The value rises from the conversation you have with the customer after the party. Why should the customer care if the latest 0day worked? What was the customer to do about it anyway?

Companies whom take their security seriously need to report back to you. Instead of hearing the customer ask you, “What did you find?”, you should ask the customer, “What did you observe?”.

Imagine how shocked you’d be to listen to a customer detail all of your efforts during a penetration test: They report to you which systems you attacked, when and how, and what information you had obtained.

That would be good security. Penetration testing should move away from “I got you with this 0day” to “You identified 90% of my efforts to compromise your systems”.

Flipping the reporting paints a clearer picture of the overall security of your customer: Those that do well you could posit have good policies in place and have built respectable awareness and response capabilities.

For those not in the know, of which I was a member, there is a far out annual summit whereby distinguished researches speak of impending craziness. The summit is about the Singularity.

One of the presenters, Eliezer Yudkowsky, described how the definition of the Singularity has taken on a few different meanings. If you’re interested, you can check this video interview of Eliezer here: http://video.google.com/videoplay?docid=6315588532367156746

Peter Thiel (co-founder of PayPal + lots of other things) offered several interesting snippets, which are captured in this wired blog entry discussing his presentation at the summit: http://blog.wired.co/business/2007/09/peter-thiel-exp.html

After attending the event, I sort of feel teased by the potential of seeing an Artificial General Intelligence within my lifetime. I think I hope to see it, although it sounds like the number of paths to a happy ending is small relative to the number of paths which lead to the end of the world.

There is nothing I’ve seen recently to promote a valuable exercise to do after receiving a security assessment. That is, as the client, what did you see?

Did you have anything alert you? If so, what did it suggest? Did you have enough information to piece together what was happening? (Bonus: do you know which tools were fired towards your IPs?)

The majority of my clients have no clue if anything occurred. That’s bad. Businesses which have little to lose may decide to ignore investing in monitoring and detection, but for others it’s turning a blind eye.

I’m going to dig a little deeper on future exit calls to get more information. I often ask clients if they detected any strange behavior, but there is definitely more room to expand the discussion.

Absent of Cisco wizard skills caused me a little pain yesterday. I remotely configured my Cisco ASA to forward syslog via TCP to a central log host. When I subsequently rebooted the central log host, I lost the ability to establish new connections to anything behind the ASA.

Luckily, I had an established session to a system with a serial connection which enabled me to recover.

I hadn’t run into this before, but I confirmed my experience:

1. If it is unable to log via a defined TCP syslog session, a PIX will not create any new connections (although connections opened before the failure of the session will continue to work). The PIX will log a message to the console stating that it is disallowing new connections.
2. In order to re-establish connection activity, the privileged set logging command, with the correct parameters, will have to be entered or the PIX reloaded.

The cat and mouse game you’re playing to protect your network against the enmity of motivated attackers is perilous.  You’re going to lose. (more and more 0day is coming: see Bejtlich's summary at the bottom of his post for Black Hat Day 1 '07 or Immuntiy Sec's recent presentation which makes a nice point, "Our time to exploit is shorter than your ability to patch")

The problem is you have to play if you want to be connected. Playing then is about choosing the best strategy, or the dominant strategy if one exist. I think there is a dominant strategy for securing your network - traffic analysis (augmented with content & context when available).

From the game theory book Thinking Strategically:

In general, a player has a dominant strategy when he has one course of action that outperforms all others no matter what the other players do.  If a player has such a strategy, his decision becomes very simple: he can choose the dominant strategy without worrying about the rival’s moves.  Therefore it is the first thing one should seek.

Traffic analysis (augmented with content & context) is the best solution when you want pervasive security (i.e. proactive in identifying all types of normal and anomalous activities and strong incident response support due to having the history of communications)

Let me make the point that I do not think traffic analysis in isolation is the winning strategy, but it is a winner when combined with other data freely available on your network. And because I'm tackling this from a defensive perspective (i.e. you have ownership of the network you're protecting), then I'm assuming you get extra defensive observation muscle - snippets of content and context parceled and sent to you by services like syslog.

To take a step back, wikipedia defines traffic analysis as the process of intercepting and examining messages in order to deduce information from patterns in communication. That means learning what's going on from only analyzing the metadata surrounding communication (e.g. the sender, the receiver, the time and length of messages, etc.).

It's amazing what traffic analysis can uncover. Taken from Blink:

The Germans were [in WW II], of course, broadcasting in code, so - at least in the early part of the war - the British couldn't understand what was being said. But that didn't necessarily matter, because before long, just by listening to the cadence of the transmission, the interceptors began to pick up on the individual fists of the German operators, and by doing so, they knew something nearly as important, which was who was doing the sending. [..] After they identified the person who was sending the message, the interceptors would the locate their signal. So now they knew something more. They new who was where.

This goes on and is only a glimpse of what can be learned of course. In IT security, then you can imagine it's possible to:

• uniquely identify all users on a network only by observing patterns (e.g. quirks about how a user types on a keyboard, command sequences a user typically executes, patterns on how they peruse Internet and Intranet sites, plus 100s or 1000s of additional ways)
• to always identify an attacker by observing that nothing the attacker is doing matches any known trusted users' patterns

At Blackhat '07, a presentation on traffic analysis had a slide titled "Why do this?", which speaks to the advantages of traffic analysis (I added stuff between []):

• crypto [i.e. you can't see the content anyways]
• too much data, already [i.e. the need to aggregate and summarize]
• it's easier than analyzing everything
• it's hard to evade [i.e. you'll either catch the attacker or possess the data to reconstruct communication paths]

Now extending traffic analysis with relevant content and context data (syslog, authentication logs, alerts from point products, etc.) allows for very powerful detection for all types of attacks, likely with much greater precision and breadth of coverage versus doing anything else (or relying on a mix of prevention focused systems). Hence, the reason why I think it is a dominant strategy for pervasive security.

There is a major challenge in analyzing all this related information though, which is called the curse of dimensionality. I'll save that one for later.

 

There is debate over the value of visualization techniques. I’m a proponent. Of course the point is to be able to deduce the information you’re looking for faster versus alternatives. Regardless, I’m fatigued by the run of the mill table views, and as such, I like to read about successful stories.

From: http://secviz.org/?q=node/74

“This directed graph reminds me of the social network you might see in a suburban high school, and revealed to us some interesting things, including the existence of a new network monitoring tool quietly installed by a rogue internal unix admin team... us and them, we're having a "come to Jesus" meeting tomorrow ;)”

That rocks.

A relation browser like http://www.der-mo.net/relationBrowser/index.html may make for a great forensics tool. Actually, imagine for a minute all the cool uses.

A timeline carousel: http://simile.mit.edu/timeline/

A big list of visual techniques with thumbnails for fast viewing: http://www.visualcomplexity.com/vc/
http://datamining.typepad.com/data_mining/graphs/index.html

Gapminder is popular: http://www.gapminder.org/downloads/applications/

It’s hard to build a decision support system based on partial views of the world.

My goal is to identify interesting events on a network and to prioritize those events based on sets of attributes. Yes, there are lots of products that do this. But most focus on a slice of the world (e.g. an IDS fires an alert based on a regex match on a single packet). And that is boring.

Doing it for the whole world is where the action is at.

Capture an alert fired from an IDS, check netflow for a session, note a “first-time” event recorded in a syslog message, mix in statistical data mining and learning techniques – and do it all in near real time. This is how things get interesting.

Unfortunately it’s hard to get complete visibility (i.e. get all syslog, all netflow, all application logs, etc.). There must be a point though where I can get enough information to successfully prioritize interesting events. I’m not sure exactly where that’s at, but it’s a fun problem to work on.

The picture is of the inside of IT-Universitetet in Copenhagen where I’m working for a few weeks. The meeting rooms all jet out into the open space in the middle – a pretty cool design.

I recently went on a vacation to Scotland. While I was there, I read a wonderful book about St. Kilda and the near utopian society that lived there until 1930, it is no uninhabited. St. Kilda is one of the most remote Hebrides and it's a particularly rocky set of islands that isn't very easy to approach by boat. It had been inhabited for at least 2000 years and isolated from the rest of the world for most of that time. As a result the islanders depended upon each other a great deal. Everyone had a say on every issue, they shared resources, crime didn't really exist. And then the outside world crashed in on them, people started visiting the island as tourists and eventually the St. Kildans' culture fell apart and they asked to be evacuated from the island.

What was fascinating to me is that Kilda wasn't utpoian by design, it was by need. Everyone depended upon everyone else on the island of only a couple hundred people. There weren't people that just did nothing while other people caught food. (Also fascinating, they didn't fish, they were fowlers and climbed up cliffs and caught sea birds) Every single member of the society pulled their weight and did something that the whole group needed. They had no leader, they had a daily "parliament" in which every man got to talk and an equal vote. While it wasn't easy living, outsides that looked in described the society as a utopia. Even more remarkably, every person on the island was equal when in the rest of the British Isles clans and royalty controlled society.

I compare this to the software industry. At only two companies have I ever seen "the process" work. One was IBM and it worked because everyone was dedicated to making it work and there were a lot of full time process people that pushed the process through as well as a couple super heroes that did above and beyond the call of duty to keep things floating. It was very expensive to make software and, honestly, it wasn't the most fun I've ever had. The other was a small startup where we had almost no management and a very tight team of developers and testers that all wanted to make the company successful; the executive staff was completely hands off. People didn't simply do their job and throw some output over the fence, everyone did more than their job description and there was almost no outside pressure in and we were fairly Agile and had enough process to provide some safety nets but we moved quickly. Also, everyone on the team didn't know anything but success, there was a lot of pride and "good enough" wasn't good enough for us and it was great while it lasted.

Everywhere else, they pretended. Process wasn't a need, so much as an excuse. (The lack of any process is the same thing, it's a process in its own right and there are usually bullies somewhere in the organization and it involves some sort of punishment feedback loop) I had this great experience once where I was encouraged to push back, but not until I exhausted every other option (that meant working 10+ hours 7 days a week, or at least that is how it was measured when you did "push back..") which is silly because it requires you to actually fail to prove that the plan didn't work in the first place. I'm watching it on a great scale now, there is a great effort being made to appear to follow a process but there really isn't one. It's easy for everyone to "do" the process but simply "doing" it doesn't make the product better or reduce risk or really even mean much of anything. It requires collective discipline, collective sacrifice and compromise, collective give and take and a lot of trust. Everyone in the organization has to take part, every single stakeholder has to be part of it. No process can create time when there isn't any or make an average team in to a great team. It's easy to pretend though. I'll talk more about that next time.

What happens when the test environment operated by MasterCard (they “own” the testing lab) is misbehaving? I know. They yank the wheel, swerve away from responsibility, and point to the PCI council. And PCI? They point back. Beautiful, no?

You see because they refuse to disclose missed results to you they duck responsibility for anything that may have been their fault. They also clearly imply if anything is missed in your attempts to identify vulnerabilities then it is surely your fault or a problem with the tools you used.

I love it: No clear pass criteria, no way to challenge a decision, and no transparency of what or how they are doing. For all this great service you get to spend thousands every year!

So what happens when you call bullshit and raise hell? They pass you. :) Let me not forget to mention we had a few extra bullets in our clip they may have unexpected us to have – bullets provided to us by friends with information.

Be forewarned; this process has serious issues.

The world of the PCI Security Standards, ASVs (Approved Scanning Vendors), and commercial scan vendors is, from my limited interactions, not exactly on the straight.

Having recently spent considerable time preparing, scanning, and writing reports with the explicit goal of becoming an ASV, I’m disturbed by my communications with all involved.

It doesn’t help that the pass criteria to become an ASV is not clear. Is it based on discovering all vulnerabilities on their test network? A subset? Which parts are subjectively reviewed?

I like to use Qualys to baseline vulnerabilities, which a test representative caught as one of the tools we’d be using based on the source IP blocks for scanning. He said something like “If you use Qualys, you’ll get 95% of what you need”. From that I guessed the example web application would have vulnerabilities which would be missed by Qualys and other network-based tools. As expected, that was true.

Fast forward. We received feedback that our reports have not been reviewed because Qualys changed something recently causing expected results to be absent which PCI requires for passing. The representative said “If you’d have scanned a month ago with Qualys, you’d have passed with flying colors”. He added that they are in communication with Qualys to resolve the issue.

Nevermind that Qualys was only one of the tools we used and we had added vulnerabilities not discovered by the popular free or commercial scanners. It was clear the representative didn’t review the report, which he said as well (and may have done to protect us from an automatic failure -- even though our current pass status is pending). But they would not reveal the gaps, which obviously makes it hard to understand what the problem is. I appealed by mentioning I had added vulnerabilities not discovered by Qualys. He then modified his previous statements by saying he in fact spot checked the reports and the items he was looking for were absent. My gut reaction to all this: bullshit.

The connections apparent make for a nice racket. You pay lots of money to PCI. PCI “communicates” with selective vendors to ensure all the vulnerabilities they expect to be discovered are discovered. You pay money to the scan vendors. And what about if you find additional vulnerabilities or a superset? Sounds like they don’t check and don’t care. I would think the idea behind all this is to make sure you are adding sufficient value to entities subject to PCI regulations (even if that means you didn't catch 100% of everything bad). If passing simply means doing a blind Qualys scan when it works right (i.e. does what PCI wants), well then, you now know what to do to as an attacker -- just go after something Qualys doesn't check.

So much for trusting this process and what it does to vet competent assessment companies.

It can take some digging to discover if you’ve successfully injected any code into a web application. I was using the ALL-FUZZ-STRINGS that comes with Suru (added additional strings from sources like ha.ckers.org XSS Cheat Sheet) to run through a list of popular input validation attacks.

Suru is a Man In The Middle (MITM) proxy that sits between the user's browser and the web application.

I was keying on the fuzzy logic results window in Suru which helps the assessor determine if any attacks were successful.

The Fuzzy logic trigger box is used to fine-tune the results received during a fuzzing run. The user has multiple options that range from content extraction to brute force fine-tuning. The results from the fuzzing run can be mined or filtered for false positives, positives or false negative findings.

Here is a screenshot of the some of strings that are included in the ALL-FUZZ-STRINGS

During this phase of testing I was interested to see if any of the XSS type attacks succeeded. The fuzzy logic results window shows calculated scores that are derived from running an equation on each response and on a base response.

In order to conduct Fuzz logic or content extraction runs, Suru needs to compare the results to a base response. For example, if we need to brute force an application we need to compare a failed attempt with a successful attempt to gauge successes or failures.

I was surprised when all of the scores were basically the same – implying that none of the attacks caused a response which was measurably different from the base response. I already knew XSS/code injection vulnerabilities existed in this part of the application.

Upon a little investigation, it was clear the reason why Suru’s fuzzing scores were remaining more or less the same. For this particular web application, the response page created from submitting values was simply an acknowledgement of the POST request.

This response page was always the same -- there was no deviation from the base response. But by manually crawling the site and clicking on deeper links, I ran across a link where the injected code was executed and displayed in the response. One of the attacks was successful.

The take-away here is that it is important to manually walk the web application after performing XSS, code injection, etc. attacks to really see if any were successful. The attacks may cause the web application to respond with the exact expected attack string (i.e. if you put in as part of the attack the string “URL=javascript:alert('XSS-IS-HERE');”, then you can scan the HTML response for the string “XSS-IS-HERE”), but the web application may modify the string enough to evade automated checks yet still be vulnerable.

And in the case above only discovered by manually crawling the app to see where the attack successfully surfaces.

A lot of folks seem to be interested in booting linux devices faster right now. Be it by parallel init processes such as SMF, launchd and initng, or various other techniques. One thing that drives me nuts and has for ages is how long my computer spends doing who knows what in the BIOS.

I've done a little BIOS work in my career and I honestly don't know what they do that takes so long. Other than drives spinning up, most things that are reset and configured by a BIOS can be done in times that are faster than a human can notice. I have an IBM system that takes about 3 minutes before the BIOS is done. I support some Gateways that I think take longer. It's long enough that I almost always get a little bit angry waiting for it. I don't reboot systems that often but if it takes 5 minutes, that is 5 minutes you have to wait to make sure it comes back. I'm fairly certain that there are intentional delays to display vendor logos which is sort of funny to me because they've already got my money at that point. Then on the other side of the fence, a normal kernel (BSD, Linux or any that I can really think of) seems to boot as fast as you can get it off the disk and then it's running user space code like init or launchd.

There is a great way to skip that time and it seems completely under utilized. kexec was written by Eric Biederman over the last 5 years and it is in the main line Linux kernel. Since 2000, I know of at least 3 other projects that reached different degrees of maturity that do the same thing: LOBO, bootimg, and "2 kernel monte." They all do fundamentally the same thing but kexec is by far the most complete and polished. Kexec replaces the current kernel in memory with a new one. You select a kernel off of a disk, load it in to memory in a specific way, turn off protected mode and re-enable real mode on the processor and then jump to the new kernel. Effectively this is a reboot, only it completely skips the BIOS.

Kexec is included as an RPM in Fedora Core, so you probably don't have to do very much to use it. It's very simple though. You just run kexec and specify a kernel, an initial ramdisk image (if you run with one) and other kernel arguments and then issue a normal reboot (look in your grub menu.lst file if you don't know what these are). The system will go through the normal shutdown process and then at the end you'll immediately see the new kernel booting up.

It doesn't take much imagination to think of other uses for this kind of technology. If you were to use some exotic filesystem like NILFS or OCFS2, or reiserfs4 that grub won't boot, you could boot any partition you can mount with any kernel that can read that partition. Kexec and LOBO were born to help with clustering problems, boot to some small version of linux that downloads the version of linux that does the work from the network and then boot to it. You also could very easily build a DVD that verified your system's integrity before booting it. Or you can just skip those annoying BIOS messages.

For a quick wireless project I grabbed the Ubiquiti SuperRange Cardbus (300mW 802.11 a/b/g), downloaded the latest BackTrack ISO, plugged in the card and booted the CD on an old Inspiron 8100 laptop.

Everything worked perfectly on the first try and I was a little surprised to find 40+ networks via Kismet in my residential neighborhood.  If you’re into the wireless stuff, this combination worked great for me and I recommend the Ubiquiti with external antenna.

I just finished a workshop covering the use of Data Stream Analysis. Its necessity is driven by the need to analyze massive volumes of data (e.g. system and network events) in near real time – essential given how fast you will hit your head on the insertion rate ceiling using standard relational databases.

Off the shelf DBs (PostgreSQL, MySQL, Oracle, etc.) are unable to simultaneously commit thousands of events per second while performing complex queries. To have a chance of analyzing events in reasonable amounts of time you must analyze the incoming streams of data before inserting the data into a database.

I ran into this scenario last year building a central log server using off the shelf components. Even a few dozen servers can stream events fast enough where you realize pretty quickly all the typical open source based how-to’s on building a system that can store, correlate, and alert are inadequate. Data stream processing is required when things get big.

I've been too busy doing other things the last few months to post much. I haven't seen this issue really addressed anywhere but it's mentioned from time to time. It's kind of a quick rationalization and it definitely has some appeal to some form of logic.

At a glance, it makes logical sense. The difference between the best engineers and the worst isn't that great (in the grand scheme of things) and the top products are usually built by good ones that all tend to kind of converge to the same quality level. So if the engineers are about the same quality then the output should have roughly the same number of defects provided that they are using similar technologies and tools. Further, the more popular product will have more eyes looking at it and so more problems will be found.

The popular context for this would be Windows vs. OS X. Is OS X any more secure than Windows? Or is it just attacked less often? Well what about OpenBSD, if it was as popular as Linux would it have the same number of security problems? I tend not to think so.

On my last several ‘exit calls’ for security assessments I’ve wanted to ask the customer if they had anything alerting them to the activities performed.

The obvious need for detection is a tiresome mantra to repeat, given that prevention will always fail. In fact, is it not better to log all activities (e.g. syslog, netflow, successful sessions, etc.) in spite of using prevention tools? If knowing you’ve been compromised is a better state that not knowing, then isn’t it better to pay appropriate attention to all the events versus haphazardly trusting prevention solutions?

I just finished an external security assessment for a Bank which had an IPS enabled firewall. They requested two rounds of scanning: one with the IPS features enabled and the other with them disabled. Results: no difference. This from normal to aggressive scanning (full 65k scans, full vuln. scans from multiple tools, few metasploit shots, exhaustive brute forcing, etc.) and without any efforts to be elusive.

I’m betting if I ask this client if he noticed any activity spikes or if he was alerted to anything he’ll say no. Furthermore, I bet he has nothing setup to help him easily go check.

I’m running across more and more of these where it seems the first indicators of something bad is when actual fraud occurs. Compromise, theft of data, spread of attackers’ control -- all missed opportunities to detect and contain because of an unbalanced reliance on prevention tools.

How do you balance a giant list of customer feature requests with your in-house splashes of innovative ideas?

It seems really hard not to get buried coding incremental improvements while keeping your head above ground. With your head buried, you’ll eventually lose sight of the vision and get blindsided by competitors.

Several start-ups I’ve played at were driven almost entirely by customer requests. I’m not debating the value of that. But when it is used to control all the development cycles you begin to create a culture allergic to creativity and risk.

Good team dynamics can help tremendously for encouraging members to be creative – I think the challenge is keeping it that way while championing a customer driven style of development.