ClearNet Security

For those not in the know, of which I was a member, there is a far out annual summit whereby distinguished researches speak of impending craziness. The summit is about the Singularity.

One of the presenters, Eliezer Yudkowsky, described how the definition of the Singularity has taken on a few different meanings. If you’re interested, you can check this video interview of Eliezer here: http://video.google.com/videoplay?docid=6315588532367156746

Peter Thiel (co-founder of PayPal + lots of other things) offered several interesting snippets, which are captured in this wired blog entry discussing his presentation at the summit: http://blog.wired.co/business/2007/09/peter-thiel-exp.html

After attending the event, I sort of feel teased by the potential of seeing an Artificial General Intelligence within my lifetime. I think I hope to see it, although it sounds like the number of paths to a happy ending is small relative to the number of paths which lead to the end of the world.

There is nothing I’ve seen recently to promote a valuable exercise to do after receiving a security assessment. That is, as the client, what did you see?

Did you have anything alert you? If so, what did it suggest? Did you have enough information to piece together what was happening? (Bonus: do you know which tools were fired towards your IPs?)

The majority of my clients have no clue if anything occurred. That’s bad. Businesses which have little to lose may decide to ignore investing in monitoring and detection, but for others it’s turning a blind eye.

I’m going to dig a little deeper on future exit calls to get more information. I often ask clients if they detected any strange behavior, but there is definitely more room to expand the discussion.

Absent of Cisco wizard skills caused me a little pain yesterday. I remotely configured my Cisco ASA to forward syslog via TCP to a central log host. When I subsequently rebooted the central log host, I lost the ability to establish new connections to anything behind the ASA.

Luckily, I had an established session to a system with a serial connection which enabled me to recover.

I hadn’t run into this before, but I confirmed my experience:

1. If it is unable to log via a defined TCP syslog session, a PIX will not create any new connections (although connections opened before the failure of the session will continue to work). The PIX will log a message to the console stating that it is disallowing new connections.
2. In order to re-establish connection activity, the privileged set logging command, with the correct parameters, will have to be entered or the PIX reloaded.

The cat and mouse game you’re playing to protect your network against the enmity of motivated attackers is perilous.  You’re going to lose. (more and more 0day is coming: see Bejtlich's summary at the bottom of his post for Black Hat Day 1 '07 or Immuntiy Sec's recent presentation which makes a nice point, "Our time to exploit is shorter than your ability to patch")

The problem is you have to play if you want to be connected. Playing then is about choosing the best strategy, or the dominant strategy if one exist. I think there is a dominant strategy for securing your network - traffic analysis (augmented with content & context when available).

From the game theory book Thinking Strategically:

In general, a player has a dominant strategy when he has one course of action that outperforms all others no matter what the other players do.  If a player has such a strategy, his decision becomes very simple: he can choose the dominant strategy without worrying about the rival’s moves.  Therefore it is the first thing one should seek.

Traffic analysis (augmented with content & context) is the best solution when you want pervasive security (i.e. proactive in identifying all types of normal and anomalous activities and strong incident response support due to having the history of communications)

Let me make the point that I do not think traffic analysis in isolation is the winning strategy, but it is a winner when combined with other data freely available on your network. And because I'm tackling this from a defensive perspective (i.e. you have ownership of the network you're protecting), then I'm assuming you get extra defensive observation muscle - snippets of content and context parceled and sent to you by services like syslog.

To take a step back, wikipedia defines traffic analysis as the process of intercepting and examining messages in order to deduce information from patterns in communication. That means learning what's going on from only analyzing the metadata surrounding communication (e.g. the sender, the receiver, the time and length of messages, etc.).

It's amazing what traffic analysis can uncover. Taken from Blink:

The Germans were [in WW II], of course, broadcasting in code, so - at least in the early part of the war - the British couldn't understand what was being said. But that didn't necessarily matter, because before long, just by listening to the cadence of the transmission, the interceptors began to pick up on the individual fists of the German operators, and by doing so, they knew something nearly as important, which was who was doing the sending. [..] After they identified the person who was sending the message, the interceptors would the locate their signal. So now they knew something more. They new who was where.

This goes on and is only a glimpse of what can be learned of course. In IT security, then you can imagine it's possible to:

• uniquely identify all users on a network only by observing patterns (e.g. quirks about how a user types on a keyboard, command sequences a user typically executes, patterns on how they peruse Internet and Intranet sites, plus 100s or 1000s of additional ways)
• to always identify an attacker by observing that nothing the attacker is doing matches any known trusted users' patterns

At Blackhat '07, a presentation on traffic analysis had a slide titled "Why do this?", which speaks to the advantages of traffic analysis (I added stuff between []):

• crypto [i.e. you can't see the content anyways]
• too much data, already [i.e. the need to aggregate and summarize]
• it's easier than analyzing everything
• it's hard to evade [i.e. you'll either catch the attacker or possess the data to reconstruct communication paths]

Now extending traffic analysis with relevant content and context data (syslog, authentication logs, alerts from point products, etc.) allows for very powerful detection for all types of attacks, likely with much greater precision and breadth of coverage versus doing anything else (or relying on a mix of prevention focused systems). Hence, the reason why I think it is a dominant strategy for pervasive security.

There is a major challenge in analyzing all this related information though, which is called the curse of dimensionality. I'll save that one for later.

 

There is debate over the value of visualization techniques. I’m a proponent. Of course the point is to be able to deduce the information you’re looking for faster versus alternatives. Regardless, I’m fatigued by the run of the mill table views, and as such, I like to read about successful stories.

From: http://secviz.org/?q=node/74

“This directed graph reminds me of the social network you might see in a suburban high school, and revealed to us some interesting things, including the existence of a new network monitoring tool quietly installed by a rogue internal unix admin team... us and them, we're having a "come to Jesus" meeting tomorrow ;)”

That rocks.

A relation browser like http://www.der-mo.net/relationBrowser/index.html may make for a great forensics tool. Actually, imagine for a minute all the cool uses.

A timeline carousel: http://simile.mit.edu/timeline/

A big list of visual techniques with thumbnails for fast viewing: http://www.visualcomplexity.com/vc/
http://datamining.typepad.com/data_mining/graphs/index.html

Gapminder is popular: http://www.gapminder.org/downloads/applications/

It’s hard to build a decision support system based on partial views of the world.

My goal is to identify interesting events on a network and to prioritize those events based on sets of attributes. Yes, there are lots of products that do this. But most focus on a slice of the world (e.g. an IDS fires an alert based on a regex match on a single packet). And that is boring.

Doing it for the whole world is where the action is at.

Capture an alert fired from an IDS, check netflow for a session, note a “first-time” event recorded in a syslog message, mix in statistical data mining and learning techniques – and do it all in near real time. This is how things get interesting.

Unfortunately it’s hard to get complete visibility (i.e. get all syslog, all netflow, all application logs, etc.). There must be a point though where I can get enough information to successfully prioritize interesting events. I’m not sure exactly where that’s at, but it’s a fun problem to work on.

The picture is of the inside of IT-Universitetet in Copenhagen where I’m working for a few weeks. The meeting rooms all jet out into the open space in the middle – a pretty cool design.

I recently went on a vacation to Scotland. While I was there, I read a wonderful book about St. Kilda and the near utopian society that lived there until 1930, it is no uninhabited. St. Kilda is one of the most remote Hebrides and it's a particularly rocky set of islands that isn't very easy to approach by boat. It had been inhabited for at least 2000 years and isolated from the rest of the world for most of that time. As a result the islanders depended upon each other a great deal. Everyone had a say on every issue, they shared resources, crime didn't really exist. And then the outside world crashed in on them, people started visiting the island as tourists and eventually the St. Kildans' culture fell apart and they asked to be evacuated from the island.

What was fascinating to me is that Kilda wasn't utpoian by design, it was by need. Everyone depended upon everyone else on the island of only a couple hundred people. There weren't people that just did nothing while other people caught food. (Also fascinating, they didn't fish, they were fowlers and climbed up cliffs and caught sea birds) Every single member of the society pulled their weight and did something that the whole group needed. They had no leader, they had a daily "parliament" in which every man got to talk and an equal vote. While it wasn't easy living, outsides that looked in described the society as a utopia. Even more remarkably, every person on the island was equal when in the rest of the British Isles clans and royalty controlled society.

I compare this to the software industry. At only two companies have I ever seen "the process" work. One was IBM and it worked because everyone was dedicated to making it work and there were a lot of full time process people that pushed the process through as well as a couple super heroes that did above and beyond the call of duty to keep things floating. It was very expensive to make software and, honestly, it wasn't the most fun I've ever had. The other was a small startup where we had almost no management and a very tight team of developers and testers that all wanted to make the company successful; the executive staff was completely hands off. People didn't simply do their job and throw some output over the fence, everyone did more than their job description and there was almost no outside pressure in and we were fairly Agile and had enough process to provide some safety nets but we moved quickly. Also, everyone on the team didn't know anything but success, there was a lot of pride and "good enough" wasn't good enough for us and it was great while it lasted.

Everywhere else, they pretended. Process wasn't a need, so much as an excuse. (The lack of any process is the same thing, it's a process in its own right and there are usually bullies somewhere in the organization and it involves some sort of punishment feedback loop) I had this great experience once where I was encouraged to push back, but not until I exhausted every other option (that meant working 10+ hours 7 days a week, or at least that is how it was measured when you did "push back..") which is silly because it requires you to actually fail to prove that the plan didn't work in the first place. I'm watching it on a great scale now, there is a great effort being made to appear to follow a process but there really isn't one. It's easy for everyone to "do" the process but simply "doing" it doesn't make the product better or reduce risk or really even mean much of anything. It requires collective discipline, collective sacrifice and compromise, collective give and take and a lot of trust. Everyone in the organization has to take part, every single stakeholder has to be part of it. No process can create time when there isn't any or make an average team in to a great team. It's easy to pretend though. I'll talk more about that next time.

What happens when the test environment operated by MasterCard (they “own” the testing lab) is misbehaving? I know. They yank the wheel, swerve away from responsibility, and point to the PCI council. And PCI? They point back. Beautiful, no?

You see because they refuse to disclose missed results to you they duck responsibility for anything that may have been their fault. They also clearly imply if anything is missed in your attempts to identify vulnerabilities then it is surely your fault or a problem with the tools you used.

I love it: No clear pass criteria, no way to challenge a decision, and no transparency of what or how they are doing. For all this great service you get to spend thousands every year!

So what happens when you call bullshit and raise hell? They pass you. :) Let me not forget to mention we had a few extra bullets in our clip they may have unexpected us to have – bullets provided to us by friends with information.

Be forewarned; this process has serious issues.

The world of the PCI Security Standards, ASVs (Approved Scanning Vendors), and commercial scan vendors is, from my limited interactions, not exactly on the straight.

Having recently spent considerable time preparing, scanning, and writing reports with the explicit goal of becoming an ASV, I’m disturbed by my communications with all involved.

It doesn’t help that the pass criteria to become an ASV is not clear. Is it based on discovering all vulnerabilities on their test network? A subset? Which parts are subjectively reviewed?

I like to use Qualys to baseline vulnerabilities, which a test representative caught as one of the tools we’d be using based on the source IP blocks for scanning. He said something like “If you use Qualys, you’ll get 95% of what you need”. From that I guessed the example web application would have vulnerabilities which would be missed by Qualys and other network-based tools. As expected, that was true.

Fast forward. We received feedback that our reports have not been reviewed because Qualys changed something recently causing expected results to be absent which PCI requires for passing. The representative said “If you’d have scanned a month ago with Qualys, you’d have passed with flying colors”. He added that they are in communication with Qualys to resolve the issue.

Nevermind that Qualys was only one of the tools we used and we had added vulnerabilities not discovered by the popular free or commercial scanners. It was clear the representative didn’t review the report, which he said as well (and may have done to protect us from an automatic failure -- even though our current pass status is pending). But they would not reveal the gaps, which obviously makes it hard to understand what the problem is. I appealed by mentioning I had added vulnerabilities not discovered by Qualys. He then modified his previous statements by saying he in fact spot checked the reports and the items he was looking for were absent. My gut reaction to all this: bullshit.

The connections apparent make for a nice racket. You pay lots of money to PCI. PCI “communicates” with selective vendors to ensure all the vulnerabilities they expect to be discovered are discovered. You pay money to the scan vendors. And what about if you find additional vulnerabilities or a superset? Sounds like they don’t check and don’t care. I would think the idea behind all this is to make sure you are adding sufficient value to entities subject to PCI regulations (even if that means you didn't catch 100% of everything bad). If passing simply means doing a blind Qualys scan when it works right (i.e. does what PCI wants), well then, you now know what to do to as an attacker -- just go after something Qualys doesn't check.

So much for trusting this process and what it does to vet competent assessment companies.

It can take some digging to discover if you’ve successfully injected any code into a web application. I was using the ALL-FUZZ-STRINGS that comes with Suru (added additional strings from sources like ha.ckers.org XSS Cheat Sheet) to run through a list of popular input validation attacks.

Suru is a Man In The Middle (MITM) proxy that sits between the user's browser and the web application.

I was keying on the fuzzy logic results window in Suru which helps the assessor determine if any attacks were successful.

The Fuzzy logic trigger box is used to fine-tune the results received during a fuzzing run. The user has multiple options that range from content extraction to brute force fine-tuning. The results from the fuzzing run can be mined or filtered for false positives, positives or false negative findings.

Here is a screenshot of the some of strings that are included in the ALL-FUZZ-STRINGS

During this phase of testing I was interested to see if any of the XSS type attacks succeeded. The fuzzy logic results window shows calculated scores that are derived from running an equation on each response and on a base response.

In order to conduct Fuzz logic or content extraction runs, Suru needs to compare the results to a base response. For example, if we need to brute force an application we need to compare a failed attempt with a successful attempt to gauge successes or failures.

I was surprised when all of the scores were basically the same – implying that none of the attacks caused a response which was measurably different from the base response. I already knew XSS/code injection vulnerabilities existed in this part of the application.

Upon a little investigation, it was clear the reason why Suru’s fuzzing scores were remaining more or less the same. For this particular web application, the response page created from submitting values was simply an acknowledgement of the POST request.

This response page was always the same -- there was no deviation from the base response. But by manually crawling the site and clicking on deeper links, I ran across a link where the injected code was executed and displayed in the response. One of the attacks was successful.

The take-away here is that it is important to manually walk the web application after performing XSS, code injection, etc. attacks to really see if any were successful. The attacks may cause the web application to respond with the exact expected attack string (i.e. if you put in as part of the attack the string “URL=javascript:alert('XSS-IS-HERE');”, then you can scan the HTML response for the string “XSS-IS-HERE”), but the web application may modify the string enough to evade automated checks yet still be vulnerable.

And in the case above only discovered by manually crawling the app to see where the attack successfully surfaces.

A lot of folks seem to be interested in booting linux devices faster right now. Be it by parallel init processes such as SMF, launchd and initng, or various other techniques. One thing that drives me nuts and has for ages is how long my computer spends doing who knows what in the BIOS.

I've done a little BIOS work in my career and I honestly don't know what they do that takes so long. Other than drives spinning up, most things that are reset and configured by a BIOS can be done in times that are faster than a human can notice. I have an IBM system that takes about 3 minutes before the BIOS is done. I support some Gateways that I think take longer. It's long enough that I almost always get a little bit angry waiting for it. I don't reboot systems that often but if it takes 5 minutes, that is 5 minutes you have to wait to make sure it comes back. I'm fairly certain that there are intentional delays to display vendor logos which is sort of funny to me because they've already got my money at that point. Then on the other side of the fence, a normal kernel (BSD, Linux or any that I can really think of) seems to boot as fast as you can get it off the disk and then it's running user space code like init or launchd.

There is a great way to skip that time and it seems completely under utilized. kexec was written by Eric Biederman over the last 5 years and it is in the main line Linux kernel. Since 2000, I know of at least 3 other projects that reached different degrees of maturity that do the same thing: LOBO, bootimg, and "2 kernel monte." They all do fundamentally the same thing but kexec is by far the most complete and polished. Kexec replaces the current kernel in memory with a new one. You select a kernel off of a disk, load it in to memory in a specific way, turn off protected mode and re-enable real mode on the processor and then jump to the new kernel. Effectively this is a reboot, only it completely skips the BIOS.

Kexec is included as an RPM in Fedora Core, so you probably don't have to do very much to use it. It's very simple though. You just run kexec and specify a kernel, an initial ramdisk image (if you run with one) and other kernel arguments and then issue a normal reboot (look in your grub menu.lst file if you don't know what these are). The system will go through the normal shutdown process and then at the end you'll immediately see the new kernel booting up.

It doesn't take much imagination to think of other uses for this kind of technology. If you were to use some exotic filesystem like NILFS or OCFS2, or reiserfs4 that grub won't boot, you could boot any partition you can mount with any kernel that can read that partition. Kexec and LOBO were born to help with clustering problems, boot to some small version of linux that downloads the version of linux that does the work from the network and then boot to it. You also could very easily build a DVD that verified your system's integrity before booting it. Or you can just skip those annoying BIOS messages.

For a quick wireless project I grabbed the Ubiquiti SuperRange Cardbus (300mW 802.11 a/b/g), downloaded the latest BackTrack ISO, plugged in the card and booted the CD on an old Inspiron 8100 laptop.

Everything worked perfectly on the first try and I was a little surprised to find 40+ networks via Kismet in my residential neighborhood.  If you’re into the wireless stuff, this combination worked great for me and I recommend the Ubiquiti with external antenna.

I just finished a workshop covering the use of Data Stream Analysis. Its necessity is driven by the need to analyze massive volumes of data (e.g. system and network events) in near real time – essential given how fast you will hit your head on the insertion rate ceiling using standard relational databases.

Off the shelf DBs (PostgreSQL, MySQL, Oracle, etc.) are unable to simultaneously commit thousands of events per second while performing complex queries. To have a chance of analyzing events in reasonable amounts of time you must analyze the incoming streams of data before inserting the data into a database.

I ran into this scenario last year building a central log server using off the shelf components. Even a few dozen servers can stream events fast enough where you realize pretty quickly all the typical open source based how-to’s on building a system that can store, correlate, and alert are inadequate. Data stream processing is required when things get big.

I've been too busy doing other things the last few months to post much. I haven't seen this issue really addressed anywhere but it's mentioned from time to time. It's kind of a quick rationalization and it definitely has some appeal to some form of logic.

At a glance, it makes logical sense. The difference between the best engineers and the worst isn't that great (in the grand scheme of things) and the top products are usually built by good ones that all tend to kind of converge to the same quality level. So if the engineers are about the same quality then the output should have roughly the same number of defects provided that they are using similar technologies and tools. Further, the more popular product will have more eyes looking at it and so more problems will be found.

The popular context for this would be Windows vs. OS X. Is OS X any more secure than Windows? Or is it just attacked less often? Well what about OpenBSD, if it was as popular as Linux would it have the same number of security problems? I tend not to think so.

On my last several ‘exit calls’ for security assessments I’ve wanted to ask the customer if they had anything alerting them to the activities performed.

The obvious need for detection is a tiresome mantra to repeat, given that prevention will always fail. In fact, is it not better to log all activities (e.g. syslog, netflow, successful sessions, etc.) in spite of using prevention tools? If knowing you’ve been compromised is a better state that not knowing, then isn’t it better to pay appropriate attention to all the events versus haphazardly trusting prevention solutions?

I just finished an external security assessment for a Bank which had an IPS enabled firewall. They requested two rounds of scanning: one with the IPS features enabled and the other with them disabled. Results: no difference. This from normal to aggressive scanning (full 65k scans, full vuln. scans from multiple tools, few metasploit shots, exhaustive brute forcing, etc.) and without any efforts to be elusive.

I’m betting if I ask this client if he noticed any activity spikes or if he was alerted to anything he’ll say no. Furthermore, I bet he has nothing setup to help him easily go check.

I’m running across more and more of these where it seems the first indicators of something bad is when actual fraud occurs. Compromise, theft of data, spread of attackers’ control -- all missed opportunities to detect and contain because of an unbalanced reliance on prevention tools.

How do you balance a giant list of customer feature requests with your in-house splashes of innovative ideas?

It seems really hard not to get buried coding incremental improvements while keeping your head above ground. With your head buried, you’ll eventually lose sight of the vision and get blindsided by competitors.

Several start-ups I’ve played at were driven almost entirely by customer requests. I’m not debating the value of that. But when it is used to control all the development cycles you begin to create a culture allergic to creativity and risk.

Good team dynamics can help tremendously for encouraging members to be creative – I think the challenge is keeping it that way while championing a customer driven style of development.

I read a WSJ Opinion article today by Harold Evans titled “The American Way” which, in a sense, paralleled the ideas presented by Tom Kelly’s recent presentation at RSA around innovation.

From WSJ:

Efficiency, once the be-all and end-all, is no longer considered enough for survival in the world economy. In a global marketplace, efficiency – and the cost cutting associated with it – is essential but may not be enough when competitors in China and India can discount you to death with demographics.

That got me to thinking in our industry how often we see claims of “innovation”, but which are really not.

We should reserve the terms ‘innovation’ and ‘innovators’ for real change and not confuse it with different functions.
[…]
Entrepreneurship, the assumption of risk, may not be innovative at all. You assume risk if you open a new auto dealership, but this is not innovative unless you are the first.

Blend in the key points from Tom Kelly’s presentation (or books) and you’ll see how powerful it is to continually aspire for true innovation.

“Tom has observed a number of roles that people can play in an organization to foster innovation and new ideas while offering an effective counter to naysayers. Among these approaches are the Anthropologist, the person who goes into the field to see how customers use and respond to products, to come up with new innovations; the Cross-Pollinator, who mixes and matches ideas, widely disparate people, and technologies to create new ideas that can drive growth; and the Hurdler, who instantly looks for ways to overcome the limits and challenges to any situation.”

Tying this back to the security industry, I’ve been stricken with the illness inducing problem of inspecting massive data sets for important events. A real innovative breakthrough to me would be if someone built an easy-to-use and easy-to-manage system (one that accepts all possible data sources) that handles the crushing volume of disparate events on enterprise networks while truly notifying me of only important events.

To cite a small example, I’m installing several pieces of “security software” for a client considerate of threats: Snort, OSSEC HIDS, Central Log Aggregator, etc. I’ve played with several products which purport to do all I wish, but none of them are succeeding or innovating to me – none feel like they are offering real change or doing something other than offering different functions. Maybe I’ll try the Hurdler role for some time to see what I can do.

How about trying to play a role?

Last week I attended presentations from the Americas Growth Capital Conference. Several of the presentations featured prominent CEO/CTOs in the security products game.

I connected with a view echoed by a panel discussing the difficulty of selling security point solutions.

They raised the issue of their desire to set their customers’ expectations that their products will fail. Catastrophic failures, they said, are rare events. Their products are designed to work most of the time, but they repeated the fact that it is too expensive to build products which escape all bad things.

To limit the embarrassment of a product failure, they work on educating their customers and suggesting to each to invest in a good incident response strategy. This keeps the prices competitive, establishes trust, helps to level expectations, and addresses product failures caused from rare events.

I don’t know how well they convey this when the goal is to sell, but I liked the message.

This is worth the extra effort -- ensuring you have bash 3.x installed (3.2 is the current version). It enables you to set an environment variable to record the time and date of executed commands.

$ export HISTTIMEFORMAT="%h/%m - %H:%M:%S "

$ history | tail -2
1006 Jan/01 - 19:24:02 ls
1007 Jan/01 - 19:24:07 history | tail -2

A potential forensic treasure when needed.

Along with others I’m sure, we’re seeing a nascent phase for security services firms in the business of performing external security assessments. Rather than blast through the battery of scanners the game is changing to include tasks which traditionally fell outside the scope of a basic external assessment.

I like the changes because it follows more of a risk assessment approach. For example, as part of an “external assessment pack” we might obtain the configurations for routers, firewalls, switches, wireless access points, concentrators, review relevant policies, architecture diagrams, and to the best of the client’s ability incorporate their identification of segments or servers which touch sensitive data.

Contrary to acquiring infrastructure visibility, running blind external assessments are losing effectiveness - especially with the growth of the commercial exploit factories (e.g. argeniss, gleg, core security, immunity sec ). Making a statement like “We scanned with Nmap, Nessus, Qualys, Webinspect, checked Secunia, SF, etc., and found no major findings” is losing value - its assurance offered more yesterday than today.

My participation in a few recent incident response/forensic assignments is testament to the weakening value of blind assessments; the cause of each compromise would not have been revealed as a vulnerability by vulnerability scanners.