ClearNet Security

I read a WSJ Opinion article today by Harold Evans titled “The American Way” which, in a sense, paralleled the ideas presented by Tom Kelly’s recent presentation at RSA around innovation.

From WSJ:

Efficiency, once the be-all and end-all, is no longer considered enough for survival in the world economy. In a global marketplace, efficiency – and the cost cutting associated with it – is essential but may not be enough when competitors in China and India can discount you to death with demographics.

That got me to thinking in our industry how often we see claims of “innovation”, but which are really not.

We should reserve the terms ‘innovation’ and ‘innovators’ for real change and not confuse it with different functions.
[…]
Entrepreneurship, the assumption of risk, may not be innovative at all. You assume risk if you open a new auto dealership, but this is not innovative unless you are the first.

Blend in the key points from Tom Kelly’s presentation (or books) and you’ll see how powerful it is to continually aspire for true innovation.

“Tom has observed a number of roles that people can play in an organization to foster innovation and new ideas while offering an effective counter to naysayers. Among these approaches are the Anthropologist, the person who goes into the field to see how customers use and respond to products, to come up with new innovations; the Cross-Pollinator, who mixes and matches ideas, widely disparate people, and technologies to create new ideas that can drive growth; and the Hurdler, who instantly looks for ways to overcome the limits and challenges to any situation.”

Tying this back to the security industry, I’ve been stricken with the illness inducing problem of inspecting massive data sets for important events. A real innovative breakthrough to me would be if someone built an easy-to-use and easy-to-manage system (one that accepts all possible data sources) that handles the crushing volume of disparate events on enterprise networks while truly notifying me of only important events.

To cite a small example, I’m installing several pieces of “security software” for a client considerate of threats: Snort, OSSEC HIDS, Central Log Aggregator, etc. I’ve played with several products which purport to do all I wish, but none of them are succeeding or innovating to me – none feel like they are offering real change or doing something other than offering different functions. Maybe I’ll try the Hurdler role for some time to see what I can do.

How about trying to play a role?

Last week I attended presentations from the Americas Growth Capital Conference. Several of the presentations featured prominent CEO/CTOs in the security products game.

I connected with a view echoed by a panel discussing the difficulty of selling security point solutions.

They raised the issue of their desire to set their customers’ expectations that their products will fail. Catastrophic failures, they said, are rare events. Their products are designed to work most of the time, but they repeated the fact that it is too expensive to build products which escape all bad things.

To limit the embarrassment of a product failure, they work on educating their customers and suggesting to each to invest in a good incident response strategy. This keeps the prices competitive, establishes trust, helps to level expectations, and addresses product failures caused from rare events.

I don’t know how well they convey this when the goal is to sell, but I liked the message.

This is worth the extra effort -- ensuring you have bash 3.x installed (3.2 is the current version). It enables you to set an environment variable to record the time and date of executed commands.

$ export HISTTIMEFORMAT="%h/%m - %H:%M:%S "

$ history | tail -2
1006 Jan/01 - 19:24:02 ls
1007 Jan/01 - 19:24:07 history | tail -2

A potential forensic treasure when needed.

Along with others I’m sure, we’re seeing a nascent phase for security services firms in the business of performing external security assessments. Rather than blast through the battery of scanners the game is changing to include tasks which traditionally fell outside the scope of a basic external assessment.

I like the changes because it follows more of a risk assessment approach. For example, as part of an “external assessment pack” we might obtain the configurations for routers, firewalls, switches, wireless access points, concentrators, review relevant policies, architecture diagrams, and to the best of the client’s ability incorporate their identification of segments or servers which touch sensitive data.

Contrary to acquiring infrastructure visibility, running blind external assessments are losing effectiveness - especially with the growth of the commercial exploit factories (e.g. argeniss, gleg, core security, immunity sec ). Making a statement like “We scanned with Nmap, Nessus, Qualys, Webinspect, checked Secunia, SF, etc., and found no major findings” is losing value - its assurance offered more yesterday than today.

My participation in a few recent incident response/forensic assignments is testament to the weakening value of blind assessments; the cause of each compromise would not have been revealed as a vulnerability by vulnerability scanners.

This is a follow-up from this post: (recap: large financial org w/fully compromised internal systems)

The path of the compromise was pegged to an errant firewall rule put in place during the pilot/testing phase of deploying a security point solution: SSH was open to any. The password lacked sufficient strength to avoid brute force guessing and the system was compromised from the external side within days.

Check this out - this security point solution is a device that sits on a SPAN port and looks for any confidential information on its way out. It records anything that triggered a “hit” in simple flat text files on the local system. It was a gold mine: text files with first name, last name, SS #, mother’s maiden name, account #s, addresses, and phone information.

The catch on all this is this is one of those vendor point security solutions designed to be deployed near the perimeter. It lacks the defense in depth muscle used to protect this type of sensitive data. Even if SSH was closed this org did not protect it as if it contained customer data.

Additional lessons learned:

• Remember to consider anything (security product or not) eavesdropping and recording traffic as potentially holding super sensitive information and take the necessary precautions.
• This org was told by the vendor the data was safe. If you hear that, sit back and think for a few minutes then fire off some questions that will help you understand what they mean by “safe”.

I keep thinking how offering a simple and cheap continuous port scanning service would’ve saved this org big time. Maybe I’ll add that as a feature to nmapTweaker.

I debated whether to write on this, but I think it’s funny and freaky. It’ll be a struggle to make this short and to the point without losing you, but here it goes:

• A former exec of a company we all worked for now evidently works with google in some capacity
• We stay connected with other ex-colleagues from this same company

So the other day I was looking to offload some javascript work and I used google to search for former engineering colleagues from this same company that specialize in UI stuff, knowing most UI developers like to show off their works. I found and surfed around their sites -- half out of boredom and half to check out the skills they were marketing, what’s new, etc.

Come the next evening I get a message to stop viewing their web sites. I thought WTF! Damn, am I compromised? I fire off a few questions for sanity and to see if this is a joke. I get back snippets of info like:

• IP address (it was one of mine -- I wasn’t using a proxy to hide)
• Google searches performed (I couldn’t remember what I did, but the terms sounded familiar)

More bizarre is I get word this exec called the UI developers and said I was “stalking” them based on my googling and surfing patterns. Wow. Is this for real? Whatever the facts end up being I had a couple of thoughts regardless:

1) This sucks; it seems deranged of someone to equate the viewing of public web pages to “stalking”… and especially to take it far enough to actually call and “warn” people. Crazy, right?

2) It’s an excellent reminder that it is not difficult for people that have no business keeping tabs on your online activity from learning more about you. Google records everything (source IP, search criteria, what you select) – mix that with a little advanced IP geolocation (or hit up your friend on the inside) and you’ve got the account holder’s details (e.g. address, names).

For speed sake I usually avoid hiding behind proxies. Then again thinking about insiders at some search company collecting and sharing all my searches is not exactly appealing.

We’re playing w/RoR and building a few web apps. I wrote a quick and easy one that helps you build a command line for nmap, nothing particularly special.

http://nmaptweaker.clearnetsec.com/

I did add some extra content: tuning tips, examples, and a time estimator. Basically you can use it to build a command line, then copy and paste it to wherever you want to run it.

It does not do syntax checking for nmap options which require input yet (e.g. targets, output file name, etc.), but the app does do some error checking for conflicting options.

It’s a little rough, but hey, it might save you a few minutes of time if you’re not an nmap option sage.

Full compromises are crazy – it happens, but nearly every one I touch still sends a little shiver down my spine mixed with a “holy shit” moment.

Case in point: we got a call last week to help out with an incident and to do an investigation of some Linux based security devices. Financial industry, lots of customers.

Their engineers were working feverishly to scope the incident and to learn all the details then slam – they hit a wall. Their firewall logs everything. Their only-way-out-is-via-a-proxy-server logs everything.

And their security point solution in the DMZ? Oops, it only logs locally (it also happens to be a party to the fully compromised club inside this trusted net). And what about their desktops which appear to have offered free remote admin VTYs? Well, this all has a kicker for an answer.

It happened a year ago. Ouch. Talk about sensing a cold sweat and suddenly not trusting anything. By the way, the desktops exhibiting the worst behavior traveled the evolutionary path and have been wiped clean and rebuilt, upgraded, or replaced since the break-in. No substantive logging or backups of mischievous desktops; no way to reconstruct the perpetrator’s methods. Security point solution in DMZ had standard local logging (i.e. full rotation in weeks), therefore it is unlikely things we're looking for are there (still investigation). No network device logging. There are lots of unanswered questions.

The details of this should strengthen your neuron paths connecting logging to your “holy shit” moments. That is to say…

• If you got important data to protect, log everything you reasonably can. All the “security” in this scenario failed and failed to help reconstruct events.
• Do you have one of those semi hands-off security appliances that you presume is working fine because you can connect to the web admin portal? Make it forward logs to somewhere.
• Do you have workstations which touch sensitive data anytime? Yes. Then boost the priority to configure central logging, stop procrastinating, then take comfort you’ll be in better shape than the poor souls at this company fighting to salvage their pride, and maybe their jobs.

Often we run into a scenario where a client wants to improve security by implementing an IDS. Now this is OK but often we find out that they are not exactly "ready". What I mean by this is to effectively deploy an IDS on a network you should first cover your bases with what you already got. Before going IDS wild, are you using your existing infrastructure to get the knowledge you need first?

Of course just throwing out an IDS is a quick hit; it can cover your ass if the regulatory audit kids drop by, but for lots it becomes a security lame duck. Do you know how to really make it a valuable component of your security?

"Yeah we have [insert IDS vendor here] in place off the 6509."

"Sweet, what does your ruleset look like?"

"Well I think we have all rules enabled right now."

"Ok, what networks are you watching?"

"We want to see both ways so we are not restricting the nets."

"Do you know what kind of traffic and how much are you seeing?"

"Well, not really, but we have all the usual suspects: SMTP, HTTP, HTTPS, FTP and even IM."

"Do you know how much activity you’re seeing for each of those?"

"Man, we just threw it out there, its working!"

Yeah, that $20k went far. The point is to get off this copy-cat buy and forget.

First, I am a believer in doing the macro before the micro. You can’t pick whether a rule should be enabled or not until you know what’s going on in your network. Enabling all will protect you is bullshit. In fact, enabling all can make things worse (easier to exhaust resources, get blinded, or rev up the false positive counters). And forget about dealing with IDS evasion techniques.

Take logging, a macro thing. Logging is an enabler. Deducing traffic patterns is a macro enabler. You need to do these things first before considering an IDS. Engineers like to complain how hard it is to watch logs and I can empathize, but it is really not that hard to get some valuable numbers. Which firewall rules are taking hits, which are not? What kind of traffic are you seeing and how much? A couple scripts can get you those numbers.

So before you do the IDS thing, do these things at a minimum:

• Grab your firewall rulesets and the counters for each allow/deny rule. With this you info you can teach your IDS where to look (vitally important both direction so ingress and egress).
• Learn traffic patterns. What protocols are in use, which networks, and in what amounts. High, average, low - all good numbers. Use this to tune the rulesets. Now you can identify something like your top 100 talkers. If some server catapults to #1, check it out.\
• What apps are running. Learn all the things your environment is hosting. When do your backups run, which servers like to talk to which? Again, information vital to getting value from an your infrastructure that will enable you to get even more value from your IDS.

I read this article and was reminded of my own post a couple months back. I generally try to avoid the religion disputes within the community but this is one I'm feeling more inclined to chime in on. I'm going to make an assertion to clarify the debate: basically, the "successful" agile projects seem to pretty much build interfaces. I think a lot of developers may be sensitive to that description but that's pretty much what web programming is.

There are some great concepts in agility, there is an overwhelmingly strong tendency for companies to talk about stuff a lot more than actually doing things and at some point the talking always outweighs the effort of just "getting it done." Most teams need a shot of "agile" somewhere. You know, just a kick in the ass to start doing more and talking about doing less. At the same time, I know of no teams that are building things like compilers or operating systems agilely. I know of no widely used protocols that were constructed with agile. I doubt you'll ever see any agile embedded projects in the near future. Basically anything that is actually hard or where mistakes are costly due to limited debugging that might be available or where the customers are so far detached from the actual technologies that they cannot contribute much to the development beyond the interface ( a lot more people use compilers than have anything to add to conversation when it turns to intermediate representations.) Interface security is much different what is meant by security in a protocol. To be fair, I think the discussion should be scoped around this. Agile seems to be at its best in the interface. Maybe it will eventually move beyond that but I don't see any reason to think that will happen soon.

So agile and security... There are a couple things here. First off, security isn't a requirement from customers until it's too late. This is changing but unless they are coached or somehow lead, I doubt most customers would list it high on the list as a fundamental requirement of a new product. It's both implicit and not and when schedules are tight and there are other features that have been sold already it becomes a lot more like a "nice to have."

There aren't a lot of easy ways to automatically test security, I wish there were. More and more software is getting back to the point where there is an automated test suite that will verify functionality, the best projects are still under 100% coverage. I know of no projects that have test suites that verify security after various refactorings. In fact once you trust that something is secure, it's a huge anchor, it's hard to change it and maintain that level of trust. If something some how is perfectly secure, all you can really do is decrease or maintain that by refactoring it.

I also think that the notion that "working software" is the measured deliverable is faulty when you factor security in. Most software works fairly well and accomplishes the goals of the user to some degree, a lot of it also has known insecurities and vulnerabilities. Security is a benchmark above and beyond "working code." In fact most vulnerabilities go unnoticed by the user. Just today we found a hole in our blog software, Typo, that we haven't known about for quite a while and it was being abused. Everything seemed to work just fine though. When security matters, it's a requirement above and beyond "just working."

So are they contradictory like the article on The Server Side asks? I think so if you take agile in a literal sense. If you look at agile as more of a cultural thing rather than a software engineering methodology (and there are a lot of good reasons to do that) then they start to work together more closely, it's a very agile thing to respond to a vulnerability quickly with code, you have to be "agile" to do that. The other change is that security is becoming less and less of a boutique feature, pure security companies are dying and security is becoming just another expectation when you purchase software.

http://jeremiahgrossman.blogspot.com/2006/10/web-application-security-professionals.html

What caught my eye was #3. 71% responded they never "use a commercial web application vulnerability scanner during security assessments".

That surprised me for a couple of reasons:

1. Clients frequently request the use of a commercial scanner
2. Many of the larger security services firms we deal with (i.e. give us projects) not only have a enterprise type license for one or more commercial tools but also require their use when doing a security assessment on their behalf.

We use lots and lots of open source tools, commercial tools, and some home grown tools to do assessments. We have priorities, flowcharts to guide which tools work under which conditions and ways we like to organize and analyze results.

Is this knowledge IP (Intellectual Property)? What are the pros and cons of being fully transparent? Most, if not all, of the information is already out there -- it is just not neatly packaged.

I tend to want to be more transparent, but I’ve recently noticed several partners asking for explicit details on our processes. They want to see the tool list and learn the “details”.

None of what we do is secret, but at the same time I feel hesitant to divulge everything freely. What is your opinion?

In a recent post we talked about if it is possible to prioritize the deployment of solutions which are widely accepted to reduce risk to a business (without completing a threat assessment). A list you can say to someone "Well, without knowing your details I can say the most frequent threats or highest risks for most companies is from THESE THINGS, but we really should do a threat assessment first".

I googled around and created a short list (I'm sure there are 1000s out there) of data points to help determine the "THESE THINGS" part:

My favorite resource:

From PrivacyRights.org, chronology of data breaches: http://www.privacyrights.org/ar/ChronDataBreaches.htm (probably the best resource because it doesn't restrict by type of threat)

Like above:

From Mailerblog.com, data loss viewer (viewer to attrition's database of data breaches): http://www.mailerblog.com/dataloss/dataloss.php

From PogoWasRight.org, collects information on data breaches: http://www.pogowasright.org/

The recent Visa USA press release: http://biz.yahoo.com/prnews/060915/dcf014.html?.v=3D64

A few network based threat stats:

From DShield.org, top ports for scanning: http://www.dshield.org/topports.php

From Incidents.org, survival time history: http://isc.incidents.org/survivalhistory.php?isc=4fcfc1652464f1b60c02afecb75d332a

From Zone-h.org, attacks archive (defacements): http://www.zone-h.org/component/option,com_attacks/Itemid,44/

Virus specific:

From SecurityStats.com, virus related statistics: http://www.securitystats.com/virusstats.html

From F-Secure, virus statistics: http://www.f-secure.com/virus-info/statistics/

From McAfree, virus activity: http://vil.mcafee.com/mast/viruses_by_continent.asp?continent_k=0&track_by=1&period_id=1

From Symantec, threat explorer: http://www.symantec.com/enterprise/security_response/threatexplorer/threats.jsp

From Postini, StatTrack (including DHA/SPAM stats): http://www.postini.com/stats/

Insider snippets:

From Bruce Schneier, news summary: http://www.schneier.com/blog/archives/2005/12/insider_threat.html

Illicity Cyber Activity in the Banking and Finance Sectors, news summary: http://www.gcn.com/online/vol1_no1/27074-1.html

Reconnex threat stats: http://www.reconnex.net/Threat/

I can probably find a lot more statistics from combing CERT pages, but I stopped: http://www.cert-in.org.in/worldcert.htm

When competing for security assessment projects it is often painful for the customer to distinguish the level of service or effort between proposals. We used to respond to RFPs with the intention of satisfying all the services the customer is soliciting – of course in the end in nearly every case that isn’t what wins the bid.

We came up with a quick flow diagram to illustrate the differences in the level of effort between network-based security assessments. This has helped us tremendously with clients and with keeping the playing field level. It’s not complete or exact by any means, but it works.

We add some verbiage to help customers relate it to real world:

Sample attacker profile:
Basic: Attacker spending minimal effort; downloading free 'hacking' tools and running them with minimal attention
Intermediate: A motivated attacker spending more time and resources with greater attention to detail and actively searching for a weakness
Advanced: A serious attacker with intent to harm or steal information assests

Security assurance profile:
Basic: Minimal; relies on a limited set of tools to discover weaknesses
Intermediate: Good; relies on running many tools with overlapping functions, specialty tools, tuned for bandwidth and latency conditions, and includes manual investigation, validation, and research into findings
Advanced: Excellent; goes beyond Intermediate to prove the existence of vulnerabilities, includes checking non-public domains for the existence of 0-day exploits

We found ourselves in a healthy debate recently over a question posed by a customer that went something like this:

What should be my top 5 things to do now to improve our security?

This was from a young startup that was about to receive their next stage of funding and desired to do “things right”.  I started down the path of listing popular security tools:

Firewalls, IDS, Anti-Virus, Central Logging, Encryption, Patch Management, etc.

I was presuming we would be able to answer this question and have some agreement on which “security” tools would have a higher priority for deployment.  I was wrong.

There are many different ways to answer this question and enough premises to fuel debate that you soon feel like you’re arguing in circles. As a group we haven’t formulated a consensus yet, but I feel there is a logical way to get there, at least for particular tools.

Let’s hypothetically say we had to choose between ‘patch management’ (i.e. keeping up on patches) and anti-virus. 

Now the context I was trying to retain to answer this question was that of a CTO asking you while taking an elevator ride (i.e. need to be quick). 

After some debate I ended up referencing my “threat modeling” docs.  Unfortunately threat modeling must come before choosing anything – you need a threat profile before selecting solutions which mitigate threats.  But that is not going to help us answer this question in 30 seconds. 

Can we use threat modeling to make some general propositions about all companies with respect to choosing a particular security solution over another?

I think that should be possible. 

In threat modeling parlance, the entry point is where an adversary can interface to the system.  To keep this somewhat simple, let’s say we have two small networks with identical systems:  same assets, same trust paradigm, and the same type environment you would typically see in a startup.  So then, which security tools are better (or provide better value or reduce the risk the most, etc.)?

Let’s also presume for this exercise that we’re dealing with what most networks see most frequently – this in the context that most systems on the internet are constantly being scanned for open and vulnerable services by potential attackers.  If we roll up, so to speak, the threats associated with how viruses propagate or how vulnerable services are found and exploited, then I think we can agree that not only is this an accurate statement about reality but also that both anti-virus and patch management solutions focus on mitigating this same threat (or set of threats).  That is to say they both are designed to prevent the masses from these threats and they both fail at exception cases (e.g. 0day).    

If the above holds true, then how can we use the risk equation to evaluate which is a better solution:  patch management or anti-virus?

Risk = Threat x Vulnerability x Cost

In our scenario we have identical networks exposed to the same threats and have the same cost and vulnerability values.  The real question is which solution lowers the threat vulnerability value. 

I would argue that patch management reduces the risk more than anti-virus.  This based on generally that patch management:

• Will reduce the number of attack vectors more than anti-virus
• Is subject to a higher frequency of attacks (i.e. vulnerable service scans and attacks happen more than virus propagation attacks). Also noting the observation that viruses typically proceed post vulnerability disclosure.

If the above assumptions are correct then we can say the company which successfully deployed a patch management solution has greater security strength.  More so that most startups of the type that posed this question to us would be better served security wise to first deploy patch management. 

Now the question is can we make some generalized statements that apply for most companies and create a list prioritizing security tools to deploy (within reason and allowing for variance). 

Thoughts?

What is systemtap?

Systemtap is a tool built upon the kprobes framework for probing, debugging and analyzing the linux kernel at runtime. IBM contributed kprobes to linux and had it included during the 2.5 development tree in 2002. It's a supported and standard portion of the Linux kernel. Systemtap builds upon kprobes and creates a fairly easy to use tool that let's you probe just about anything and everything in the linux kernel with fairly simple commands, you can almost effortlessly modify any variable you wish on a running kernel. All with minimal performance impact. Essentially it's the linux version of dtrace the differences being that systemtap is probably more flexible in kernel space and dtrace currently does a lot more for probing userspace applications.

So how do I get this working?

It's fairly easy and straight forward to install. There are RPMs for Fedora, Redhat Enterprise and Suse based linux distributions. There is a slight trick, at least with Fedora Core 5, you need to have the kernel debug RPM also installed. Basically, it's just a set of object code files with debug symbols. I couldn't find it in the latest Fedora Core 5 and upon closer look the kernel RPM didn't have the debug package enabled by default. I placed the following in my kernel-2.6.spec from Fedora Core 5 and was able to build the kernel debug symbol package.

%define _enable_debug_packages 1

With that added to your kernel spec file you can use rpmbuild and rebuild it and it will emit the debug package.

What can you do now?

Now you can start probing stuff. The systemtap site has some fairly simple demos, one is to print the top 20 most common syscalls every 5 seconds.

#!/usr/bin/env stap
#
# This script continuously lists the top 20 systemcalls on the system
#

global syscalls

function print_top () {
        cnt=0
        log ("SYSCALL\t\t\t\tCOUNT")
        foreach ([name] in syscalls-) {
                printf("%-20s\t\t%5d\n",name, syscalls[name])
                if (cnt++ == 20)
                        break
        }
        printf("--------------------------------------\n")
        delete syscalls
}

probe kernel.function("sys_*") {
        syscalls[probefunc()]++
}

# print top syscalls every 5 seconds
probe timer.ms(5000) {
        print_top ()
}

This little probe will sleep for 5 seconds counting each system call as it happens and then dumps the list out and starts counting again. Here is a sample output:

SYSCALL                         COUNT
sys_gettimeofday                  771
sys_futex                         637
sys_clock_gettime                 214
sys_rt_sigprocmask                141
sys_select                        136
sys_poll                          106
sys_newstat                        84
sys_ioctl                          62
sys_read                           48
sys_write                          47
sys_fcntl                          47
sys_rt_sigaction                   38
sys_time                           30
sys_getppid                        26
sys_setitimer                      16
sys_recvfrom                       13
sys_rt_sigreturn                   13
sys_nanosleep                      12
sys_close                          11
sys_open                           10
sys_newfstat                        6

Try altering the load on your system and seeing how that affects things. That's kind of interesting but not terribly useful. Try this version:

#! /usr/bin/env stap
#
# This script continuously lists the top 20 systemcalls on the system
#

global syscalls

function print_top () {
        cnt=0
        log ("SYSCALL\t\t\t\tCOUNT")
        foreach ([name] in syscalls-) {
                printf("%-20s\t\t%5d\n",name, syscalls[name])
                if (cnt++ == 20)
                        break
        }
        printf("--------------------------------------\n")
        delete syscalls
}

probe kernel.function("sys_*") {
        if (target() == tid())
                syscalls[probefunc()]++
}

# print top syscalls every 5 seconds
probe timer.ms(5000) {
        print_top ()


}

I added a line to the probe which checks to see if the target is a value that system let's you pass in. Now you can add a -x argument and the pid of a process and it will perform the same probes only against that particular process. I have a postgresql database running and ps tells me that pid 16679 is the writer thread.

...
postgres 16679  0.0  4.4 199916 91000 ?        S    Jul21   0:00 postgres: writer process
...

Let's look at it.

SYSCALL                         COUNT
sys_getppid                        25
sys_time                           25
sys_select                         25

over and over and over, since the database isn't doing anything this makes sense, it's just waiting for something to tell it what to do. You can attach strace to that process (strace -p 16679) and see that it is correct. Let's distburb the database and make it do some I/O and see how it behaves. I'll insert a record into a table.

SYSCALL                         COUNT
sys_getppid                        25
sys_time                           25
sys_select                         25
sys_lseek                           4
sys_write                           4

That makes some sense, one record insereted resulted in 4 seeks and 4 writes. Maybe that's a write to an index, a couple writes to b+tree nodes and then an actual record being written, or maybe it is writing to a journal file. I don't know, but it sounds kind of reasonable and I haven't looked at the postgresql code yet. In fact we haven't looked at any source code yet or really compiled anything. This is kind of a cheap example since strace and the ptrace API can provide all of this information (although there is considerable impact on system performance with them) so here is something a bit harder to do.

#! /usr/bin/env stap
global comps
global compsizes
probe kernel.function("memcmp")
{
                comps++
                compsizes = compsizes + $count
}

probe timer.ms(5000)
{
        printf("memcmp call count      bytes compared\n")
                    printf("%d                    %d\n", comps, compsizes)
        comps = 0
        compsizes = 0
}

This probe monitors the kernel's memcmp function which compares two buffers of memory. $count is an argument to that function which contains the number of bytes to compare.

memcmp call count      bytes compared
17151                    114214
memcmp call count      bytes compared
5455                    34990
memcmp call count      bytes compared
17165                    114269
memcmp call count      bytes compared
5398                    34817

As you can see memcmp is called a lot and on average to compare about 6 bytes worth of data. I wonder what it sort of oscillates? I'll have to look in to that.

Here is another simple one, __kmalloc is used to allocate memory in the kernel.

#! /usr/bin/env stap
global allocs
global allocsizes


probe kernel.function("__kmalloc")
{
        allocs ++
        allocsizes = allocsizes + $size
}


probe timer.ms(5000)
{
        printf("malloc count      bytes malloced   \n")
        printf("%d                     %d              \n", allocs, allocsizes)
        allocs = 0
        allocsizes = 0
}

which produces something looking like this:

malloc count      bytes malloced 
16                     30128              
malloc count      bytes malloced
36                     69312              
malloc count      bytes malloced    
76                     64469             
malloc count      bytes malloced    
28                     56651              

So every 5 seconds this particular system is doing about 40 mallocs and it's averaging to about 5K per malloc if my napkin math is correct.

That's just a taste, next week I'll do a more interesting example.

We have a potential pen. test  project coming up with crazy numbers:  260,000+ range of publicly addressable IPs (spread over several non-contiguous blocks).  Our previous experience with scanning large blocks aggressively via the internet has made us, well, super sensitive.  It’s freakin’ hard. 

260,000 x 65535 TCP ports x 2 (number of port query attempts) = 34,078,200,000 TCP SYN packets (nmap, default).

That is ~4x larger compared to our last "large" effort (~70,000 IPs).

No info yet on the bandwidth available per block, latency conditions, or other factors, but let’s run some numbers to see how this would work.

From recall, the Sun v40z’s were averaging ~3500pps (packets per second) outbound per server.  I didn’t use packets per second to estimate previous effort, I created a quick excel chart similar to below to get a ballpark number (updated using 260,000 for the total number of IP addresses to scan):

approx. number of IP addresses to scan	260,000
number of TCP ports to query per IP address	65,535
total # of TCP ports to query	17,039,100,000
time-out setting per port query	1.25	seconds (allow 1.25 seconds for query response)
number of query attempts per port	2	which implies we need 34,078,200,000 SYN requests
total seconds per port query	2.50	seconds
number of parallel port queries	2	(we set it to 100, but empirical evidence shows this to be ~2)
total # of hours to complete single IP port scan	22.76
total number of seconds to perform scan (if sequential)	21,298,875,000
total number of hours to perform scan (if sequential)	5,916,354
number of scans in parallel	1,600	(2 servers, each running 8 unique nmap processes with min_hostgroup set to 100)
number of hours to complete scan	3,698
number of days to complete scan	154

Now I realize pps is an important factor.  Remember these are still pretty fast servers: quad dual core opterons with 16 GB RAM. Doing a sanity check with our previous results of obtaining ~3500 pps per server goes as follows:  

total # of SYN requests	34,078,200,000
average outbound packets per second / per server	3500
number of dedicated servers	2
total number of seconds to complete scan	4,868,314.29
number of hours to complete scan	1,352.31
number of days to complete scan	56

So the numbers don't match. The first chart says it'll take ~154 days, the second says ~56 days. In the first chart I set "number of scans in parallel" to 1600. I got that by having 8 unique nmap processes each running with min_hostgroup of 100 (800 hosts being scanned simultaneously per server; we have two dedicated scanners, so the total is 1600 hosts). The number that is probably way off is the "number of parallel port queries" in the first chart. Although we had set the value to 100 (meaning scan 100 ports simultaneously on each host) it often seemed like we were only getting 2 or so in parallel (observed from watching tcpdump output). That was probably after running into the memory issues I reported with nmap (which Fyodor subsequently reported to fix).

I just ran a few quick checks on a Sun Fire x2100 we have (dual core opteron 175 / 4 GB of RAM) using a newer nmap version (4.11). Firing off a bunch of nmap scans on one server with parameters similar to below resulted in a sustained 9000+ pps (~4.5 Mbits/s).

/usr/local/bin/nmap -vv -sS -P0 -p 1-65535 -n --min_hostgroup 100 --max_rtt_timeout 1250 --min_parallelism 100 <a_/24_block>

I didn't let the test run for very long to see if issues arise like before. If we could sustain 9000 pps per server and be allowed to push ~9Mbits/s, then the overall time is greatly reduced. It drops to ~22 days. I really doubt we'll be able to or allowed to push 9Mbits/s, but at least we now have ballpark figures to play with.

Who else has had this kind of crazy port scanning fun?

Related entry: maximizing nmap scans for accuracy

 

In part 2 of my VA auditing experience I told you all about our "training" for the VA assessment. I am going to finish this out with my thoughts on the first site experience. If you missed it here is part 1. With all the things that had gone on with this project I was very interested in how the actual audit was going to go for each site. Before I could think long on it I was off to the wonderful state of Maine in February.

Now I live in Colorado and most people's preconception of Colorado in the winter is exactly what Maine was... Cold, snowy, and dark. For those of you that don't know, Denver Colorado has a very mild winter and snow barely stays a week on the ground. In the mountains is a different story but Denver is on the plains not the mountains.

So back in Virginia we were told that we needed to car pool with the other auditors and that each auditor was responsible for ensuring the whole team got to the site. This was interesting to say the least as the audit teams were thrown together maybe 2 days before we actually flew out. Each trip I went on had a team with different people. This fact was great for meeting new people but horrible for car pooling as the one person who had the car was expected to ferry us around! Now the issue that greeted me first was that I got to Portland, Maine at about 11:00 PM EST and had to get to Augusta which is about 1 1/2 hours away. Trying to get ahold of the guy with the car did not happen as it went to VM suprisingly enough. Suffice to say I had to take a taxi to Augusta which costs about 170 dollars, footed by the tax payers of course. For people that don't know Maine, Portland is in the south and Augusta, the capital, is in the lower center of the state so a taxi ride was costly.

The second issue was that none of the audit staff could get ahold of each other. In fact I didn't even get to the facility till later on Monday cause we all were staying at different hotels. Hotels, flights, and rental cars were chosen by the coordinators not the auditors so this was not negotiable. Anyhow we were scheduled to be at the facility for 4 days and leaving the 5th day so I was already thinking of how much fun I was going to have.

Onto Monday we go! After I get to the facility with my chauffeur. I finally find out how many computers we are testing. Lets see the audit team had 3 "windows testers" including me so that means we can get pretty good coverage in 4 days right? Well we had to test a grand total of 26 computers and all the mobile nursing stations for a grand total of 30. Remember the checklist, the one that takes about 20 minutes per computer max? 30 / 3 = 10 computers over 4 days. So doing some more math we can estimate about a 4 hour work day including lunch. Now this facility was pretty big. So big that I would have easily gotten lost without my VA companion. Off I went to verify the VA is secure with my clipboard! Suffice to say that my VA companion was pleased to only waste 4 hours running MBSA and Dumpsec.

At this point I am sure a few of you are thinking that it was easier for me to test this minuscule amount of computers and then just chill till it is time to leave but it wasn't. We were not allowed to have cell phones on in the building because of possible interference with medical equipment, we were not allowed to go onto the VA network with our laptops, which makes sense, and we were in the middle of nowhere. Luckily we got to go home on the 3rd day meaning that we had only spent 4 days total in snowy Maine.

A few thoughts on my whole VA auditing experience. First, I did actually like meeting the other auditors and the technical VA personnel. They were great and made the whole project actually move forward. I also got to go to places I would never have gone to if not on business. What a waste of money the whole endeavor was. As Bruce Schneier likes to always say, this definitely had the perception of being a proactive security measure but that is all it was, a perception. I think that there were some serious loopholes somewhere that allows this sort of thing to go on. Like I said earlier, if this kind of project happened elsewhere everyone would be fired, unless of course they are interested in the perception. We ended up doing 10 facilities before we just could not take it anymore. We were not alone in that feeling as I think every team I was on had people that were new who had replaced someone that went to the "training".