ClearNet Security

We recently performed a relatively large TCP port scan for a client; a full 65k SYN scan of ~70,000 IP addresses.  Nmap is a great port scanner and was our first choice.  We had two new and beautiful Sun quad v40z dual core opteron servers (16GB of RAM each) dedicated to the task.  We were under a restrictive change control window and time was the limiting factor.  We broke the scans down like this:

• Executed 8 unique nmap instances on each system (one for each ‘virtual’ processor)
• Divided the scans on /24 blocks (the optimal breakdown would’ve been on a 100 boundary, but we ran with this)
• Set min_hostgroup to 100 (minimum number of devices to scan in parallel)
• Set min_parallelism to 100 (minimum number or ports to scan in parallel)
• Set max_rtt_timeout to 1250 (wait a maximum of 1.25 seconds to receive a reply from a port query)
• Other command line options used (-vv: verbose, -sS: SYN Scan, -P0: no ICMP, -p: port range)

All together, a single nmap statement looked like the following:
/usr/local/bin/nmap -vv -sS -P0 -p 1-65535 -n --min_hostgroup 100 --max_rtt_timeout 1250 --min_parallelism 100 <a_/24_block>

We paid close attention to the number of outbound pps (packets per second) using iptraf for a couple reasons:  To watch our bandwidth utilizations to avoid ISP overage charges and to gain a rough baseline so we could detect a problem.

If I remember right, the outbound pps initially was between 2k and 4k per server.  Things were rocking and it looked like we would sail through the port scans.  Alas, when doing a quick check after ~30 hours of scanning, we noticed the pps had slowed to ~550 per server.  We deduced nmap had some memory management issues when used the way we crafted.  Each nmap instance was consuming ~1.2GBytes of RAM, appeared to be increasing, and the CPU idle time for all processors was a continuous 0.  This caught us off guard somewhat because we had successfully performed an nmap TCP services scan (~1668 ports) of the 70,000 IP addresses in less than 40 hours.  This was all on SuSE Linux Enterprise Server 9 (x86_64) with nmap version 3.93.  We knew now this was not going to be easy.  The number of SYN requests to do this is big, roughly:  65,535 hosts x 65535 ports x 2 (number of port query attempts) = 8,589,672,450 outbound SYN packets.  If we could sustain ~3,500 outbound pps on each server, then we could finish in approximately 15 days (within the change control window).  At 1,100 total pps, it is ~90 days, ouch! 

In the mix of this engagement, there was a timely posting on the network security pen-test newsgroup asking about scanning a large network with nmap in which I posted a reply.  I subsequently received a response from Fyodor (the author of nmap) which not only confirmed our experiences but also contained a link to an updated version to better enable nmap to handle this scenario (thanks Fyodor!!).  I haven’t had a chance to use this updated version yet, but I’m excited to check it out.  Also I am going to explore some home grown scanners we used while building a vulnerability scan engine and play around with the scanrand and unicorn scanners.  This ended up being a great experience and a wake-up call to verify the tools we depend on work at the scale we need before accepting the next job.

A while back, I knew a couple friends in a somewhat compromised situation. They had been breaking the rules and running LC5 at work and were running the risk of getting caught, being sued and maybe worse. Due to some scheduling mishaps, they found themselves with a program running on a machine they no longer had access to clean up. I won’t get into the specifics just: don’t do that. It’s unethical and once you’re upset enough to do something like that at work you should just quit because you’re not going to learn anything that will make you happy. Worse, rather than control your own destiny you risk getting terminated and like I said before, maybe a lot worse and there is no way having that on your record or in your past makes you a more desirable IT professional to employ. . 

LC5 is Symantec's password cracker, originally developed by l0pht which was acquired by @stake and then eventually @stake was bought by Symantec.  LC5 is frightening, it can sniff LM hashes off the wire and crack them at frightening rate. If you don't have the 10 to 20 hours to wait for the brute force on an old computer, you can pay to use someone's rainbow tables or create your own and "recover passwords" much much more quickly. There is nothing terribly complex or difficult to understand about how LC5 works, it has a set of tools for capturing windows password hashes either from the local machine or as they cross the network and then it has a set of tools for finding a password that creates the same hash either by brute force or by looking it up in a database. There are free tools that are similar for cracking password hashes, such as ophcrack.  

A more captivating problem is how do you actually capture the password hashes in the first place and do so undetected. Now this is all hypothetical but how difficult is it to capture a stream of data which may contain password hashes and then replay it back through LC5?

Screenshots of LC5: