$70,000 worth of new opteron servers for nmap scanning and they suck?

tate@ClearNetSec.com (Tate Hansen) — Tue, 27 Dec 2005 22:51:00 -0700

We recently performed a relatively large TCP port scan for a client; a full 65k SYN scan of ~70,000 IP addresses. Nmap is a great port scanner and was our first choice. We had two new and beautiful Sun quad v40z dual core opteron servers (16GB of RAM each) dedicated to the task. We were under a restrictive change control window and time was the limiting factor. We broke the scans down like this:

Executed 8 unique nmap instances on each system (one for each ‘virtual’ processor)
Divided the scans on /24 blocks (the optimal breakdown would’ve been on a 100 boundary, but we ran with this)
Set min_hostgroup to 100 (minimum number of devices to scan in parallel)
Set min_parallelism to 100 (minimum number or ports to scan in parallel)
Set max_rtt_timeout to 1250 (wait a maximum of 1.25 seconds to receive a reply from a port query)
Other command line options used (-vv: verbose, -sS: SYN Scan, -P0: no ICMP, -p: port range)

All together, a single nmap statement looked like the following:
/usr/local/bin/nmap -vv -sS -P0 -p 1-65535 -n --min_hostgroup 100 --max_rtt_timeout 1250 --min_parallelism 100 <a_/24_block>

We paid close attention to the number of outbound pps (packets per second) using iptraf for a couple reasons: To watch our bandwidth utilizations to avoid ISP overage charges and to gain a rough baseline so we could detect a problem.

If I remember right, the outbound pps initially was between 2k and 4k per server. Things were rocking and it looked like we would sail through the port scans. Alas, when doing a quick check after ~30 hours of scanning, we noticed the pps had slowed to ~550 per server. We deduced nmap had some memory management issues when used the way we crafted. Each nmap instance was consuming ~1.2GBytes of RAM, appeared to be increasing, and the CPU idle time for all processors was a continuous 0. This caught us off guard somewhat because we had successfully performed an nmap TCP services scan (~1668 ports) of the 70,000 IP addresses in less than 40 hours. This was all on SuSE Linux Enterprise Server 9 (x86_64) with nmap version 3.93. We knew now this was not going to be easy. The number of SYN requests to do this is big, roughly: 65,535 hosts x 65535 ports x 2 (number of port query attempts) = 8,589,672,450 outbound SYN packets. If we could sustain ~3,500 outbound pps on each server, then we could finish in approximately 15 days (within the change control window). At 1,100 total pps, it is ~90 days, ouch!

In the mix of this engagement, there was a timely posting on the network security pen-test newsgroup asking about scanning a large network with nmap in which I posted a reply. I subsequently received a response from Fyodor (the author of nmap) which not only confirmed our experiences but also contained a link to an updated version to better enable nmap to handle this scenario (thanks Fyodor!!). I haven’t had a chance to use this updated version yet, but I’m excited to check it out. Also I am going to explore some home grown scanners we used while building a vulnerability scan engine and play around with the scanrand and unicorn scanners. This ended up being a great experience and a wake-up call to verify the tools we depend on work at the scale we need before accepting the next job.

ClearNet Security: Tag port

$70,000 worth of new opteron servers for nmap scanning and they suck?