Déjà vu with big time port scanning

tate@ClearNetSec.com (Tate Hansen) — Tue, 22 Aug 2006 23:13:00 -0600

We have a potential pen. test project coming up with crazy numbers: 260,000+ range of publicly addressable IPs (spread over several non-contiguous blocks). Our previous experience with scanning large blocks aggressively via the internet has made us, well, super sensitive. It’s freakin’ hard.

260,000 x 65535 TCP ports x 2 (number of port query attempts) = 34,078,200,000 TCP SYN packets (nmap, default).

That is ~4x larger compared to our last "large" effort (~70,000 IPs).

No info yet on the bandwidth available per block, latency conditions, or other factors, but let’s run some numbers to see how this would work.

From recall, the Sun v40z’s were averaging ~3500pps (packets per second) outbound per server. I didn’t use packets per second to estimate previous effort, I created a quick excel chart similar to below to get a ballpark number (updated using 260,000 for the total number of IP addresses to scan):

approx. number of IP addresses to scan	260,000
number of TCP ports to query per IP address	65,535
total # of TCP ports to query	17,039,100,000

time-out setting per port query	1.25	seconds (allow 1.25 seconds for query response)
number of query attempts per port	2	which implies we need 34,078,200,000 SYN requests
total seconds per port query	2.50	seconds

number of parallel port queries	2	(we set it to 100, but empirical evidence shows this to be ~2)
total # of hours to complete single IP port scan	22.76

total number of seconds to perform scan (if sequential)	21,298,875,000
total number of hours to perform scan (if sequential)	5,916,354

number of scans in parallel	1,600	(2 servers, each running 8 unique nmap processes with min_hostgroup set to 100)
number of hours to complete scan	3,698
number of days to complete scan	154

Now I realize pps is an important factor. Remember these are still pretty fast servers: quad dual core opterons with 16 GB RAM. Doing a sanity check with our previous results of obtaining ~3500 pps per server goes as follows:

total # of SYN requests

34,078,200,000

average outbound packets per second / per server

3500

number of dedicated servers

total number of seconds to complete scan

4,868,314.29

number of hours to complete scan

1,352.31

number of days to complete scan

So the numbers don't match. The first chart says it'll take ~154 days, the second says ~56 days. In the first chart I set "number of scans in parallel" to 1600. I got that by having 8 unique nmap processes each running with min_hostgroup of 100 (800 hosts being scanned simultaneously per server; we have two dedicated scanners, so the total is 1600 hosts). The number that is probably way off is the "number of parallel port queries" in the first chart. Although we had set the value to 100 (meaning scan 100 ports simultaneously on each host) it often seemed like we were only getting 2 or so in parallel (observed from watching tcpdump output). That was probably after running into the memory issues I reported with nmap (which Fyodor subsequently reported to fix).

I just ran a few quick checks on a Sun Fire x2100 we have (dual core opteron 175 / 4 GB of RAM) using a newer nmap version (4.11). Firing off a bunch of nmap scans on one server with parameters similar to below resulted in a sustained 9000+ pps (~4.5 Mbits/s).

/usr/local/bin/nmap -vv -sS -P0 -p 1-65535 -n --min_hostgroup 100 --max_rtt_timeout 1250 --min_parallelism 100 <a_/24_block>

I didn't let the test run for very long to see if issues arise like before. If we could sustain 9000 pps per server and be allowed to push ~9Mbits/s, then the overall time is greatly reduced. It drops to ~22 days. I really doubt we'll be able to or allowed to push 9Mbits/s, but at least we now have ballpark figures to play with.

Who else has had this kind of crazy port scanning fun?

Related entry: maximizing nmap scans for accuracy

maximizing nmap scans for accuracy

tate@ClearNetSec.com (Tate Hansen) — Wed, 03 May 2006 01:05:00 -0600

There have been some recent newsgroup postings about consistency problems with nmap results (especially when scanning over the internet). In my experience, if you want the best chance of obtaining accurate nmap scan results, you need to:

check the latency to your targets (ping some device on the target net block for awhile, what is the average rtt value?)
- use the average to configure the --max_rtt_timeout nmap option (e.g. if your ping time is 300ms, you obviously want to set this option to more than 300ms to prevent nmap for missing an open port if/when a probe takes longer than 300ms. Maybe set it to 500ms or even higher. Speed suffers, but we're chasing accuracy)
if you can, check the available bandwidth limit for you and the target net (e.g. T3:45mbps, T1:1.544mbps, DSL, etc.)
- use this to configure the --min-hostgroup and --max-hostgroup nmap options and the --min-parallelism and --max-parallelism options (i.e. the more bandwidth each side has available, the more hosts and ports you can scan in parallel). This can significantly reduce overall scan time while keeping accuracy as the highest priority. I typically use IPTraf to watch pps (packets per second) to guage my bandwidth usage. If my options are too aggressive (i.e. consuming too much bandwidth), I'll kill the current scan and start over with different tuning values. The objective here is to set --min-hostgroup and min-parallelism to something that both sides can handle without sacrificing accuracy. If bandwidth permits, I like to set min_parallelism to 100 (number of parallel SYN requests or whatever scan type you're doing). I've gone up to 300 for --min-hostgroup (had lots of bandwidth and plenty of server horsepower). You don't want to cause your scan server, the target, or any device the probes traverse to drop packets, so the more information you have about all the components involved the better you'll be able to tune these values.
if you can, verify there are no 'smart' devices impeding scans (e.g. IPS, rate limiting device, etc.)
run multiple nmap scans of the same target, preferably at different times of the day and different days of the week (e.g. nightly backups can saturate certain hosts/links which causes nmap to fail to discover open ports). I like to run multiple services scans (~1600 ports) and do one or two 65k scans depending on circumstances. If time permits, the confidence boost you get knowing you've maximized accuracy by running repeated scans is satisfying. It also shows your assiduity; you can tell your customer you fired off 5 scans at different times to obtain your final list of exposed services. If you missed something after repeated scans (assuming you used safe tuning values), something is likely wrong.
set --host_timeout to something you are comfortable with.
- if you determine a scan may take 20 hours to perform a 65k port scan of the target device, then set this option accordingly. Note: you can use 's', 'm', or 'h' to specify seconds, minutes, or hours.
set --max-scan-delay to something low if nothing is impeding scans
- I typically set this option to 0 because I'm confident in my other tuning values (no delay between probes, just fire them off)

While tuning nmap options can do lots of good things it is still important to verify your scan server can keep up. We've knocked down 8-way Opteron systems w/16GB of RAM, so optimal nmap options may not be realistic for your hardware.

ClearNet Security: Tag performance

Déjà vu with big time port scanning

maximizing nmap scans for accuracy