Déjà vu with big time port scanning

tate@ClearNetSec.com (Tate Hansen) — Tue, 22 Aug 2006 23:13:00 -0600

We have a potential pen. test project coming up with crazy numbers: 260,000+ range of publicly addressable IPs (spread over several non-contiguous blocks). Our previous experience with scanning large blocks aggressively via the internet has made us, well, super sensitive. It’s freakin’ hard.

260,000 x 65535 TCP ports x 2 (number of port query attempts) = 34,078,200,000 TCP SYN packets (nmap, default).

That is ~4x larger compared to our last "large" effort (~70,000 IPs).

No info yet on the bandwidth available per block, latency conditions, or other factors, but let’s run some numbers to see how this would work.

From recall, the Sun v40z’s were averaging ~3500pps (packets per second) outbound per server. I didn’t use packets per second to estimate previous effort, I created a quick excel chart similar to below to get a ballpark number (updated using 260,000 for the total number of IP addresses to scan):

approx. number of IP addresses to scan	260,000
number of TCP ports to query per IP address	65,535
total # of TCP ports to query	17,039,100,000

time-out setting per port query	1.25	seconds (allow 1.25 seconds for query response)
number of query attempts per port	2	which implies we need 34,078,200,000 SYN requests
total seconds per port query	2.50	seconds

number of parallel port queries	2	(we set it to 100, but empirical evidence shows this to be ~2)
total # of hours to complete single IP port scan	22.76

total number of seconds to perform scan (if sequential)	21,298,875,000
total number of hours to perform scan (if sequential)	5,916,354

number of scans in parallel	1,600	(2 servers, each running 8 unique nmap processes with min_hostgroup set to 100)
number of hours to complete scan	3,698
number of days to complete scan	154

Now I realize pps is an important factor. Remember these are still pretty fast servers: quad dual core opterons with 16 GB RAM. Doing a sanity check with our previous results of obtaining ~3500 pps per server goes as follows:

total # of SYN requests

34,078,200,000

average outbound packets per second / per server

3500

number of dedicated servers

total number of seconds to complete scan

4,868,314.29

number of hours to complete scan

1,352.31

number of days to complete scan

So the numbers don't match. The first chart says it'll take ~154 days, the second says ~56 days. In the first chart I set "number of scans in parallel" to 1600. I got that by having 8 unique nmap processes each running with min_hostgroup of 100 (800 hosts being scanned simultaneously per server; we have two dedicated scanners, so the total is 1600 hosts). The number that is probably way off is the "number of parallel port queries" in the first chart. Although we had set the value to 100 (meaning scan 100 ports simultaneously on each host) it often seemed like we were only getting 2 or so in parallel (observed from watching tcpdump output). That was probably after running into the memory issues I reported with nmap (which Fyodor subsequently reported to fix).

I just ran a few quick checks on a Sun Fire x2100 we have (dual core opteron 175 / 4 GB of RAM) using a newer nmap version (4.11). Firing off a bunch of nmap scans on one server with parameters similar to below resulted in a sustained 9000+ pps (~4.5 Mbits/s).

/usr/local/bin/nmap -vv -sS -P0 -p 1-65535 -n --min_hostgroup 100 --max_rtt_timeout 1250 --min_parallelism 100 <a_/24_block>

I didn't let the test run for very long to see if issues arise like before. If we could sustain 9000 pps per server and be allowed to push ~9Mbits/s, then the overall time is greatly reduced. It drops to ~22 days. I really doubt we'll be able to or allowed to push 9Mbits/s, but at least we now have ballpark figures to play with.

Who else has had this kind of crazy port scanning fun?

Related entry: maximizing nmap scans for accuracy

"Déjà vu with big time port scanning" by Søren Maigaard

Thu, 21 Sep 2006 05:14:08 -0600

Even on internal networks, a port scan of 25,000 hosts on a Class A network takes a very long time.

We use nMap to do this (and sometimes FoundScan), but we also hit the limits of the L3 devices around the company. Also VPN devices connecting remote offices, firewalls etc. have limitations.
What I really want is a “tune tool”. Send probe packets around the network, identify L3 devices (and know their default limitations) and configure the scan accordingly.
How do other big companies do this?

s0ren

"Déjà vu with big time port scanning" by Tate Hansen

Wed, 23 Aug 2006 15:33:57 -0600

I’ve played with scanrand and the unicorn scanner a couple times awhile back, but I didn’t use them enough to gain confidence they’ll work well. I’m afraid of missing open services. But you’re right, I think these tools are better (or better designed) for the job. Time to play more.

I’ve looked several times for a site that has a big chart of all the forwarding rates for the most common networking devices, but I’ve never found anything comprehensive. It would also be nice to know how fast unacknowledged TCP SYN packets linger in connection tables (by default at least, I think most can be tuned).

"Déjà vu with big time port scanning" by Chris Byrd

Wed, 23 Aug 2006 08:04:15 -0600

For large scale TCP scanning, you might want to look at scanrand2, which is part of Dan Kaminsky’s Paketto Keiretsu toolkit http://www.doxpara.com/read.php/code/paketto.html

It uses forged Stateless TCP SYN requests and a technique Dan calles ‘inverse syn cookies’ to bypass kernel connection table limitations. Keep a mind on watching limitations of any L3 devices between the scanner and target network. I’ve seen L3 connection tables filled up, DoSing these devices even with limited nmap scans.

ClearNet Security: Déjà vu with big time port scanning

Déjà vu with big time port scanning

"Déjà vu with big time port scanning" by Søren Maigaard

"Déjà vu with big time port scanning" by Tate Hansen

"Déjà vu with big time port scanning" by Chris Byrd