ClearNet Security: Tag nmap

Follow-up on using unicornscan for a big scan (400,000+ public IPs)

tate@ClearNetSec.com (Tate Hansen) — Thu, 27 Dec 2007 12:36:00 -0700

I’m happy to report our growing experience using unicornscan for large discovery sweeps is a positive one. Our confidence in using this tool has increased and it is now our preferred weapon of choice for scanning large IP swaths.

To recap: We performed a sweep of 400,000+ public IPs across multiple continents by configuring the scans to do a full TCP port scan of each IP, sustained ~55 Mbits/s using between 3 and 5 systems, and completed it in a matter of days.

This is pretty good considering by sending two SYN probes per port it meant sending ~52.5 billion packets and producing some 3 Terabytes of data.

Nmap is often our preferred tool, and we used it to spot check our results with unicornscan, but from now on it will come down to the details of the gig to make the choice.

Tech note: We avoided problems with table overflows and other like issues by placing the systems directly on the internet and with iptables turned off.

Trying out unicornscan

tate@ClearNetSec.com (Tate Hansen) — Sun, 14 Oct 2007 22:10:00 -0600

We’ve hit a new high. We’ve soaked ourselves in a bandwidth bath on behalf of a client whom would like us to discover active services across a range of six public /16 blocks plus some scattered /17s, /24s, etc. The range is close to a total of 400,000 IPs.

We started out with five dual Xeon systems running 20 to 40 instances of nmap, each tuned, and each instance targeting 64 IPs. This client wants the job completed in weeks, so we decided it was a good time to get more experience with unicornscan.

By luck, we tapped into a Danish provider that is allowing us to push 55Mbits/s. I have no idea how much that amount of bandwidth would normally cost, especially if sustaining it 24x7 for a few weeks, but I’m guessing it is way over $10,000. Our client would allow us to go up to 100Mbits/s, alas, our luck doesn’t go that far.

Anyway, we now have faster dual-core systems each pushing ~25 Mbits/s via unicornscan like so:

sudo nohup /usr/local/bin/unicornscan -mT -p –r25000 -vv xxx.zz.0.0/16:a -w unicorn.output.for.xxx.zz..0.0.fullTCP > unicorn.output.fullTCP &

We have lots of results from nmap; so far unicornscan is matching the nmap results. Having the ability to specify packets per second with unicornscan is super nice.

We’ll create a follow up post on how all our scanning worked out on this gig when we’re finished (sometime in late November).

nmap junkie?

tate@ClearNetSec.com (Tate Hansen) — Thu, 23 Nov 2006 23:02:00 -0700

We’re playing w/RoR and building a few web apps. I wrote a quick and easy one that helps you build a command line for nmap, nothing particularly special.

http://nmaptweaker.clearnetsec.com/

I did add some extra content: tuning tips, examples, and a time estimator. Basically you can use it to build a command line, then copy and paste it to wherever you want to run it.

It does not do syntax checking for nmap options which require input yet (e.g. targets, output file name, etc.), but the app does do some error checking for conflicting options.

It’s a little rough, but hey, it might save you a few minutes of time if you’re not an nmap option sage.

Déjà vu with big time port scanning

tate@ClearNetSec.com (Tate Hansen) — Tue, 22 Aug 2006 23:13:00 -0600

We have a potential pen. test project coming up with crazy numbers: 260,000+ range of publicly addressable IPs (spread over several non-contiguous blocks). Our previous experience with scanning large blocks aggressively via the internet has made us, well, super sensitive. It’s freakin’ hard.

260,000 x 65535 TCP ports x 2 (number of port query attempts) = 34,078,200,000 TCP SYN packets (nmap, default).

That is ~4x larger compared to our last "large" effort (~70,000 IPs).

No info yet on the bandwidth available per block, latency conditions, or other factors, but let’s run some numbers to see how this would work.

From recall, the Sun v40z’s were averaging ~3500pps (packets per second) outbound per server. I didn’t use packets per second to estimate previous effort, I created a quick excel chart similar to below to get a ballpark number (updated using 260,000 for the total number of IP addresses to scan):

approx. number of IP addresses to scan	260,000
number of TCP ports to query per IP address	65,535
total # of TCP ports to query	17,039,100,000

time-out setting per port query	1.25	seconds (allow 1.25 seconds for query response)
number of query attempts per port	2	which implies we need 34,078,200,000 SYN requests
total seconds per port query	2.50	seconds

number of parallel port queries	2	(we set it to 100, but empirical evidence shows this to be ~2)
total # of hours to complete single IP port scan	22.76

total number of seconds to perform scan (if sequential)	21,298,875,000
total number of hours to perform scan (if sequential)	5,916,354

number of scans in parallel	1,600	(2 servers, each running 8 unique nmap processes with min_hostgroup set to 100)
number of hours to complete scan	3,698
number of days to complete scan	154

Now I realize pps is an important factor. Remember these are still pretty fast servers: quad dual core opterons with 16 GB RAM. Doing a sanity check with our previous results of obtaining ~3500 pps per server goes as follows:

total # of SYN requests

34,078,200,000

average outbound packets per second / per server

3500

number of dedicated servers

total number of seconds to complete scan

4,868,314.29

number of hours to complete scan

1,352.31

number of days to complete scan

So the numbers don't match. The first chart says it'll take ~154 days, the second says ~56 days. In the first chart I set "number of scans in parallel" to 1600. I got that by having 8 unique nmap processes each running with min_hostgroup of 100 (800 hosts being scanned simultaneously per server; we have two dedicated scanners, so the total is 1600 hosts). The number that is probably way off is the "number of parallel port queries" in the first chart. Although we had set the value to 100 (meaning scan 100 ports simultaneously on each host) it often seemed like we were only getting 2 or so in parallel (observed from watching tcpdump output). That was probably after running into the memory issues I reported with nmap (which Fyodor subsequently reported to fix).

I just ran a few quick checks on a Sun Fire x2100 we have (dual core opteron 175 / 4 GB of RAM) using a newer nmap version (4.11). Firing off a bunch of nmap scans on one server with parameters similar to below resulted in a sustained 9000+ pps (~4.5 Mbits/s).

/usr/local/bin/nmap -vv -sS -P0 -p 1-65535 -n --min_hostgroup 100 --max_rtt_timeout 1250 --min_parallelism 100 <a_/24_block>

I didn't let the test run for very long to see if issues arise like before. If we could sustain 9000 pps per server and be allowed to push ~9Mbits/s, then the overall time is greatly reduced. It drops to ~22 days. I really doubt we'll be able to or allowed to push 9Mbits/s, but at least we now have ballpark figures to play with.

Who else has had this kind of crazy port scanning fun?

Related entry: maximizing nmap scans for accuracy

maximizing nmap scans for accuracy

tate@ClearNetSec.com (Tate Hansen) — Wed, 03 May 2006 01:05:00 -0600

There have been some recent newsgroup postings about consistency problems with nmap results (especially when scanning over the internet). In my experience, if you want the best chance of obtaining accurate nmap scan results, you need to:

check the latency to your targets (ping some device on the target net block for awhile, what is the average rtt value?)
- use the average to configure the --max_rtt_timeout nmap option (e.g. if your ping time is 300ms, you obviously want to set this option to more than 300ms to prevent nmap for missing an open port if/when a probe takes longer than 300ms. Maybe set it to 500ms or even higher. Speed suffers, but we're chasing accuracy)
if you can, check the available bandwidth limit for you and the target net (e.g. T3:45mbps, T1:1.544mbps, DSL, etc.)
- use this to configure the --min-hostgroup and --max-hostgroup nmap options and the --min-parallelism and --max-parallelism options (i.e. the more bandwidth each side has available, the more hosts and ports you can scan in parallel). This can significantly reduce overall scan time while keeping accuracy as the highest priority. I typically use IPTraf to watch pps (packets per second) to guage my bandwidth usage. If my options are too aggressive (i.e. consuming too much bandwidth), I'll kill the current scan and start over with different tuning values. The objective here is to set --min-hostgroup and min-parallelism to something that both sides can handle without sacrificing accuracy. If bandwidth permits, I like to set min_parallelism to 100 (number of parallel SYN requests or whatever scan type you're doing). I've gone up to 300 for --min-hostgroup (had lots of bandwidth and plenty of server horsepower). You don't want to cause your scan server, the target, or any device the probes traverse to drop packets, so the more information you have about all the components involved the better you'll be able to tune these values.
if you can, verify there are no 'smart' devices impeding scans (e.g. IPS, rate limiting device, etc.)
run multiple nmap scans of the same target, preferably at different times of the day and different days of the week (e.g. nightly backups can saturate certain hosts/links which causes nmap to fail to discover open ports). I like to run multiple services scans (~1600 ports) and do one or two 65k scans depending on circumstances. If time permits, the confidence boost you get knowing you've maximized accuracy by running repeated scans is satisfying. It also shows your assiduity; you can tell your customer you fired off 5 scans at different times to obtain your final list of exposed services. If you missed something after repeated scans (assuming you used safe tuning values), something is likely wrong.
set --host_timeout to something you are comfortable with.
- if you determine a scan may take 20 hours to perform a 65k port scan of the target device, then set this option accordingly. Note: you can use 's', 'm', or 'h' to specify seconds, minutes, or hours.
set --max-scan-delay to something low if nothing is impeding scans
- I typically set this option to 0 because I'm confident in my other tuning values (no delay between probes, just fire them off)

While tuning nmap options can do lots of good things it is still important to verify your scan server can keep up. We've knocked down 8-way Opteron systems w/16GB of RAM, so optimal nmap options may not be realistic for your hardware.

Scan fast and evade triggers

tate@ClearNetSec.com (Tate Hansen) — Tue, 14 Mar 2006 03:48:00 -0700

I've wanted to build this for a long time, alas the pain and costs of obtaining disparate public IPv4 blocks is high. I want to perform 65k port scans fast, accurately, and avoid 95% of the IDSes, IPSes, or whatever other ‘smart’ devices are in my way. It can be done.

Buy or lease some servers
Find a few data centers that connect to different Tier 1 providers
Justify and purchase IP blocks from ARIN (or another regional registry)
Setup scan server(s)
Setup NAT server(s)
Write some code to distribute port scans
Feel cool when you can scan like crazy
Feel really cool when no ‘smart’ devices alert, block, or rate limit you because you haven’t triggered any threshold ‘rules’
Act surprised when the client mentions his team didn’t see or report any anomalous behavior

Here is a high-level diagram of what I want:

Of course, there are some realities which make this hard to build. Registries prefer to hand out contiguous net blocks, but it would be far more desirable to have a bunch of smaller non-contiguous net blocks. Some ‘smart’ devices do detect scans based on the source net block, not just via a single source IP. Bandwidth and latency conditions are always in play. I still want it. A scan setup like this can increase accuracy, be fast, is distributed, and raises the difficulty for detection.

FYI: Initial costs from ARIN for different net block sizes

Category	Initial Registration Fee (US Dollars)	Assignment Size
X-small/ Micro-allocation	$1,250	/24 - < /20
Small	$2,250	/20 - /19
Medium	$4,500	> /19 - /16
Large	$9,000	> /16 - /14
X-large	$18,000	> /14

Quick list of cool new things in Nmap 4.00

tate@ClearNetSec.com (Tate Hansen) — Fri, 03 Feb 2006 11:57:00 -0700

I'm catching up on the new features in Nmap 4.00 from this Security Focus interview with Fyodor. Some good things to remember:

press [enter] anytime to get an estimate of when nmap will finish
press 'v' anytime to enable verbose mode / press 'V' anytime to disable verbose mode
there are now 3,153 signatures to detect an application or service (and possibly version) of a listening port
there is a new --version-intensity option which specifies how hard nmap will interrogate a listening port
new --badsum option which tells nmap to use invalid TCP or UDP checksums (can give you more information about a FW/IPS)
- Here is the link to Phrack #60 which gives the why and how of this new feature (written by Ed3f)
much faster (although this depends on things like bandwidth, latency, and which command line options you specified)
better OS detection (uses more tests to gain accuracy)

Here is a link to the official quick list of options for Nmap 4.00

$70,000 worth of new opteron servers for nmap scanning and they suck?

tate@ClearNetSec.com (Tate Hansen) — Tue, 27 Dec 2005 22:51:00 -0700

We recently performed a relatively large TCP port scan for a client; a full 65k SYN scan of ~70,000 IP addresses. Nmap is a great port scanner and was our first choice. We had two new and beautiful Sun quad v40z dual core opteron servers (16GB of RAM each) dedicated to the task. We were under a restrictive change control window and time was the limiting factor. We broke the scans down like this:

Executed 8 unique nmap instances on each system (one for each ‘virtual’ processor)
Divided the scans on /24 blocks (the optimal breakdown would’ve been on a 100 boundary, but we ran with this)
Set min_hostgroup to 100 (minimum number of devices to scan in parallel)
Set min_parallelism to 100 (minimum number or ports to scan in parallel)
Set max_rtt_timeout to 1250 (wait a maximum of 1.25 seconds to receive a reply from a port query)
Other command line options used (-vv: verbose, -sS: SYN Scan, -P0: no ICMP, -p: port range)

All together, a single nmap statement looked like the following:
/usr/local/bin/nmap -vv -sS -P0 -p 1-65535 -n --min_hostgroup 100 --max_rtt_timeout 1250 --min_parallelism 100 <a_/24_block>

We paid close attention to the number of outbound pps (packets per second) using iptraf for a couple reasons: To watch our bandwidth utilizations to avoid ISP overage charges and to gain a rough baseline so we could detect a problem.

If I remember right, the outbound pps initially was between 2k and 4k per server. Things were rocking and it looked like we would sail through the port scans. Alas, when doing a quick check after ~30 hours of scanning, we noticed the pps had slowed to ~550 per server. We deduced nmap had some memory management issues when used the way we crafted. Each nmap instance was consuming ~1.2GBytes of RAM, appeared to be increasing, and the CPU idle time for all processors was a continuous 0. This caught us off guard somewhat because we had successfully performed an nmap TCP services scan (~1668 ports) of the 70,000 IP addresses in less than 40 hours. This was all on SuSE Linux Enterprise Server 9 (x86_64) with nmap version 3.93. We knew now this was not going to be easy. The number of SYN requests to do this is big, roughly: 65,535 hosts x 65535 ports x 2 (number of port query attempts) = 8,589,672,450 outbound SYN packets. If we could sustain ~3,500 outbound pps on each server, then we could finish in approximately 15 days (within the change control window). At 1,100 total pps, it is ~90 days, ouch!

In the mix of this engagement, there was a timely posting on the network security pen-test newsgroup asking about scanning a large network with nmap in which I posted a reply. I subsequently received a response from Fyodor (the author of nmap) which not only confirmed our experiences but also contained a link to an updated version to better enable nmap to handle this scenario (thanks Fyodor!!). I haven’t had a chance to use this updated version yet, but I’m excited to check it out. Also I am going to explore some home grown scanners we used while building a vulnerability scan engine and play around with the scanrand and unicorn scanners. This ended up being a great experience and a wake-up call to verify the tools we depend on work at the scale we need before accepting the next job.