ClearNet Security: Tag scanning

Follow-up on using unicornscan for a big scan (400,000+ public IPs)

tate@ClearNetSec.com (Tate Hansen) — Thu, 27 Dec 2007 12:36:00 -0700

I’m happy to report our growing experience using unicornscan for large discovery sweeps is a positive one. Our confidence in using this tool has increased and it is now our preferred weapon of choice for scanning large IP swaths.

To recap: We performed a sweep of 400,000+ public IPs across multiple continents by configuring the scans to do a full TCP port scan of each IP, sustained ~55 Mbits/s using between 3 and 5 systems, and completed it in a matter of days.

This is pretty good considering by sending two SYN probes per port it meant sending ~52.5 billion packets and producing some 3 Terabytes of data.

Nmap is often our preferred tool, and we used it to spot check our results with unicornscan, but from now on it will come down to the details of the gig to make the choice.

Tech note: We avoided problems with table overflows and other like issues by placing the systems directly on the internet and with iptables turned off.

PCI: Not our problem...

tate@ClearNetSec.com (Tate Hansen) — Wed, 16 May 2007 20:51:00 -0600

What happens when the test environment operated by MasterCard (they “own” the testing lab) is misbehaving? I know. They yank the wheel, swerve away from responsibility, and point to the PCI council. And PCI? They point back. Beautiful, no?

You see because they refuse to disclose missed results to you they duck responsibility for anything that may have been their fault. They also clearly imply if anything is missed in your attempts to identify vulnerabilities then it is surely your fault or a problem with the tools you used.

I love it: No clear pass criteria, no way to challenge a decision, and no transparency of what or how they are doing. For all this great service you get to spend thousands every year!

So what happens when you call bullshit and raise hell? They pass you. :) Let me not forget to mention we had a few extra bullets in our clip they may have unexpected us to have – bullets provided to us by friends with information.

Be forewarned; this process has serious issues.

Scan fast and evade triggers

tate@ClearNetSec.com (Tate Hansen) — Tue, 14 Mar 2006 03:48:00 -0700

I've wanted to build this for a long time, alas the pain and costs of obtaining disparate public IPv4 blocks is high. I want to perform 65k port scans fast, accurately, and avoid 95% of the IDSes, IPSes, or whatever other ‘smart’ devices are in my way. It can be done.

Buy or lease some servers
Find a few data centers that connect to different Tier 1 providers
Justify and purchase IP blocks from ARIN (or another regional registry)
Setup scan server(s)
Setup NAT server(s)
Write some code to distribute port scans
Feel cool when you can scan like crazy
Feel really cool when no ‘smart’ devices alert, block, or rate limit you because you haven’t triggered any threshold ‘rules’
Act surprised when the client mentions his team didn’t see or report any anomalous behavior

Here is a high-level diagram of what I want:

Of course, there are some realities which make this hard to build. Registries prefer to hand out contiguous net blocks, but it would be far more desirable to have a bunch of smaller non-contiguous net blocks. Some ‘smart’ devices do detect scans based on the source net block, not just via a single source IP. Bandwidth and latency conditions are always in play. I still want it. A scan setup like this can increase accuracy, be fast, is distributed, and raises the difficulty for detection.

FYI: Initial costs from ARIN for different net block sizes

Category	Initial Registration Fee (US Dollars)	Assignment Size
X-small/ Micro-allocation	$1,250	/24 - < /20
Small	$2,250	/20 - /19
Medium	$4,500	> /19 - /16
Large	$9,000	> /16 - /14
X-large	$18,000	> /14

Tools for fingerprinting apps, services, and OSes

tate@ClearNetSec.com (Tate Hansen) — Mon, 13 Mar 2006 22:08:00 -0700

I was wondering how many different network-based fingerprinting tools are out there which use unique detection techniques. I know several commercial network scanners use Nmap, so if you decide to run Nmap by yourself and commercial tool X to see how they compare, you may (or even likely) be running the same thing. Obviously it can be a lot more helpful to have a handful of tools in which each has their own way to guess what the remote OS version is, or application version, or service. I've started to compile my own list and I haven't delved into the details of how each performs fingerprinting, but here is the list so far.

Tool	Date of last version	version	OS	Service	Protocol
nmap	Feb, 2006	4.01	yes	yes	yes
xprobe2	Feb, 2005	0.2.2	yes	no	no
p0f	Sep, 2004	2.0.6	yes	no	no
amap	Sep, 2005	5.2	no	yes	yes
nessus	Mar, 2006	3.02	yes	yes	yes
winfingerprint	Mar, 2006	0.6.x	yes	yes	yes
httprint	Dec, 2005	301	no	no	yes
queso	Aug, 1998	980922	yes	no	no
NTP-fingerprint	Feb, 2005	0.1a	yes	no	no
ike-scan	Dec, 2005	1.8	no	yes	yes
thcrut	May, 2003	1.2.5	yes	no	no
smtpmap	Dec, 2001	0.6	no	yes	no
smtpscan	May, 2003	0.5	no	yes	no
snacktime	Jun, 2003	0.5	yes	no	no
synscan	Apr, 2004	0.1	yes	no	no
telnetfp	Jan, 2001	0.1.2	yes	no	no
ldistfp	May, 2001	0.1.4	yes	no	no
telnet	N/A	N/A	yes	yes	yes
siphon	May, 2000	666	yes	no	no
ring		0.0.1
scanssh	Mar, 2005	2.1	no	yes	yes
hackbot	Dec, 2003	2.21	no	yes	yes
hping3	Nov, 2005	3.0.0	yes	no	no
induce-arp.pl	May, 2000	0.27	yes	no	no
vmap	Aug, 2003	0.6	no	yes	yes
disco	Jul, 2003	1.2	yes	no	no
k9			yes	no	no
ettercap	May, 2005	NG-0.7.3		yes
Net::SinFP	Mar, 2006	1.00	yes	no	no
Archaeopteryx	Jul, 2001	1.0	yes	no	no
iQ	Apr, 2002	0.2	yes	no	no
sprint	Mar, 2003	0.4.1	yes	no	no

Web application scanners often fail crawling

tate@ClearNetSec.com (Tate Hansen) — Sun, 05 Feb 2006 23:37:00 -0700

I attended a webinar Friday hosted by Watchfire which covered their web application scanner titled AppScan 6.0. The two big competitors I've run across in this space are Watchfire (formerly Sanctum) and SPI Dynamics. SPI Dynamics' web application scanner is titled WebInspect.

These scanners are great at capturing all the low-hanging fruit (i.e. vulnerabilities) if they can successfully crawl the target website. The problem, one the can cause a consultant considerable pain, is when you hit a site which uses technology that 'builds' URLs dynamically (e.g. JavaScript).

A JavaScript Example:
<script language="JavaScript">
function goToPage(element_name) {
  window.location = "http://www.mysite.com?tracking=" + getelementbyname(element_name).value;
}
</script>

As you can read above, the complete URL is generated using the value of a variable.

Let's take a quick look at a recent feature comparison from a September 2005 review of web application scanners by Secure Enterprise (link to the article)

If you look at the chart, it says all three of these scanners perform JavaScript parsing. Have you ever wondered why they don't seem to discover all the possible links in a web application? There is kind of a trick word here; can you guess which it is? It is the word 'parsing'. This is the word which makes us think these scanners can blaze through dynamic web applications. What they really mean by this is they can search through all the code and locate static URL paths like http://www.mysite.com. But, if the target site builds their entire menu system, or navigation, or forms, or whatever via JavaScript (or VBScript), then you are likely out of luck. Execution is what is needed, not just parsing. The scanner must execute code (e.g. window.location = "http://www.mysite.com?tracking=" + getelementbyname(element_name).value;) to generate all the potentially valid URL paths within an application.

Now all of these web application scanners support a work-around - what do you think that is? Here is a hint: You better have an excellent idea of how the site works and what all the application can do. The work-around is you must crawl the entire site for the scanner. No problem you say? Well, that may be true, but our experience often results in pain. Like the time we were covering for another consultant and realized we had to manually enumerate one of the largest web-based business performance management (BPM) systems on the market in two days. It was one of those types of experiences you grow stronger from.

So, if you are unfamiliar with all the different views a web application can generate and you are counting on a commericial web application scanner to do most of the heavy lifting, then be cautious. The time it would take to really do a good job may easily be 10x longer than you estimated.

Note: To be fair, WatchFire did say in their webinar they would be adding execution capabilities in their next release in 9 to 12 months. It'll be interesting to see how much they execute.

Update 2/22/2006: The release notes for the new WebInspect version 5.8 says: "Support for Advanced Asynchronous JavaScript and XML (AJAX) Applications. Improvements to the JavaScript and Audit engines now allow WebInspect to crawl and audit AJAX-based applications."

This vulnerability scanner is so fast it must be good!

tate@ClearNetSec.com (Tate Hansen) — Sun, 22 Jan 2006 11:11:00 -0700

I wish that was true, but it is furthest from the truth. There is an unfortunate conception for many tasked with reviewing or choosing a scanner (a conception espoused by the marketing of several Vulnerability Assessment vendors) which is the quality of a scanner is directly based on how fast it can scan. In this race for speed, vendors’ almost always default to performing less than complete checks. In fact, some vendors shy away from adding checks to their products in an attempt to retain speed (i.e. adding more checks will slow a scanner down). That is probably not what you want. I also have yet to see a single commercial scanner default to performing a full TCP port check. Totally understandable; if a scanner’s default policy opted for full coverage then you would surely dismiss the quality after running a test scan and learning you have to wait 40 hours for the results. Too bad. In the world of network-based vulnerability scanners there is a trade-off that spans long: speed and accuracy. Speed kills accuracy.

“Watch this man, I can scan a class B in 10 seconds.”
“Really? How did you do that?”
“Oh, well, it is only checking if one port is open.”
“Ah, nice.”

So if you want accuracy, then slow it down. Don’t run a thousand threads, allow more time for remote devices to respond, choose complete port coverage, and don’t parallelize so much you saturate switch ports or test your operating system's TCP/IP stack limitations.

A late entry to our scanning laptop race

Cory Stoker — Thu, 19 Jan 2006 13:39:00 -0700

I run a Powerbook 15", as my main system. It handles most things pretty well except for running Windows in VirtualPC. VirtualPC is so painful on the Powerbook that I hardly ever use it. Now with the Apple line going Intel their could be a possibility to dual boot Mac OS X and Windows. It all depends on the the new BIOS called EFI. If Windows can boot with using EFI, then I think I have found my new scanning laptop!

Getting crazy with proxy chaining

tate@ClearNetSec.com (Tate Hansen) — Thu, 19 Jan 2006 02:33:00 -0700

For efficiency, thoroughness, or comparison you can chain several popular web application assessment tools together. Three tools I sometimes chain in a series are the BURP Spider, Paros Proxy, and WebInspect. To do this on a single system, you simply configure a listening port for each tool. Check the diagram below:

You can configure each tool to do this by specifying a listening port for incoming requests and an IP address:listening port for outgoing requests. In the diagram above, BURP Spider is listening on localhost:9002 (port #), Paros Proxy is listening on localhost:9001, and WebInspect on localhost:9000. Each tool forwards incoming requests to the next in line (WebInspect, in the diagram above, sends the original request to the target site).

Paros distinguishes the proxy setting configurations as follows:

“Local proxy”: This is for incoming requests
“Use an outgoing proxy server”: This is for outgoing requests

BURP Spider:

“Proxy running on port”: This is for incoming requests
“Use proxy server”: This is for outgoing requests

WebInspect:

“Step Mode Listening IP Address and Port”: This is for incoming requests
“Proxy server”: This is for outgoing requests

Below are screenshots of the tools in action with the above configuration.

If you want to get super crazy, you can do exploratory investigating of target websites with the above tools and do it all anonymously with Tor and Privoxy (albeit potentially sacrificing thoroughness due to Privoxy filtering)

$70,000 worth of new opteron servers for nmap scanning and they suck?

tate@ClearNetSec.com (Tate Hansen) — Tue, 27 Dec 2005 22:51:00 -0700

We recently performed a relatively large TCP port scan for a client; a full 65k SYN scan of ~70,000 IP addresses. Nmap is a great port scanner and was our first choice. We had two new and beautiful Sun quad v40z dual core opteron servers (16GB of RAM each) dedicated to the task. We were under a restrictive change control window and time was the limiting factor. We broke the scans down like this:

Executed 8 unique nmap instances on each system (one for each ‘virtual’ processor)
Divided the scans on /24 blocks (the optimal breakdown would’ve been on a 100 boundary, but we ran with this)
Set min_hostgroup to 100 (minimum number of devices to scan in parallel)
Set min_parallelism to 100 (minimum number or ports to scan in parallel)
Set max_rtt_timeout to 1250 (wait a maximum of 1.25 seconds to receive a reply from a port query)
Other command line options used (-vv: verbose, -sS: SYN Scan, -P0: no ICMP, -p: port range)

All together, a single nmap statement looked like the following:
/usr/local/bin/nmap -vv -sS -P0 -p 1-65535 -n --min_hostgroup 100 --max_rtt_timeout 1250 --min_parallelism 100 <a_/24_block>

We paid close attention to the number of outbound pps (packets per second) using iptraf for a couple reasons: To watch our bandwidth utilizations to avoid ISP overage charges and to gain a rough baseline so we could detect a problem.

If I remember right, the outbound pps initially was between 2k and 4k per server. Things were rocking and it looked like we would sail through the port scans. Alas, when doing a quick check after ~30 hours of scanning, we noticed the pps had slowed to ~550 per server. We deduced nmap had some memory management issues when used the way we crafted. Each nmap instance was consuming ~1.2GBytes of RAM, appeared to be increasing, and the CPU idle time for all processors was a continuous 0. This caught us off guard somewhat because we had successfully performed an nmap TCP services scan (~1668 ports) of the 70,000 IP addresses in less than 40 hours. This was all on SuSE Linux Enterprise Server 9 (x86_64) with nmap version 3.93. We knew now this was not going to be easy. The number of SYN requests to do this is big, roughly: 65,535 hosts x 65535 ports x 2 (number of port query attempts) = 8,589,672,450 outbound SYN packets. If we could sustain ~3,500 outbound pps on each server, then we could finish in approximately 15 days (within the change control window). At 1,100 total pps, it is ~90 days, ouch!

In the mix of this engagement, there was a timely posting on the network security pen-test newsgroup asking about scanning a large network with nmap in which I posted a reply. I subsequently received a response from Fyodor (the author of nmap) which not only confirmed our experiences but also contained a link to an updated version to better enable nmap to handle this scenario (thanks Fyodor!!). I haven’t had a chance to use this updated version yet, but I’m excited to check it out. Also I am going to explore some home grown scanners we used while building a vulnerability scan engine and play around with the scanrand and unicorn scanners. This ended up being a great experience and a wake-up call to verify the tools we depend on work at the scale we need before accepting the next job.