ClearNet Security

A client this month was observing strange FORM submittal activity and asked if I was still doing security testing of their web application.

Since I was not, I participated in investigating. I received a copy of their Apache log file.

I downloaded apache-scalp:

“Scalp! is a log analyzer for the Apache web server that aims to look for security problems. The main idea is to look through huge log files and extract the possible attacks that have been sent through HTTP/GET”

From viewing the output of Scalp! it was obvious an automated application scanner was being used. Three bits struck my curiosity:

• The scanner was one I hadn’t heard of: netsparker
• Netsparker is commercial, implying someone was willing to pay money to find vulnerabilities
• The source of the scans was Microsoft

My client was dumbfounded why MS was doing this. I suggested they send an email to abuse@microsoft.com with the details and request their assistance.

Lo and behold, Microsoft replied and was responsible. Their Online Services Security and Compliance (OSSC) team was actively scanning, unbeknownst to my client .

Given this client has a relationship with MS, my guess is they signed something with fine print explicitly allowing MS to perform ad-hoc vulnerability assessments.

Good to know this can happen.

It’s been some time, but I just had one those scope calls from out of left field. University, mid size, private, budget sensitive, one security engineer and he is relatively new.

He is unsure of the state of his domain.

Me: Have any security assessments ever been performed?
Client: No.
Me: pause Do you trust the integrity of your servers?
Client: No.
Me: pause You are processing credit cards?
Client: Yes.

Calls like this one ensue brain freeze in me. I get so hung up on them I mumble for a bit until I catch a clear thought.

On cue, this question soon followed: “I was thinking of starting with vulnerability assessments, but where do you recommend I start?”

Is this not a scenario security aficionados are to love?

I dread them now. I’m being tasked with teaching everything a functioning adult is to know on a phone call. Don’t get me wrong, I do love the flood of ideas this scenario stimulates, but I’m weary of working for clients so far behind in the game. My experience with like clients is they come to the play with little budget, support, and with even fewer resources.

But in the spirit of the holidays, I’m going to draft a list of tasks I’d recommend he do and I’ll share it here.

A recent web application I was testing was generating a unique hidden token on every render of the login page (revealed in the pink section in the screenshot below by WebScarab).

I had setup a BURP intruder run to iterate through a list of attack strings when I noticed a few of the responses contained a message displaying “unable to verify session”.

That is when I checked for a hidden value, and when discovered, guessed it was a dynamic key identifying each individual form submit.

The fuzzers in WebScarab and BURP typically work by repeatedly POSTing to a web form, like the login form above, then capturing and/or analyzing each response. Commercial scanners do the same. Repeatedly POSTing fails to work in this case.

Screenshot showing BURP’s intruder and that I want to run a sniper attack (i.e. send one attack string at this position (red text below) per POST request).

Screenshot showing a few custom attack strings loaded into BURP

Screenshot showing the execution and results of an intruder attack (i.e. looping through the attack string list and submitting each via a POST request)

Screenshot showing WebScarab’s fuzzer setup.

The problem in this scenario is the hidden token is only good for one POST request. Maybe I missed it, but I didn’t see an easy way to tell WebScarab’s fuzzer or BURP’s intruder to first perform a GET request (so I can capture a valid one time token value) then submit a POST request.

A few lines of Ruby along with the Mechanize library made for a simple work-around.

#!/usr/bin/ruby
require 'rubygems'
require 'mechanize'

agent = WWW::Mechanize.new
agent.set_proxy('localhost', '8008')

File.open("fuzzlist.txt") do |file|
  file.each_line {|fuzz_string| 
    agent.get('http://target/signin') do |page|
      form = page.forms.first 
      form.field_with(:name => 'data[user][email]').value = fuzz_string
      form.field_with(:name => 'data[user][password]').value = fuzz_string
      form.click_button
    end
  }
end

I configured mechanize to use my preferred proxy (BURP). The script loops through the fuzzlist.txt file, performs a GET request (now we have a valid token), then submits the form. Problem solved.

I just saw that Thawte is ending their Web of Trust email keys program. I think they should have sold directory services in addition to providing keys, perhaps I'll write about that next. It got me thinking about SSL/TLS. I've always felt that the openssl program does a lot more than most people know to do with it. 

For example:

openssl dst -sha1 yourfile

is about 20% faster than "sha1sum yourfile" on my OpenSuse 10.1 AMD Opteron system. That might not be terribly interesting but 20% adds up on bigger files, like DVD images.

 Another thing is in this world of automatic updates and appstores, it's more important than ever to have a verification mechanism in place. OpenSSL provide all of the tools for some simple, yet strong, verification. A lot of people I know are confused by certificates and don't fully understand the different formats and what you can do with them but OpenSSL provides the raw basics to simply perform signing and verification.

Generate a 2048bit RSA key:

openssl genrsa -out rsapriv.pem 2048

Extract the public portion:

openssl rsa -in rsapriv.pem -pubout -out rsapub.pem

Sign a file and store the signature in testsig.bin:

openssl sha1 -sign rsapriv.pem -out testsig.bin yourfile

To verify:

openssl sha1 -verify rsapub.pem -signature testsig.bin yourfile

There you go, signing and verification without any of the "complicated" PKCS stuff. It's only marginally more work to actually add a CA to the mix.

For me, it’s hard to downplay my joy from successfully assailing a web application and finding valuable faults: more so when nothing of value is reported from scanning using the most expensive app scanners.

On a recent gig I was focusing on finding issues with a couple of forms. Wielding BURP, I was intercepting all the GETs and POSTs and noticed the server responding with JSON data (i.e. http://[…]/data.json)

I tossed the JSON blob into an online JSON validator (http://www.jsonlint.com/) to make it easier to read (below is a small snippet).

"sanitized": 
  { “aaaDetail": 
    { "aa_id": "276060", 
      "bb_id": "103065", 
      "cc_id": "515ad8933c821b2f72a8cbb7054zed3c", 
      "x_time": "2009-08-12 21:42:40",
      "y_time": "2009-08-12 21:47:34",
      "full_name": "T H", 
      "comment": null, 
      "country": "United States", 
      "state": "Colorado",
       "city": "Broomfield"
       [...]
     }
  } 

I soon realized the JSON objects contained great information for crafting precision attacks.

The JSON data reads like they are database column values. I wondered for a moment what would happen if I just use the strings in the JSON objects for POST keys, came up with logical values matching the keys, then started POSTing the modified form data.

Bingo. By simply intercepting POST requests and appending new key/value pairs to the list of key/value pairs already being submitted I was able to modify database values unattended by the developers. In this case, it meant I was able to modify information inserted and owned by other users of the site.

I’m not espousing Harry Potter skills here, rather illustrating one of many examples of why Man is needed to go beyond where automated web app scanners stop.

Why I am proud to admit that I will never talk to any police officer. In Praise of the Fifth Amendment Right to Not Be a Witness Against Yourself.

I recently completed a pen test from the sole position of possessing remote administrative privileges to a few guest VMs.

I fired up yersinia to learn if launching layer 2 attacks could disrupt normal operations. While doing that I grabbed a copy of this book:
VMware vSphere and Virtual Infrastructure Security: Securing the Virtual Environment

vSwitches are reportedly immune to layer 2 attacks.

"Currently, a vSwitch will protect you from the following types of attacks by not providing the underlying functionality that these attacks require."

• MAC Flooding
• Double Encapsulation attacks (multiple 801.q envelopes)
• Multicast Brute Force Attacks
• Spanning Tree Attacks
• Random Frame Attacks
• DTP/VTP

I was able to perform MiTM via ARP cache poisoning on other visible guest VMs sharing the same network segment (because the attack targets the guest VM and not the vSwitch) – but in this case, each segment was allocated to a unique customer and therefore not a valuable test (i.e. poisoning one’s self was not in scope).

You can configure VMware to further limit attacks within a broadcast domain:

• Security | Promiscuous Mode (default is to reject) : Reject
• Security | MAC Address Changes (default is to accept): Reject
• Security | Forged Transmits (allow outbound frame w/a source MAC that is different) (default is to accept) : Reject

I just completed an external pen test whereby the rules of engagement limited the scan windows to two hours per night.  Requests for longer were rejected.

I hadn’t run within this tight of windows in some time and now I remember why I hate it so much.

I spent more time jacking any and every configuration setting I could tweak to boost each tool for balls out speed and baby-sitting (because failing seems to be a popular thing to do if you’re a tool sprinting at 50 threads and spending 0ms between requests) that I didn’t get nearly the time I wanted to concentrate on what I was paid to do: bust in.   

As a case in point I was working a SQLi point that was allowing me to download their entire database, alas, I only ever retrieved four of the 200+ tables during any one window.  Worse is I spun my wheels for several critical hours exerting fervent trial and error effort tweaking tool options, largely in vain, in hopes of making things go faster.  The consequence was tool tweaking dominated my attention.  Creativity, the force summoned for powersploiting, remained unconscious.    

• On small startup type teams, the team is everything, a bad team member can kill the product and the company.
• On particularly agile teams, having agile people is better than having technology experts.

Defending boils down to skill and incentive.

Tools that provide visibility are required, but that’s another topic. Skill is also an obvious requisite.

How about incentives? What incentives are in place for security engineers to really dig in and be great defenders?

Tier one engineers working out of traditional MSSPs are paid in the $20 per hour range or lower and by the nature of their position they have minimal understanding of anything of substance about their clients’ networks.

Opposite that, I know a very talented group of engineers working at an expensive outsourced IT/software company whom are responsible for their company’s top paying customers. They breathe uptime. Heads roll and the company loses money when downtime occurs. Security is barely an afterthought. Lest they do see a security issue, they may skip notifying for why create more support tickets and work.

It doesn’t make sense to punish defenders for failing to prevent infiltration – that is to be expected today. Simply detecting one is a great accomplishment.

To move defenders to be great defenders, reward them for detecting infiltrations.

I think it would be great for company’s to hire an outside party to perform unauthorized activity at an increasing pace and breadth until someone responsible for monitoring sounds the alarm. Reward those who discovered the activity. Do it frequently. Change it up. Make a competition out of it. No doubt this would help weed out bad performers, be they internal or external.

This is very similar to what I’ve seen a few hospitals implement. If an employee stops and challenges a person without a badge then they receive a $100 bonus.

I’m not promoting pen. testing here, though it’s a good example of a challenge.  I think simple and frequent small scale tests tied to rewards would work wonders for many security groups and for the company’s wanting to keep their assets protected.

I ran into a technical logging obstacle and I finally passed it today.

Setup:

• A large number of linux servers deployed over disparate locations and connected via consumer cable and xDSL lines
• All of them are configured to forward logs to a remote central log server via syslog-ng and UDP (client prefers UDP over TCP)

Problem:

• We were noticing a growing percentage would stop forwarding logs to the central log server

Troubleshooting notes:

• Timing seemed random; many clients forwarded logs successfully for weeks or months, some clients would stop forwarding after a few weeks, a few would stop in a couple of days
• All other connectivity appeared fine when troubleshooting (e.g. ssh worked, http worked, icmp, etc.)
• On each server, syslog-ng was running, logging locally fine, and showed no error conditions (but we observed no outbound traffic when executing tcpdump port 514)
• We did notice if we restart syslog-ng, forwarding would work again and we would immediately observe new logs appearing on the central log server

Quick thoughts and things we tried:

• Okay, configs are likely good because all works when restarting
• Network issues? But then why does restarting syslog-ng have an immediate and somewhat lasting effect?
• Alright, let’s upgrade to the lastest sylog-ng 2.x suite (libol, eventlog, syslog-ng) because maybe we’re running into a weird bug

Finali:

• After upgrading a dozen or so and waiting a week, we noticed two or three stopped forwarding logs again
• We executed tcpdump port 514 and again observed no outbound traffic
• We restarted syslog-ng and as before, observed new logs appearing on the central log server. This time it caught our attention that our tcpdump window still showed no outbound traffic even after the syslog-ng restart. Ah, now we’ve identified the information we were neglecting to consider: multiple interfaces.

[tt@00-e0-4d-8d-ce-3b ~]$ /sbin/ifconfig
eth0 Link encap:Ethernet HWaddr
[]
tun0 Link encap:UNSPEC HWaddr
[]


We executed tcpdump -i tun0 port 514 on a server for which we had not observed log messages on the central log server in some time. By watching tun0, we now observed outbound UDP packets, but the central log server was still not receiving them. When we restarted syslog-ng on this server, we now noticed a significant change in the tcpdump output. The first three lines below were observed when the central syslog server was not receiving messages and the second three lines are after we restarted syslog-ng.

15:57:22.009114 IP 01-01-01-01.att-inc.com.40655 > 555.55.253.145.syslog: SYSLOG authpriv.notice, length: 132
15:57:22.187959 IP 01-01-01-01.att-inc.com.40655 > 555.55.253.145.syslog: SYSLOG syslog.info, length: 104
15:57:22.188023 IP 01-01-01-01.att-inc.com.40655 > 555.55.253.145.syslog: SYSLOG syslog.notice, length: 97
15:57:22.699615 IP 777-777-229-216.static.data393.net.60408 > 555.55.253.145.syslog: SYSLOG kernel.notice, length: 249
15:57:22.700051 IP 777-777-229-216.static.data393.net.60408 > 555.55.253.145.syslog: SYSLOG kernel.notice, length: 254
15:57:22.700108 IP 777-777-229-216.static.data393.net.60408 > 555.55.253.145.syslog: SYSLOG kernel.notice, length: 246

Problem discovered (some of the IP information above was changed for obvious reasons).

On reboot or startup, tun0 is brought up and syslog-ng forwarded logs with the correct source IP for this environment (the one assigned to tun0). The catch is when an openvpn connection would bounce on a live system (i.e. tun0 would go down), syslog-ng would fall back to using the source IP assigned to eth0 and continued to use that IP even after tun0 was brought back up, which broke the forwarding of logs in this environment. Syslog-ng required a restart after tun0 would go down in order to use the proper IP (as observed in the tcpdump output for tun0 above). The flapping openvpn connection was the culprit.

Try inotify.

Inotify is a Linux kernel feature that monitors file systems and immediately alerts an attentive application to relevant events, such as a delete, read, write, and even an unmount operation.

The Inotify-tools library provides a pair of command-line utilities to monitor file system activity:

• inotifywait simply blocks to wait for inotify events. You can monitor any set of files and directories and monitor an entire directory tree (a directory, its subdirectories, its sub-subdirectories, and so on). Use inotifywait in shell scripts.
• inotifywatch collects statistics about the watched file system, including how many times each inotify event occurred.

We just need to check two events:

• # IN_CREATE - File/directory created in watched directory
• # IN_MODIFY - File was modified

#!/usr/bin/ruby
require 'inotify'
require 'find'

raise("Specify a directory") if !ARGV[0]
directory = ARGV[0]

i = Inotify.new

t = Thread.new do
      i.each_event do |event|
        File.open(directory + "/#{event.name}") do |f|
          f.grep( /\b(?:\d[ -]*){13,16}\b/) do |line|
            puts "Detected credit card pattern in directory #{directory}, file #{event.name}\n"
          end
        end
      end
end

Find.find(directory) do |e|
  begin
    i.add_watch(e, Inotify::CREATE | Inotify::MODIFY)
  rescue
    puts "Skipping #{directory}: #{$!}"
  end
end

t.join

Not too bad for an hour of playing around. It works.

I used the ruby inotify version 0.0.2 from http://raa.ruby-lang.org/project/ruby-inotify/, but if you do that, then you need to fix line 47 - change it to

r = rb_thread_select (fd+1, &rfds, NULL, NULL, NULL);

• credit card number regex found here: http://www.regular-expressions.info/creditcard.html
• inotify-tools: http://inotify-tools.sourceforge.net/
• ibm inotify tutorial: http://www.ibm.com/developerworks/linux/library/l-ubuntu-inotify/index.html

removed for the time being...

A friend told me recently his new security hire has two passions in life: security and poker. The word passion is strong and I can imagine this new hire’s use of that word and his subsequent explanation helped him to win the position. If his win was from describing his poker passion, then that’s cool and I have nothing to add.

Anyhow, this made me wonder if there's an area of security I could say I'm currently passionate for. One area did stand out and the distinction I made was between offensive and defensive security.

For me, defensive security is it.

I’m not convinced my career survival instincts are somehow overpowering my reason, but to me it feels like to be awesome at offensive security is to be an insect. Thinking back over several penetration tests I clearly remember wishing I had more knowledge of the workings of a particular exposed service or application.

This is not to say it’s not fun to be an insect at the right time. With penetration testing, it can be crazy fun to be an insect, surrounding by lots of other insects of different skills, and all attacking the same target with abandon. It’s not though when you run with a small crew and run into a service or app for which no one has familiarity.

Defense on the other hand is a lot less insect like. You can be really good yet skip out on the training for your newly deployed app server. Playing defense therefore is an evolutionary step up of sorts and requires more intelligence that playing the offensive insect (pun intended).

Ha. Only kidding.

My take is a small and intelligent team can play defense very well without the need to dive deep into their infrastructure's every exposure. That is why I like defense - you can grasp its entirety.

There is a good newsgroup thread running on Dailydave.
http://seclists.org/dailydave/2008/q4/0085.html

The takeaways are familiar reminders for the cognoscenti, but it’s still good to read and good for referencing.

“Patch management, IDS, Anti-Virus, scanners of all shapes and sizes. Audits” don’t work against competent attackers.

“And they [Penetration testers] agreed on two things: the threats you know about are not the ones you need to worry about; and every network is own-able. Every. Single. One.”

“If you accept the premise that it's not possible to protect every asset (or even protect any single asset completely), then the logical action is to identify the most valuable assets and secure them to the best of your ability” column by Dennis Fisher

“Baseline system and network behaviour. Analyse any abnormal behaviour. (Easier said than done. You may never see anything.)” (raus)

“I would also note that it's misleading to say you should throw in the towel because one unpublished vuln can pop your box. There is more to it than that if you are doing your job right. Can they pop it without being discovered... for how long, and how often?” (Dragos Ruiu")

Marc Maiffret opined:

“The biggest threat to the average computer user is not zeroday vulnerabilities but system misconfigurations and vulnerabilities within third party applications. Most organizations are only just starting to get a handle on patching Microsoft vulnerabilities let alone third party applications. This becomes even more apparent with consumers and small to medium sized businesses where they only have Windows Update and WSUS to depend on. There is simply no third party patching being done in these environments making it a LOT more likely for them to get owned with a 6 month old Adobe Acrobat vulnerability than some zeroday vulnerability. This is currently the lowest hanging fruit for attackers and does not require an attacker to have large sums of money to waste on buying zeroday attacks.

It’s clear security teams must deploy tools that add to their sense of understanding for what is normal activity. You want intuition and clarity. You want to have that gut-instinct and confidence that you can detect if something is not quite right. The way to do that is to deploy tools that enchance visibility (i.e. tools that show you traffic patterns and volumes, running applications, logins, tools that point out unusual activity, etc.).

Having never looked back until now, I decided to link to my more popular posts of time past. Here are five:

Unbalanced reliance on prevention

Competing for network based security assessments

Compromised, Where are the logs?

Attackers will win so what can you do?

Test commercial web app scanners for free and without restrictions

How soon will we see crazy apps built on top of something like macrosense? The more wile and guile types will welcome this to sweeten their malevolence.

• "What if you could look at your cell phone and see a heat map of where everybody in the city was at that very moment?"
• "Over time, it learns about where you like to go (fancy restaurants or punk rock clubs) and shows you other people like you, and where they are—right now."
• "Imagine a GPS that didn’t just tell you where you are or where your programmed destination is, but a GPS in your phone that actually predicts what you want to do and where you want to go."

Yeah, it’ll be wonderful when you’re driving along and the GPS announces the strip club is up on the right when your wife is riding along. Honey, I have no idea why the GPS thinks I want to go there. Stupid GPS.

News article: http://www.iht.com/articles/2008/06/22/business/22proto.php

If it is good at doing A, it sucks at doing B. This is the banal trade-off of security point solutions.

WSJ recently featured SIEM in an article titled: Looking for Trouble

What is a good SIEM? I would say one that is an anomaly and misuse detection system, a sink for other like systems, and a sink for other observable facts (e.g. logs).

What does a good SIEM do? For most I believe the best answer is one that only taps on your shoulder when there is a real problem.

What does a SIEM need to do to be good? Tricky question. I would say one that understands which streams of incoming data are good for doing A (or identifying A), understands why certain streams are bad for making inferences (e.g. it's not good to automatically infer something is really important because an IDS is sending 1000 alerts per minute), and one that’s forged an algorithm mix that works.

This post is really about the answer to the last question above.

SIEMs rely on both sides of the detection coin:

• Misuse: good at detecting known attacks using signatures
• Anomaly: good at detecting unknown attacks by modeling behavior

The misuse side of the coin is clean and shiny; you can see a picture of the SNORT icon. SNORT is an example of a solid misuse detection system.

The anomaly side of the detection coin is dirty - it’s hard to see anything clearly. Why?

It’s because there is no single anomaly technique representing perfection. Stated in another way, if you fall into the hole of anomaly detection techniques you’ll never hit bottom, the hole as no bottom.

• Statistics
• Probability
• Machine Learning: there are literally hundreds (maybe thousands?) of papers applying machine learning techniques to computer and network security

Anomaly detection is compounded by the fact that algorithms are often combined in different ways to detect different types of anomalies. The gigantic streams of comingled and fragmented data (e.g. logs, xflows, IDS alerts, HIDS alerts) means huge numbers of permutations.

Circling back, to build a smart SIEM that excels at its job, it must employ and combine algorithms in way that it focuses on using the good at doing A information. This means you have to experiment with mixing algorithms and chain them together so the output only taps you on the shoulder when there is a problem.

1. A simple example is taking the deluge of alerts a snort instance emits and wrapping them up in a statistics model. The better value may come from recognizing numerical changes (min, max, median, mean, standard deviation, etc.).

2. A more complicated example may be applying NLP (natural language processing) techniques to analyze logs and extract user information, coalesce misuse detection alerts, associate statistical values derived from modeling xflows, then layering additional algorithms on top to correlate and present compelling evidence of strange behavior (i.e. a problem).

We’ve heard and recently noticed companies scaling back their investments in the research needed to advance solving the hard problems the developers of SIEM face. It may indicate the bigger players are planning to coast for a while on the past decade’s techniques. Fall behind though, and you’re out.

Given the advances in computing power (a big reason why AI and machine learning are so hot), it is also becoming acceleratingly more difficult to keep up, understand, and evaluate techniques beneficial to both the builders and consumers of SIEM systems.

If you find yourself evaluating SIEM products, dig in and investigate how each works - you don’t want yesterday’s product.

A big event happened in July and it went largely unnoticed, or so it seems, so I'll announce it here. Info-zip, one of the most popular programs around, has released version 3.0! There are actually a lot of good and timely new features in the 3.0 release of the PKZip clone.

• large-file support (i.e., > 2GB)
• support for more than 65536 files per archive
• multi-part archive support
• bzip2 compression support
• Unicode (UTF-8) filename and (partial) comment support
• difference mode (for incremental backups)
• filesystem-synch mode
• among others.

Bzip2 compression is interesting, it modernizes zip a bit but the things that are really important is the large-file support and support for more than 65536 files per archive, those limitations have become almost regular problems for some of us lately.

A SecTor keynote presenter put forward something close to that line in a PowerPoint slide.

Don't buy technology to detect

I didn’t get all the details down given I was trying to zero in on his line of thinking once I read such a startling suggestion.

He did provide his reasoning (which was derived from surveying business consumers of security solutions). The gist of it was that companies were deploying detection technologies (aka SIEM/log management products) and were unable, technically or resource wise, to handle the added compulsorily work load spiked by the enhanced visibility.

Paraphrasing, he further suggested that companies should purchase products that do something, not ones that only do detection. He cited examples of business consumers whom lack knowledgeable staff to understand the alerts detection systems produce and ones unable to tackle the volume of alerts. I think we all can get that.

But is this really a practical suggestion? Prevention (i.e. tools that do something) is great, but detection is King! The conjecture to skip detection tools in favor or tools that do something is weak, especially if the data you are protecting has value.

How about the World Bank as a good example? It reads like they made prevention King and detection something much less.