ClearNet Security

If you want to use a custom word dictionary with ElcomSoft Distributed Password Recovery (Break complex passwords, recover strong encryption keys and unlock documents in a production environment) (I’m using version 2.96 currently), then try this:

First, you have to save your dictionary (which is in fact just the word list, one word per line) into UNICODE, and give it *.udic extension. Then, put that file into the following folder (on server and all agents; automatic distribution of dictionaries from the server to agents is not implemented yet):

%PROGRAMFILES%\Elcomsoft\Common Files\dic\

%PROGRAMFILES% usually points to ”C:\Program Files\” on 32-bit systems and ”C:\Program Files (x86)\” on 64-bit ones. Then, restart the console, and you should be able to use the newly added (custom) dictionaries.

Thanks to V.Katalov for the instructions.

If you’ve exhausted your efforts at gaining control of additional hosts while performing a penetration test then you may want to throw a Hail Mary and check if a device you control is connected via the serial port to something special.

# minicom

Like everyone else I’m hearing the drum beat getting louder – penetration testing as practiced by most today is considered a moribund service for pay: its value to businesses diminishing.

The chaos and confusion besetting all involved in security work due to how each participant defines popular security services doesn’t appear to be heading toward sanity any time soon (i.e. what is a pen. test to you?).

I keep getting hit with the same type client inquiries that Chris Nickerson recently received and posted on twitter the other day:

I fired off a couple comments.

I recently watched Eric Smith’s and Dave Kennedy’s B-sides presentation on vimeo and then read Dave’s blog entry on Traditional Pen Testing is Dead.

For the last year I’ve been spending way more time within the high level recommendations section of my reports and on the phone than in previous years – this coming from my personal frustrations clueing clients in on the sheer variety of ways an attacker can cause harm to their business.

Note: I’d be remiss here not to say I have liberally borrowed content from others – much of it recently from Richard Beijtlich’s blog and Mandiant’s blog and webinar series.

Like the process others’ go through when writing a penetration test report I would spend considerable time reviewing vulnerabilities and reducing the volume of the details into something digestible.

I don’t do that so much now. For example, today if I scan a client and discover a crazy number of vulnerabilities then I try to jump out of the vulnerability mud quickly. It’s much easier to ask the client if they have a patch management program: if the answer is no, you tell them to get one, if the answer is yes, then you raise the problem flag.

More to the point, my recommendation section now reads more like a kick start to maturing incident response (I think pen. testing should be more of an activity to test incident response (aka red team, but maybe without active resistance) http://blog.clearnetsec.com/2007/09/24/flip-the-reporting-and-get-the-customer-to-speak

In any case no matter what I was originally hired to do I like to put content like below in each report to stimulate conversation:

• What do we do if we discover malware on a critical system?
• What do we do if we are unable to explain particular log events or strange activity on a critical system?
• What do we do if a critical resource becomes unavailable?
• What do we do if sensitive data we store begins showing up in public?
• What do we do if we suspect a trusted or internal person is causing harm?

I have yet to discover a client that answers the above questions as Mandiant recommends for how you should answer them.

• Mandiant’s “State of the Hack” webinar series: http://fred.mandiant.com/SOH/SOH072010/lib/playback.html
• A Mandiant blog entry on the challenges of remediation if an attacker is successful: http://blog.mandiant.com/archives/1260

I also like the threat-centric approach, so I often add in the following questions:

• What are the most important assets and why?
• What threats worry the team the most?

The chart above is great for stimulating conversations with executives. They typically start rambling immediately about what on this list worries them. Even better, it immediately adds clarity to what you’re trying to accomplish as they ponder how each of these threats may impact their business.

Obviously I’m cherry picking from various formal security activities, but for clients new to maturing their security beyond addressing vulnerability scan reports, it works.

And given the propensity of most businesses to think of security only in terms of their servers (à la Chris’ tweet), I make sure to point out what wasn’t tested:

Your business and common threat tactics:

• Servers on the public Internet: tested
• Application Security: minimal tests performed
• Social Engineer: no tests performed
• Phishing: no tests performed
• Client-side: no tests performed
• Wireless: no tests performed
• Physical Security: no tests performed

Name the arcade game the title is from and know that you’re cool.

Jettison responsibility, and go! SaaS!

• http://www.zerigo.com/ for dns
• http://www.dropsend.com/ for secure file transfer
• http://mediatemple.net/wordpress-webhosting.php for blogging
• http://chargify.com/ for accepting payments
• http://mozy.com/pro/features/ for backups
• search & replace the above or add your own here

Advantages galore and better security! If you do it right then you have a LOT less to worry about. Let the admins of the above panic over breaches, hunt for lateral movement, and binge on incident response fodder. Keep your secrets safer by NOT running the above in-house. Think of how much less logs, less flows, and less time it’ll take you to vet any security weirdness in your infrastructure.

SaaS vendors are motivated to ensure data security - too many incidents and their customers will flee. And from what I’ve seen recently, the better SaaS vendor teams would crush in-house infrastructure teams in a security fight. It’d be like the in-house teams were walking into the Teutoburg Forest.

Do you ever suffer from client-induced nausea? You know, the feeling you get after the CISO jollily asks “How are we doing at security” soon after you’ve discovered they do next to NOTHING.

If you’re like me and if you get too many of these in short order you get sick. Sick like “I don’t care anymore, f*** this”. My nausea peaked recently and I wondered if complete cynicism is the unfortunate end.

I then re-read a snippet of an email I received:

“But they _did_ hire you, so there must be a spark of initiative there.”

Yes. I was delighted to ponder this thought for a minute. Complete cynicism be damned!

How is it then you, I, anyone in security can keep healthy amidst what reality is like today? For me the answer is I like and have fun doing security. So to all my future gigs that make me sick, I don’t care, that is, I don’t care how bad or stupid it all is, security is a trip I like to be on.

You’re in, but you don’t have God power yet. Before giving up, add a VIP and scan for 192.168.20.1.

“Your DRAC contains an integrated 10BASE-T/100BASE-T Ethernet NIC and supports TCP/IP. The NIC has a default address of 192.168.20.1 and a default gateway of 192.168.20.1.”

If that IP responds, you’re that much closer.

The default user name for this account is “root” and the default password is “calvin”.

Of course the above is well known and vulnerability scanners have checks for it. But as I just witnessed at a client, none of their historical vulnerability scan results discovered the cards because this client doesn’t use that IP block, yet several Dell servers had default DRAC cards waiting for some love. Be a good God.

If you like linux, capistrano, and ossec then you may like these scripts to automate installing ossec agents. I’m assuming you already have decent familiarity with ossec & capistrano.

prerequisites:

• you can ssh to each of the hosts you want to install the ossec agent on using ssh pki (e.g. ssh-copy-id -i .ssh/id_rsa.pub alice@xxx.xxx.xxx.xxx)
• you have nopasswd sudo privileges on each host (check via sudo -l)
• you have a compiler (+ dependencies) installed on each host (yum -y install gcc glic-devel)
• you have capistrano installed
• you have the ossec source (current ver. = ossec-hids-2.4.1.tar.gz) (just put in your homedir)

steps you need to do:

• uncompress the ossec source
• use ossec-batch-manager.pl to quickly add agents to the server (located in the ossec contrib dir) (e.g. [alice@server]$ sudo ./ossec-batch-manager.pl -a -n host1 -p 10.10.10.10)
  note: you need to restart the ossec server after adding agents
• copy ./contrib/specs/agent/preloaded-vars.conf to your homedir then configure it for how you want your agents to work (make sure you specify your USER_AGENT_SERVER_IP) - mine looks like:

  [alice@server]$ cat preloaded-vars.conf | egrep -v “^#|^$”
  USER_LANGUAGE=”en”
  USER_NO_STOP=”y”
  USER_INSTALL_TYPE=”agent”
  USER_DIR=”/var/ossec”
  USER_DELETE_DIR=”y”
  USER_ENABLE_ACTIVE_RESPONSE=”n”
  USER_ENABLE_SYSCHECK=”y”
  USER_ENABLE_ROOTCHECK=”y”
  USER_UPDATE=”y”
  USER_UPDATE_RULES=”y”
  USER_AGENT_SERVER_IP=”xxx.xxx.xxx.xxx”
  

• create a file titled capfile in your homedir then copy & paste in the following code

  default_run_options[:pty] = true
  default_run_options[:max_hosts] = 25
  
  # VARIABLES
  set :ossec_version,     "2.4.1"
  
  # ROLES
  role :new_agents,
  '10.10.10.10',
  '10.10.10.11',
  '10.10.10.12'
  
  # TASKS
  # ================================================================================
  # Automatically install new UNIX based OSSEC agents  
  # ================================================================================
  namespace :new_agents do
  
    task :install_ossec_agent do
      upload("ossec-hids-#{ossec_version}.tar.gz", "/tmp/", :via => :scp)
      run "cd /tmp && tar zxf ossec-hids-#{ossec_version}.tar.gz"
      upload("preloaded-vars.conf", "/tmp/ossec-hids-#{ossec_version}/etc/preloaded-vars.conf", :via => :scp)
      run "cd /tmp/ossec-hids-#{ossec_version} && sudo ./install.sh"
      sudo "\\rm -rf /tmp/ossec-hids-#{ossec_version} && \\rm -f /tmp/ossec-hids-#{ossec_version}.tar.gz"
    end
  
    task :install_ossec_agent_keys do
      servers = find_servers_for_task(current_task)
       servers.each do |server|
        key = `sudo grep #{server} /var/ossec/etc/client.keys`
        logger.info "installing key on #{server}\n"
        put(key, "/tmp/client.keys", :hosts => server)
       end
      sudo "mv /tmp/client.keys /var/ossec/etc"
      sudo "chown root:ossec /var/ossec/etc/client.keys"
    end
  
    task :start_ossec_agents do
     sudo "/var/ossec/bin/ossec-control start"
    end
  
    task :do_all, :roles => :new_agents do
      install_ossec_agent
      install_ossec_agent_keys
      start_ossec_agents
    end
  
  end
  
      



• modify the above code to match your environment (see below):

  # VARIABLES
  set :ossec_version,     "2.4.1" <---  put the version you downloaded here
  
  # ROLES
  role :new_agents,
  'xxx.xxx.xxx.xxx', <--- list the IPs of the hosts you want the ossec agent installed on
  'xxx.xxx.xxx.xxx',
  'xxx.xxx.xxx.xxx'  <--- no comma after listing the last IP
      



• in your homedir (or wherever you have the ossec source, preloaded-vars.conf, & the capfile), execute the following to initiate scripted agent installation:
  [alice@server] cap new_agents:do_all

You should see lots of output and if the agents installed successfully then the tail end looks something like:

 ** [out :: 10.10.10.10] Starting OSSEC HIDS v2.4.1 (by Trend Micro Inc.)...
 ** [out :: 10.10.10.10] Started ossec-execd...
 ** [out :: 10.10.10.10] Started ossec-agentd...
 ** [out :: 10.10.10.10] Started ossec-logcollector...
 ** [out :: 10.10.10.10] Started ossec-syscheckd...
 ** [out :: 10.10.10.11] Starting OSSEC HIDS v2.4.1 (by Trend Micro Inc.)...
 ** [out :: 10.10.10.11] Started ossec-execd...
 ** [out :: 10.10.10.11] Started ossec-agentd...
 ** [out :: 10.10.10.11] Started ossec-logcollector...
 ** [out :: 10.10.10.11] Started ossec-syscheckd...
 ** [out :: 10.10.10.10] Completed.
 ** [out :: 10.10.10.11] Completed.
    command finished

bonus: here is a cap task to install syslog-ng from source minus the syslog-ng.conf file (you need to create that)

# =================================================================================================
# Automatically install syslog-ng from source
# =================================================================================================
task :install_syslog_ng, :roles => :new_agents do
  desc <<-DESC
  Disabling the default syslogd process on startup and stopping the current syslogd process
  DESC
  sudo "/etc/init.d/syslog stop"
  sudo "/sbin/chkconfig syslog off" 
  sudo "yum -y install gcc libnet" 
  run "wget https://www.balabit.com/downloads/files/syslog-ng/sources/2.1.4/source/syslog-ng_2.1.4.tar.gz --no-check-certificate" 
  run "wget https://www.balabit.com/downloads/files/eventlog/0.2/eventlog_0.2.9.tar.gz --no-check-certificate" 
  run "wget https://www.balabit.com/downloads/files/libol/0.3/libol-0.3.18.tar.gz --no-check-certificate" 
  run "\\rm -rf ./eventlog-0.2.9/" 
  run "tar zxf eventlog_0.2.9.tar.gz" 
  run "cd ./eventlog-0.2.9/;./configure" 
  run "cd ./eventlog-0.2.9/;make" 
  run "cd ./eventlog-0.2.9/;sudo make install" 
  run "\\rm -rf ./libol-0.3.18/" 
  run "tar zxf libol-0.3.18.tar.gz" 
  run "cd ./libol-0.3.18/;./configure" 
  run "cd ./libol-0.3.18/;make" 
  run "cd ./libol-0.3.18/;sudo make install" 
  run "\\rm -rf ./syslog-ng_2.1.4/" 
  run "tar zxf syslog-ng_2.1.4.tar.gz" 
  run "export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/;cd ./syslog-ng-2.1.4/;./configure" 
  run "cd ./syslog-ng-2.1.4/;make" 
  run "cd ./syslog-ng-2.1.4/;sudo make install" 
  desc <<-DESC
  creating the directory home to the syslog-ng.conf file
  DESC
  sudo "mkdir -p /usr/local/etc/syslog-ng" 
end

A client this month was observing strange FORM submittal activity and asked if I was still doing security testing of their web application.

Since I was not, I participated in investigating. I received a copy of their Apache log file.

I downloaded apache-scalp:

“Scalp! is a log analyzer for the Apache web server that aims to look for security problems. The main idea is to look through huge log files and extract the possible attacks that have been sent through HTTP/GET”

From viewing the output of Scalp! it was obvious an automated application scanner was being used. Three bits struck my curiosity:

• The scanner was one I hadn’t heard of: netsparker
• Netsparker is commercial, implying someone was willing to pay money to find vulnerabilities
• The source of the scans was Microsoft

My client was dumbfounded why MS was doing this. I suggested they send an email to abuse@microsoft.com with the details and request their assistance.

Lo and behold, Microsoft replied and was responsible. Their Online Services Security and Compliance (OSSC) team was actively scanning, unbeknownst to my client .

Given this client has a relationship with MS, my guess is they signed something with fine print explicitly allowing MS to perform ad-hoc vulnerability assessments.

Good to know this can happen.

It’s been some time, but I just had one those scope calls from out of left field. University, mid size, private, budget sensitive, one security engineer and he is relatively new.

He is unsure of the state of his domain.

Me: Have any security assessments ever been performed?
Client: No.
Me: pause Do you trust the integrity of your servers?
Client: No.
Me: pause You are processing credit cards?
Client: Yes.

Calls like this one ensue brain freeze in me. I get so hung up on them I mumble for a bit until I catch a clear thought.

On cue, this question soon followed: “I was thinking of starting with vulnerability assessments, but where do you recommend I start?”

Is this not a scenario security aficionados are to love?

I dread them now. I’m being tasked with teaching everything a functioning adult is to know on a phone call. Don’t get me wrong, I do love the flood of ideas this scenario stimulates, but I’m weary of working for clients so far behind in the game. My experience with like clients is they come to the play with little budget, support, and with even fewer resources.

But in the spirit of the holidays, I’m going to draft a list of tasks I’d recommend he do and I’ll share it here.

A recent web application I was testing was generating a unique hidden token on every render of the login page (revealed in the pink section in the screenshot below by WebScarab).

I had setup a BURP intruder run to iterate through a list of attack strings when I noticed a few of the responses contained a message displaying “unable to verify session”.

That is when I checked for a hidden value, and when discovered, guessed it was a dynamic key identifying each individual form submit.

The fuzzers in WebScarab and BURP typically work by repeatedly POSTing to a web form, like the login form above, then capturing and/or analyzing each response. Commercial scanners do the same. Repeatedly POSTing fails to work in this case.

Screenshot showing BURP’s intruder and that I want to run a sniper attack (i.e. send one attack string at this position (red text below) per POST request).

Screenshot showing a few custom attack strings loaded into BURP

Screenshot showing the execution and results of an intruder attack (i.e. looping through the attack string list and submitting each via a POST request)

Screenshot showing WebScarab’s fuzzer setup.

The problem in this scenario is the hidden token is only good for one POST request. Maybe I missed it, but I didn’t see an easy way to tell WebScarab’s fuzzer or BURP’s intruder to first perform a GET request (so I can capture a valid one time token value) then submit a POST request. A few lines of Ruby along with the Mechanize library made for a simple work-around.

#!/usr/bin/ruby
require 'rubygems'
require 'mechanize'

agent = WWW::Mechanize.new
agent.set_proxy('localhost', '8008')

File.open("fuzzlist.txt") do |file|
  file.each_line {|fuzz_string| 
    agent.get('http://target/signin') do |page|
      form = page.forms.first 
      form.field_with(:name => 'data[user][email]').value = fuzz_string
      form.field_with(:name => 'data[user][password]').value = fuzz_string
      form.click_button
    end
  }
end

I configured mechanize to use my preferred proxy (BURP). The script loops through the fuzzlist.txt file, performs a GET request (now we have a valid token), then submits the form. Problem solved.

I just saw that Thawte is ending their Web of Trust email keys program. I think they should have sold directory services in addition to providing keys, perhaps I'll write about that next. It got me thinking about SSL/TLS. I've always felt that the openssl program does a lot more than most people know to do with it. 

For example:

openssl dst -sha1 yourfile

is about 20% faster than "sha1sum yourfile" on my OpenSuse 10.1 AMD Opteron system. That might not be terribly interesting but 20% adds up on bigger files, like DVD images.

 Another thing is in this world of automatic updates and appstores, it's more important than ever to have a verification mechanism in place. OpenSSL provide all of the tools for some simple, yet strong, verification. A lot of people I know are confused by certificates and don't fully understand the different formats and what you can do with them but OpenSSL provides the raw basics to simply perform signing and verification.

Generate a 2048bit RSA key:

openssl genrsa -out rsapriv.pem 2048

Extract the public portion:

openssl rsa -in rsapriv.pem -pubout -out rsapub.pem

Sign a file and store the signature in testsig.bin:

openssl sha1 -sign rsapriv.pem -out testsig.bin yourfile

To verify:

openssl sha1 -verify rsapub.pem -signature testsig.bin yourfile

There you go, signing and verification without any of the "complicated" PKCS stuff. It's only marginally more work to actually add a CA to the mix.

For me, it’s hard to downplay my joy from successfully assailing a web application and finding valuable faults: more so when nothing of value is reported from scanning using the most expensive app scanners.

On a recent gig I was focusing on finding issues with a couple of forms. Wielding BURP, I was intercepting all the GETs and POSTs and noticed the server responding with JSON data (i.e. http://[…]/data.json)

I tossed the JSON blob into an online JSON validator (http://www.jsonlint.com/) to make it easier to read (below is a small snippet).

"sanitized": 
  { “aaaDetail": 
    { "aa_id": "276060", 
      "bb_id": "103065", 
      "cc_id": "515ad8933c821b2f72a8cbb7054zed3c", 
      "x_time": "2009-08-12 21:42:40",
      "y_time": "2009-08-12 21:47:34",
      "full_name": "T H", 
      "comment": null, 
      "country": "United States", 
      "state": "Colorado",
       "city": "Broomfield"
       [...]
     }
  } 

I soon realized the JSON objects contained great information for crafting precision attacks.

The JSON data reads like they are database column values. I wondered for a moment what would happen if I just use the strings in the JSON objects for POST keys, came up with logical values matching the keys, then started POSTing the modified form data.

Bingo. By simply intercepting POST requests and appending new key/value pairs to the list of key/value pairs already being submitted I was able to modify database values unattended by the developers. In this case, it meant I was able to modify information inserted and owned by other users of the site.

I’m not espousing Harry Potter skills here, rather illustrating one of many examples of why Man is needed to go beyond where automated web app scanners stop.

Why I am proud to admit that I will never talk to any police officer. In Praise of the Fifth Amendment Right to Not Be a Witness Against Yourself.

I recently completed a pen test from the sole position of possessing remote administrative privileges to a few guest VMs.

I fired up yersinia to learn if launching layer 2 attacks could disrupt normal operations. While doing that I grabbed a copy of this book:
VMware vSphere and Virtual Infrastructure Security: Securing the Virtual Environment

vSwitches are reportedly immune to layer 2 attacks.

"Currently, a vSwitch will protect you from the following types of attacks by not providing the underlying functionality that these attacks require."

• MAC Flooding
• Double Encapsulation attacks (multiple 801.q envelopes)
• Multicast Brute Force Attacks
• Spanning Tree Attacks
• Random Frame Attacks
• DTP/VTP

I was able to perform MiTM via ARP cache poisoning on other visible guest VMs sharing the same network segment (because the attack targets the guest VM and not the vSwitch) – but in this case, each segment was allocated to a unique customer and therefore not a valuable test (i.e. poisoning one’s self was not in scope).

You can configure VMware to further limit attacks within a broadcast domain:

• Security | Promiscuous Mode (default is to reject) : Reject
• Security | MAC Address Changes (default is to accept): Reject
• Security | Forged Transmits (allow outbound frame w/a source MAC that is different) (default is to accept) : Reject

I just completed an external pen test whereby the rules of engagement limited the scan windows to two hours per night.  Requests for longer were rejected.

I hadn’t run within this tight of windows in some time and now I remember why I hate it so much.

I spent more time jacking any and every configuration setting I could tweak to boost each tool for balls out speed and baby-sitting (because failing seems to be a popular thing to do if you’re a tool sprinting at 50 threads and spending 0ms between requests) that I didn’t get nearly the time I wanted to concentrate on what I was paid to do: bust in.   

As a case in point I was working a SQLi point that was allowing me to download their entire database, alas, I only ever retrieved four of the 200+ tables during any one window.  Worse is I spun my wheels for several critical hours exerting fervent trial and error effort tweaking tool options, largely in vain, in hopes of making things go faster.  The consequence was tool tweaking dominated my attention.  Creativity, the force summoned for powersploiting, remained unconscious.    

• On small startup type teams, the team is everything, a bad team member can kill the product and the company.
• On particularly agile teams, having agile people is better than having technology experts.

Defending boils down to skill and incentive.

Tools that provide visibility are required, but that’s another topic. Skill is also an obvious requisite.

How about incentives? What incentives are in place for security engineers to really dig in and be great defenders?

Tier one engineers working out of traditional MSSPs are paid in the $20 per hour range or lower and by the nature of their position they have minimal understanding of anything of substance about their clients’ networks.

Opposite that, I know a very talented group of engineers working at an expensive outsourced IT/software company whom are responsible for their company’s top paying customers. They breathe uptime. Heads roll and the company loses money when downtime occurs. Security is barely an afterthought. Lest they do see a security issue, they may skip notifying for why create more support tickets and work.

It doesn’t make sense to punish defenders for failing to prevent infiltration – that is to be expected today. Simply detecting one is a great accomplishment.

To move defenders to be great defenders, reward them for detecting infiltrations.

I think it would be great for company’s to hire an outside party to perform unauthorized activity at an increasing pace and breadth until someone responsible for monitoring sounds the alarm. Reward those who discovered the activity. Do it frequently. Change it up. Make a competition out of it. No doubt this would help weed out bad performers, be they internal or external.

This is very similar to what I’ve seen a few hospitals implement. If an employee stops and challenges a person without a badge then they receive a $100 bonus.

I’m not promoting pen. testing here, though it’s a good example of a challenge.  I think simple and frequent small scale tests tied to rewards would work wonders for many security groups and for the company’s wanting to keep their assets protected.

I ran into a technical logging obstacle and I finally passed it today.

Setup:

• A large number of linux servers deployed over disparate locations and connected via consumer cable and xDSL lines
• All of them are configured to forward logs to a remote central log server via syslog-ng and UDP (client prefers UDP over TCP)

Problem:

• We were noticing a growing percentage would stop forwarding logs to the central log server

Troubleshooting notes:

• Timing seemed random; many clients forwarded logs successfully for weeks or months, some clients would stop forwarding after a few weeks, a few would stop in a couple of days
• All other connectivity appeared fine when troubleshooting (e.g. ssh worked, http worked, icmp, etc.)
• On each server, syslog-ng was running, logging locally fine, and showed no error conditions (but we observed no outbound traffic when executing tcpdump port 514)
• We did notice if we restart syslog-ng, forwarding would work again and we would immediately observe new logs appearing on the central log server

Quick thoughts and things we tried:

• Okay, configs are likely good because all works when restarting
• Network issues? But then why does restarting syslog-ng have an immediate and somewhat lasting effect?
• Alright, let’s upgrade to the lastest sylog-ng 2.x suite (libol, eventlog, syslog-ng) because maybe we’re running into a weird bug

Finali:

• After upgrading a dozen or so and waiting a week, we noticed two or three stopped forwarding logs again
• We executed tcpdump port 514 and again observed no outbound traffic
• We restarted syslog-ng and as before, observed new logs appearing on the central log server. This time it caught our attention that our tcpdump window still showed no outbound traffic even after the syslog-ng restart. Ah, now we’ve identified the information we were neglecting to consider: multiple interfaces.

[tt@00-e0-4d-8d-ce-3b ~]$ /sbin/ifconfig
eth0 Link encap:Ethernet HWaddr
[]
tun0 Link encap:UNSPEC HWaddr
[]


We executed tcpdump -i tun0 port 514 on a server for which we had not observed log messages on the central log server in some time. By watching tun0, we now observed outbound UDP packets, but the central log server was still not receiving them. When we restarted syslog-ng on this server, we now noticed a significant change in the tcpdump output. The first three lines below were observed when the central syslog server was not receiving messages and the second three lines are after we restarted syslog-ng.

15:57:22.009114 IP 01-01-01-01.att-inc.com.40655 > 555.55.253.145.syslog: SYSLOG authpriv.notice, length: 132
15:57:22.187959 IP 01-01-01-01.att-inc.com.40655 > 555.55.253.145.syslog: SYSLOG syslog.info, length: 104
15:57:22.188023 IP 01-01-01-01.att-inc.com.40655 > 555.55.253.145.syslog: SYSLOG syslog.notice, length: 97
15:57:22.699615 IP 777-777-229-216.static.data393.net.60408 > 555.55.253.145.syslog: SYSLOG kernel.notice, length: 249
15:57:22.700051 IP 777-777-229-216.static.data393.net.60408 > 555.55.253.145.syslog: SYSLOG kernel.notice, length: 254
15:57:22.700108 IP 777-777-229-216.static.data393.net.60408 > 555.55.253.145.syslog: SYSLOG kernel.notice, length: 246

Problem discovered (some of the IP information above was changed for obvious reasons).

On reboot or startup, tun0 is brought up and syslog-ng forwarded logs with the correct source IP for this environment (the one assigned to tun0). The catch is when an openvpn connection would bounce on a live system (i.e. tun0 would go down), syslog-ng would fall back to using the source IP assigned to eth0 and continued to use that IP even after tun0 was brought back up, which broke the forwarding of logs in this environment. Syslog-ng required a restart after tun0 would go down in order to use the proper IP (as observed in the tcpdump output for tun0 above). The flapping openvpn connection was the culprit.

Try inotify.

Inotify is a Linux kernel feature that monitors file systems and immediately alerts an attentive application to relevant events, such as a delete, read, write, and even an unmount operation.

The Inotify-tools library provides a pair of command-line utilities to monitor file system activity:

• inotifywait simply blocks to wait for inotify events. You can monitor any set of files and directories and monitor an entire directory tree (a directory, its subdirectories, its sub-subdirectories, and so on). Use inotifywait in shell scripts.
• inotifywatch collects statistics about the watched file system, including how many times each inotify event occurred.

We just need to check two events:

• # IN_CREATE - File/directory created in watched directory
• # IN_MODIFY - File was modified

#!/usr/bin/ruby
require 'inotify'
require 'find'

raise("Specify a directory") if !ARGV[0]
directory = ARGV[0]

i = Inotify.new

t = Thread.new do
      i.each_event do |event|
        File.open(directory + "/#{event.name}") do |f|
          f.grep( /\b(?:\d[ -]*){13,16}\b/) do |line|
            puts "Detected credit card pattern in directory #{directory}, file #{event.name}\n"
          end
        end
      end
end

Find.find(directory) do |e|
  begin
    i.add_watch(e, Inotify::CREATE | Inotify::MODIFY)
  rescue
    puts "Skipping #{directory}: #{$!}"
  end
end

t.join

Not too bad for an hour of playing around. It works.

I used the ruby inotify version 0.0.2 from http://raa.ruby-lang.org/project/ruby-inotify/, but if you do that, then you need to fix line 47 - change it to

r = rb_thread_select (fd+1, &rfds, NULL, NULL, NULL);

• credit card number regex found here: http://www.regular-expressions.info/creditcard.html
• inotify-tools: http://inotify-tools.sourceforge.net/
• ibm inotify tutorial: http://www.ibm.com/developerworks/linux/library/l-ubuntu-inotify/index.html

removed for the time being...