ClearNet Security

edit

How not to build a server

Posted by Ian S. Nelson Mon, 28 Jan 2008 17:18:00 GMT

A few years ago I was in a job I wasn't particularly fond of. One of the things that I disliked was that it felt like we have a bad heuristic for deciding when to roll our own verison of something and when to use something off the shelf. This is critical for 21st century business, it's laughable the things companies did as experiments last century, the IBM PC was arguably an experiment that IBM was never fully committed, it was never their core competency...

This company did things completely backwards. They'd homebrew things like Linux platforms (or if you were marketing the effort you could call it "hardening" which it really wasn't) and then for the core technology of the products we'd take off the shelf products and try to brand it our own and put our own little spin on it.

One particular episode really stands out to me, we were migrating our offices and we had to start running our own mail server. The old office set us up with an active directory, you'd think we'd just drop in exchange, flip on pop, imap and webmail and call it done. Instead a multi-week process that involved "hardening" a Redhat linux box was undertaken by our Security Architect. Part of this process was default replacing bash with c-shell, for that old skool BSD-like flavor, it puzzled me because in the 2.5 years there I never once saw a c-shell script, most of that stuff was done in perl which sort of makes the shell irrelevant. The thing with c-shell is that nobody but sickos will want to use the system should it ever be compromised so there might be some security value but... Some of the other hardening procedures included system scripts that deleted history and log files on logouts, some odd changes to the default Redhat "lokkit" firewall and then a fairly bizarre process of mirroring authentication from the active directory. I'll be honest, I don't know how it really worked but every account was duplicated and you could actually log in with your AD password. When the project was announced as "finished" we had a real email server that you could get email at and it used your windows password.

It also had a number of interesting side effects, like the fact that I could simply ssh in to it. There was a host based IDS that did record my ssh attempts but every user could ssh in to it, it was inside a fairly friendly environment so that's not terrible but most mail servers you'd get off the shelf wouldn't allow just anyone to log in. Another side effect was that our Windows authentication hashes were all NTLM which made them a bit easier to crack with l0pht crack, this is probably still lost on them, it wasn't clear to me when I left that they'd ever recognize the significance of this or recognize it at all. Then maybe the best side-effect of all was very nature of the mail server and how the custom chopper had been built, it was pure spool files and sendmail (or maybe it was postfix, I don't remember.) One day we stumbled on to this gem: ls -lh /var/spool/mail

...
-rw------- 1 cory users     5.8M 2005-01-24 15:33 cory
-rw------- 1 ian  users     3.4M 2005-01-24 15:33 ian
-rw------- 1 root root      2.8M 2005-01-24 03:31 root
-rw-r--r-- 1 someguy users 10.2M 2005-01-24 15:33 someguy
-rw------- 1 tate users     3.5M 2005-01-24 15:33 tate
...

That's just a recreation, names have been changed. At a glance you might not notice anything. Turns out that "someguy" must have manually edited his spool file with the wrong umask, also turns out that "someguy" was the Security Architect and the initiator of this experiment. His spool file was world readable for months and months. When I first saw it, I immediately read the file with "cat." I deleted my history file and logged out and back in to check if I had left any obvious tracks. Turns out that the system automatically did the same thing for reasons I to this day don't understand, someguy added that logic to the various .exit system files. I waited a couple days for some fallout to see if anything happened, thinking I might have tripped an alarm or left a gory log message but nothing happened and nothing changed so I started downloading "someguy's" email on a daily basis. To my chagrin I learned of another defect.

I could run this command:

ssh ian@mailserver "cat /var/spool/mail/someguy" > someguys.email.dump

and it wasn't even recording a "login" on the HIDS that was running, further I know this because I was reading the emails from the HIDS in "someguy's" email. I shouldn't confess this, it was unethical, but this went on for months. I got to read about who he wanted fired, salaries, messages from his wife about his dog getting sick from eating too many treats, and private messages to his friends about how he hated his job there, messages about how he thought he should be the next CTO and some other guy got the job; there was some good stuff and I could pile on but I won't. I had the whole thing streamlined, I'd dump the file, zip it up and send it to myself at home where I could read it in safety and comfort while drinking a cocktail.

Lessons?

• What's funniest to me about this was the amount of effort that went in to being "non-Microsoft," a fresh exchange server would have cost much less man time (and therefore money,) been maintainable, easily outsource-ready and more secure than this "secure" hand rolled linux system. To a degree, the same could be said about simply slamming Suse or Redhat on a box and installing Cyrus or dbmail.
• What's also funny is that just about any out of the box Linux without this "hardening" would have been more secure too, I'd have left tracks and marks and various events would have been logged.
• None of this had anything to do with what the company was in place to actually do. We could have used hotmail accounts if we had to and that would have been more secure.
• There was no thought about when you should be on "main street" and when you should "roll your own" or build a "custom chopper." If there was some clear reason to hand build it then it should be done but there was no competitive advantage or any really compelling reason to do it.
• We all tend to go to our strengths when we are given a challenge we may not know the solution for. There is something to be said for actually knowing your strengths. "Someguy" liked to act a BSD guy and liked to be a "security expert" and this project was one of the more clear demonstrations of how little of each he is/was. You have to know what your strengths are and you have to know if they are actually strengths. I can't help but think that most "BSD guys" would have realized how janky the whole thing was long before it was deployed.
• Lastly, watch what you say in email, especially at work where you use computers and networks that don't belong to you. Even security experts botch it from time to time.