ClearNet Security: Tag testing

Us and them development

ian@ClearNetSec.com (Ian S. Nelson) — Fri, 14 Mar 2008 10:57:00 -0600

This actually relates to security. It's round about but it'll tie together.

I was speaking with a college buddy the other day, a guy I respect the hell out of who happens to be an extremely senior engineer and now a fairly wealthy man due to applying that skill in some successful companies. I can't remember the exact thing I said but I was chatting about work, sort of complaining or venting and he called me out for speaking negatively about my current company's QA. I was basically saying that they don't seem to try very hard and he pretty much asked me why they ever would if you treated them like inferiors?

I was a little taken back, I don't think I treat them that way, it's not a conscious thing but clearly there is a caste. I haven't had that many jobs but the 2 places where test was treated as equals and with the same respect as development they also had the same level of responsibility and we ultimately had much much better testing. It works better when everyone takes ownership of the whole product and it generally doesn't work that well when people try to just slice off little pieces and ignore the rest. Everywhere else, most other places, QA were second class citizens, it seems like a normal sort of way of operating. People can rise to the level of expectation, if that level is too low then that's what you'll get.

So how does this affect security? Network ops and marketing usually have very different missions. In the last 10 years, businesses have added security groups and security officers to “add security” to the business, and to basically fill in a hole that nobody else owned. It doesn't work very well. A CSO should be more like an ombudsman. If it's not everybody's responsibility and everybody's job then it simply won't work, there are no tools you can simply buy that will make your business “secure.”

Flip the reporting and get the customer to speak

tate@ClearNetSec.com (Tate Hansen) — Mon, 24 Sep 2007 14:24:00 -0600

I can’t resist rehashing this topic.

Why not, after performing a penetration test, ask the customer what they observed? And when doing so, using commensurate effort to clearly surface the importance and point from asking that question.

So many are ordering up vulnerability and penetration tests like it’s a quick immunization shot against being hacked. They may get a wee bit nauseous after viewing their vulnerability report, but once they mitigate and recover they feel invulnerable, however temporary.

Penetration testing proves nothing of invulnerability, and it naturally follows, ad nauseam, to ask, again, what does it really prove then? Marcus Ranum summarizes Gary McGraw’s answer to that:

Gary McGraw likes to refer to pen testing as the "badness-o-meter" - it's a test that registers, at one end of the dial "your network sucks" and at the other end, "we don't know."

If the “We don’t know” club doesn’t already list as members all penetration testers, it surely should. No one has the knowledge or possession of everything needed to break all systems, so everyone is playing their own part in the “don’t know”.

Even if you are named Mr. Exploiter and pay cash to subscribe to every 0day exploit factory, the value of showing your customer they get owned when you call an 0day blitz is not only from proving you can sprint past their defense. Anyone with an exploit can do that.

The value rises from the conversation you have with the customer after the party. Why should the customer care if the latest 0day worked? What was the customer to do about it anyway?

Companies whom take their security seriously need to report back to you. Instead of hearing the customer ask you, “What did you find?”, you should ask the customer, “What did you observe?”.

Imagine how shocked you’d be to listen to a customer detail all of your efforts during a penetration test: They report to you which systems you attacked, when and how, and what information you had obtained.

That would be good security. Penetration testing should move away from “I got you with this 0day” to “You identified 90% of my efforts to compromise your systems”.

Flipping the reporting paints a clearer picture of the overall security of your customer: Those that do well you could posit have good policies in place and have built respectable awareness and response capabilities.

Anything alert you?

tate@ClearNetSec.com (Tate Hansen) — Wed, 05 Sep 2007 19:37:00 -0600

There is nothing I’ve seen recently to promote a valuable exercise to do after receiving a security assessment. That is, as the client, what did you see?

Did you have anything alert you? If so, what did it suggest? Did you have enough information to piece together what was happening? (Bonus: do you know which tools were fired towards your IPs?)

The majority of my clients have no clue if anything occurred. That’s bad. Businesses which have little to lose may decide to ignore investing in monitoring and detection, but for others it’s turning a blind eye.

I’m going to dig a little deeper on future exit calls to get more information. I often ask clients if they detected any strange behavior, but there is definitely more room to expand the discussion.

PCI: Not our problem...

tate@ClearNetSec.com (Tate Hansen) — Wed, 16 May 2007 20:51:00 -0600

What happens when the test environment operated by MasterCard (they “own” the testing lab) is misbehaving? I know. They yank the wheel, swerve away from responsibility, and point to the PCI council. And PCI? They point back. Beautiful, no?

You see because they refuse to disclose missed results to you they duck responsibility for anything that may have been their fault. They also clearly imply if anything is missed in your attempts to identify vulnerabilities then it is surely your fault or a problem with the tools you used.

I love it: No clear pass criteria, no way to challenge a decision, and no transparency of what or how they are doing. For all this great service you get to spend thousands every year!

So what happens when you call bullshit and raise hell? They pass you. :) Let me not forget to mention we had a few extra bullets in our clip they may have unexpected us to have – bullets provided to us by friends with information.

Be forewarned; this process has serious issues.