/kevinbehr/home

Bleep Bleep

I had just finished almost ten solid minutes of uninterrupted work on my bosses’ pet project.  I had three ssh sessions and about 25 other windows going on my two monitors.

The phone caller-ID read Tom Lispon, an infosec Ninja, and a great friend of mine from Information Security.

“What’s the haps Tom?” I sighed into the phone.

“Dude, don’t sound so enthusiastic just for me.  I am just delivering the friendly pre-atom-bomb-clue-stick to you.”

“Oh, yeah?” I managed while putting the finishing touches on a script and saving it.

“Yeah, remember those old servers you found in the datacenter last week?”

As I secure copied the script to a server, su-d and ran it I remembered, Tom was speaking of two servers that had no visible identification and had multiple nics in them.

At the time, I thought they were some sort of experiment by the security team.  They certainly fit the mold of a skunk works project, made of retired server hardware and located in the back corner rack of row one of the old data center sans any physical asset tags and or labels of any kind. So naturally I queried my buddy Tom and acted like I was in the know just to spook him.

Turns out nobody owned these boxes and they had been here forever.

They were Microsoft boxes and still had their own domain setup on them, we had migrated to Active Directory eons ago so either these boxes were for some app that was not happy with AD or they were just missed during the migration.

“Yeah, I remember.  Did you ever figure out who they belonged to or even what they are doing in there?”

“Um, yeah, and boatloads more. So check this out, remember we couldn’t find them in any docs? Well turns out when we did that whole reverse merger years back they were the original Exchange and Outlook Web Access servers for the company we bought.”

I started to laugh one of those ‘this-is-not-really-funny-but-everyday-here-is-like-a-masters-program-in-how-not-to-manage-anything’ laughs, but then it hit me.

If the servers were from the company we bought in the reverse merger many of the accounts on it would have been from the now senior executives of our company. Holy crap, this was getting interesting.

“Tell me the boxes are dormant dude.  Please tell me.” I quietly prayed. 

The other reason I was sure that the boxes were a security project had to do with the fact the multiple network cards and their connections actually connected to several different networks.  I was pretty sure that one of the NICs plugged directly into a DMZ or non-firewalled VLAN.

“I wish I could, but they are still being used. Only by about a dozen accounts as far as I can tell”

“You found some creds to get in?”

“Yep, actually we didn’t need them.”

“What?  Was the console just logged in with no time out after like 4 years?”

“No, the boxes had all been owned for, like, many years, dude. They had reactivated and modified very handy backup-administrator accounts, removed the passwords, and gave them full administrator privileges”

This was getting worse by the minute.

“Looks like they used an old IIS hack and from there they just owned the box.  It’s got all kinds of tar files in a temp directory.  Including a nice rootkit and a bunch of tools to explore the inside network from the servers. Oh yeah we found some old CDC stuff and a old Back-Orifice app listening for connections.”

God, the thought of someone opening and closing the CD-ROM drive of the old servers in our datacenter, not to mention all of the data… Oh man.

“Well at least not many users are on the box, right Tom?

“True, but it is the entire executive management team minus like two of our original execs.  But that list includes the CEO, CFO, CMO, COO, Chief legal counsel, and several VPs.  This is not pretty my friend.”

“Indeed.  Have you told anyone yet?” I asked.

“I had to.  I scheduled a quick meeting with our new interim CIO.  He was less than pleased.  I guess he has been on the job all of about four days now.  Poor guy, I actually felt bad for him, unlike the last guy who let this happen on his watch.”

“Yeah, I hear you.”  I chimed in.

I could count the CIOs I had met or even read about that understood security let alone operations with just a couple of fingers.

“So you wanna hear the real kicker we found?” Tom baited.

“Probably not, but go ahead.  Tell me.”

“The Outlook Web App box had a capture app intercepting the usernames and passwords as everyone was logging in.”

“Oh man they were harvesting creds from that box.  Good thing we have a policy mandating different creds than your internal ones for external facing web apps like that”

“Well… not back then we didn’t.  All of the account names and passwords were owned for years. When you consider that people usually recycle three different passwords for mandatory passwords over the course of a year they have a good chunk of our execs creds. We are still trying to figure out what you can see from these boxes.  Even though they are not part of the Active Directory they have access to a lot of stuff via legacy domain trust relationships with other old stuff. ”

This whole scenario was getting out of control.  There would be no way to put a lid on this, as we would need to interrupt the senior exec’s email to resolve the problem.

I rounded the corner  adjacent to the break room and overheard what started as a distant, dull roar and bloomed into a full on verbal cacophony as I peered inside the door.  My nose immediately detected the smell of burnt coffee and some sort of Indian food in the microwave.  It was not an appealing combination.  The noise was emanating from one our helpdesk managers, a guy that I recognized from an audit, and two folks from information security. 

I could not make heads or tails of the conversation except that the auditors wanted to know why there were so many helpdesk workers with the ability to grant administrative privileges to IT staffers without a clearly defined approval and decommissioning process. Apparently there were more exceptions to the policy than actual IT workers.  Man, I did not want any part of this.  If only they knew how many shared admin accounts I had with the creds tucked away in my cranium, they would all have heart palpitations so bad they wouldn’t even be able to steady their little hands to write an urgent doomsday opinion letter to management.

I decided I would actually rather get robo-coffee from the vending machine than even walk through that fray so I backtracked and headed downstairs to the vending room.

As I approached the row of machines I spied Dick Majors arguing with the candy machine.  Dick was our VP of Marketing and a major pain in our collective IT posterior.

“Hey you’re an IT-guy maybe you can make this thing give me the Baby Ruth I paid for”, he vented.

“Well let’s just say we have both met our match.  I call that machine the bully. It has stolen more lunch money from me than any actual bully I ever knew. You are pretty much screwed, I’d say.”

“Hey, so how are you guys coming on the new web property launch for corporate? I keep getting told there are all kinds of technical issues and they are being blamed on network and security engineering.  What’s your take?”

*Quagmire alert*

My internal klaxons are ringing at 120db and my prefrontal cortex just went to maximum self control status to prevent a major career-threatening verbal miscue of stupendous proportions.

“You got me on that one sir.  I just found out about it two days ago when the developers went rogue and somehow overran data center security and were actually installing apps on servers without any change approval or ops guys in the loop.  The only reason I know that the roll out was happening is because they called my desk when they ran in to problems.”

“Really?” He shrieked.

“Your boss said he had you clear the way for them and that you had set all of the firewalls and other network gear up for them over a week ago and that the servers were done three days ago. Are you telling me this is not the case?”

Crap. My prefrontal cortex had just panicked necessitating a cranial reboot.  This was not the time to tax my caffeine deprived brain.

I managed

“Well the network gear was set up, although I did not realize it was for this project.  I still have the servers to complete. They should be easy as soon as the dev and staging team tell me what they need them to do.”

In the back of my head a growing sense of foreboding met up with a wave of nausea as I realized there was no way the launch would work.  No one had ever told me what I was setting the gear up for.  All I had done was set up interfaces, vlans, routing, and trunking.  There were no rules permitting the traffic to and from the various segments the dev guys had mentioned. I had to get out of here…… and fast.

Knock Knock.. 

Sure enough, my door is duly darkened by a salt-and-pepper-haired VP.

“So I assume you got those VMs built for me and I need to get apps installed?”

“I guess you didn’t get my email?” I queried.

“Nope, why?” he sighed and slumped against my door frame.

“Well I got called in to fix a crap storm that is still raging pretty strong and I haven’t even had a chance to bug the dev and staging team to get the specs for the servers yet.  Once I have that I can create the VMs in a heart- beat.  But the way things have been going I have no idea when that will get done.”

He looked both ways outside of the door then ducked inside my office and whispered, “But, you did get those firewall and switch config changes done yesterday right?”

I nodded. “Yes, I had to actually do them myself because I couldn’t find any of the network ops guys that I know and I wasn’t going to go through the hell of submitting all of the paperwork just to have three hundred questions asked about setting up some vlans and trunking. Oh I did have to push some new firmware code to the switches as they were like three revs back and there were some known issues with dropping trunks intermittently so I had to do that first.”

“Good, I assumed you got them going, so yesterday I had my team start the code push for the project.”  He turned to leave then turned back like Colombo. “Let me know when those servers are built.  They are a huge priority!” He then headed off to find his project team.

Just as he walked off my phone started ringing. This time I looked at the caller ID before picking it up.  Sure enough it read Datacenter-02.  Quickly I checked my trouble ticket queue to see if I could be busy with something else.  Sadly there were no hyper urgent tickets. 

I picked the phone up on the fourth ring.

“Hello”, I managed.

“Hey, this is Dave from the launch team again.  We are still having the same issues as yesterday and we have tried everything you told us to and nothing is working.  We flagged down one of the new security dudes in the datacenter and he is running Nessus to figure out if something is configured wrong.  Do you guys have a sniffer?  The security guy thinks we need one. ”

My head is struggling to find, let alone follow, a logic trail here.  What the heck is an infosec  guy doing running Nessus?  Why do we need a sniffer?  I am not even sure I understand what part of our network these guys are even talking about.  We have over 1000 web properties.

“What network and which web properties are you talking about?” I ask.

“We are in the Cleveland datacenter and we are working on the new web cluster 40, app cluster 4, and db clusters 4 and 6.  We can’t get anything to see anything else.  The only thing we can see is the new web servers from the outside.”

Who put in the new web servers?  I quickly checked the change schedule to see what was supposed to be rolling and saw nothing about new web servers.  But of course that means absolutely nothing. This whole place has gone mad max since we outsourced our data center operations.  Random bands of paramilitary raiders traversing the aisles in their rolling carts loaded with guns and USB drives full of mutant half baked projects.  Every department had some under-the-radar project going and it looked like a pirate-con in certain datacenters on Monday mornings.

Something about datacenter-02 was sticking in the back of my head.  I had configured those switches there the other day but there it would be inconceivable that those firewall interfaces and the new switches could have anything to do with these new web servers and their “issues”.

“Let me go talk with some of the other network engineering team and see if I can find anyone who knows about this.  How can I reach you?”

“Well cell phones don’t really work in here so call back on this extension.  I thinks it’s 2207.  I will be right here waiting.  Hopefully that veep gives us some space to get this figured out.  He was down ranting about some huge print spend that is on the line if we miss the launch date again.”

“OK, I will ring you back as soon as I can figure something out” I reply.

As I set the phone back in its cradle I muttered to myself  “It is far too early for this kind of crap. I need some freaking brain juice.”  I found my trusty mug and headed to the break room to fill it.

Many of us have been deep in concentration on some task or deliverable due in mere hours and then, ring ring, the phone goes off.

 ”Hello,” I mumble in my best Marvin-the depressed-robot-voice as I pick up the phone like it contains a toxic substance that I want to avoid contact with. I make sure to intone all of my passive aggressive sub-rage into my one word protest.  The white noise in the background betrays the location of the caller, as does the caller ID that reads datacenter-02.  Some developer gives me his credentials and reveals his stereotypical lack of social skills by blurting “Yeah, hey the app crashed and we think the network or firewall is down or something.”

My blood pressure immediately increases by a magnitude of order. 

“What is happening? Wait, who are you and what are you even saying?” I am answered only by the data center’s ghostly hvac voice.  “Hello?  You are not making any sense,” I reply a second time.

“I am sorry.  My name is David and I am in charge of the code push across all of the web properties and something took a nose dive, we think it’s the network, and caused our entire property to serve up 404s.  One of the dev team guys is sure it is the firewall while QA thinks it has something to do with the test and deployment environments not matching up.”

With my head in my hands, I have so many questions forming I fear I will need a DeWalt drill to relieve the ensuing pressure. I inhale slowly and breathe out of my mouth Buddha-style, in an effort to peace out my rage and avoid saying something I will regret. Finally, I manage a line of questioning with a subdued tone.

 ”OK. So can someone explain why we are rolling out property wide code at 2:00 pm eastern? Wait, don’t even answer that. Why are we not rolling out property wide code updates in small waves? Wait, I see no change ticket in the system for anything during this time window, or…ever.  This is not a small break in logic. What the heck is going on and how did developers get in to the datacenter anyway?  Am I in a bad dream?”

Before these people decided to hatch their half-baked scheme they called an “upgrade,” I was working on the most important thing ever.  At least that is what the VP of IT told me.  We were supposed to finish the configurations for all 12 of the new virtual servers and build run-books for them by yesterday.

Looks like another round of “Sir, I was interrupted by a massive web outage and wasn’t able to get through to the dev team to figure out what these servers are supposed to do yet” when Mr. VP darkens my door tomorrow.

It’s 1999 about seven o’clock on a rainy evening and Gene Kim and I are sitting in the corner at Pazzos in Portland Oregon. By now the windows are fogging we are on our second or maybe fifth round of Widmer Hefewizens. Our voices are very animated and we are getting more excited by the minute. I remember feeling like I had met my mental twin.What were we talking about? Mostly about how screwed up IT was as a corporate function. Management made little sense at times. We traded IT disaster stories and even what we thought were some solutions. I parted feeling like I had made an important friend but not knowing where the friendship would lead.

Little did I know the significance of that evening in terms of the research projects, books and breakthroughs we would spearhead as a result of our friendship and collaboration over the next decade. We were put together by a mutual friend, Jon Speer. Jon felt like Gene and I spoke the same language and felt we needed to meet. Thanks Jon, you were dead on!

Over the course of several meetings we performed many brain and book dumps on each other. We both described the work we were doing for our respective CEOs and found a important vein to begin working on together.

In both of our day jobs we had customers that were stand outs because they behaved differently and spoke a different language. In my business these clients were more profitable and required less man hours to support. In Gene’s business these customers were using his software (Tripwire) for operations as opposed to security. We began to study what we suspected were high performing organizations and see what made them so radically different than the others.

Two years passed and we progressed from suspicions to deeply held beliefs and developed what is now known as the Visible Ops methodology. We found four areas of focus that separated these special teams from the rest. We normalized all of their self derived playbooks and terminology back to the common parlance and taxonomy of ITIL. We then laid out an easy-to-follow approach to stemming IT chaos and firefighting with four simple projects. After completion of the methodology we co-founded the IT Process Institute to continue our research work and teamed up with George Spafford to write the Visible Ops Guide.

Since the publishing of Visible Ops it has sold nearly one hundred thousand copies. The second book in the series “Visible Ops Security” is selling very well. Our empirical research and benchmarking has helped hundreds of IT organizations answer the questions of “where should we start?” when it comes to process improvement and ITIL adoption. We have isolated the 80/20 of IT controls and processes and know just what high performers do to get their amazing results.

Over the last decade Gene and I have spoken at dozens of conferences, hosted countless roundtables with CIOs, CSOs and Chief Audit Execs, consulted CIOs, VPs, taken interim executive roles, consulted large IT turn-around efforts, written dozens of articles and benchmarked hundreds of IT organizations. It has been an amazing ten years but I am most excited at what lies ahead. We have not only proven our deepest beliefs and confirmed our restless suspicions but we have done it with empirical research and science.

In the next blog entry I am going to examine a topic all too familiar to IT executives and staff. It is the “hidden sucking sound” present in all IT shops whether large or small. High Performing shops have much less of it and manage it very carefully. What am I talking about? Unplanned Work. You may not know it by this name but if you have worked in IT you will recognize this all too well.

till then-

kb