My definition (purpose) of IT Operations  ”The Operating System”

“An explicitly specified system of work that consists of a structured, controlled and measured approach to the continuous delivery of existing IT Services and Service Assets in order to achieve the goals of business.  The system accurately illustrates capacity, describes, manages and measures the flow of work in to operations via Service Transition and Standard Operations Requests and compares their actual resulting output to business expectations.

The Operations System collectively focuses on  preventive action first, detective controls that instrument variance from specified acceptable operating states, and lastly when operations are interrupted, specified structured  corrective measures that deliver root cause analysis with corresponding countermeasures to prevent reoccurrence are used to restore operation.”

I know you have operating systems in IT but do you have an IT Operations System? If you do, could it be better? Next post I will look in to the pillars of “The Operating System”

Kevin Behr

 “I wish we had dedicated project resources. I am so busy with operations that I just don’t have time for projects.”

“Why does every IT issue get escalated to my top network and security people?”

“I don’t care if you have enough time. I need this stuff done now.”

“It takes me more time to fill out a change request than to make the actual change”

“We spend over 70% of our time doing operations which only leaves me 10-15% to work on projects after I read my email”

Sound Familiar?

I love the provocative statement that Goldratt made (I paraphrase):

“Technology CAN (not does) provide value IF and only IF it diminishes a business constraint.”

Before you go off emailing me that technology has many other values please reflect deeply on this statement.  Please reflect deeply on what business value IT provides. 

I love the notion of continuity that the “diminishes” brings to the statement.  In other words the constraint must be continually diminished as opposed to the word vanquished.

In order for the constraint to be continually diminished the technology must operate without ceasing or the constraint is re-introduced and the business is forced to deal with the issue once more, usually without any warning.

One could argue that not all applications in service deliver the best business value. But for the purposes of this discussion on IT operations let us assume that everything we are running in our datacenters provides the business with some critical constraint-busting capability.

By doing everything we can to ensure that those systems continue to operate free of interruption we are performing IT operations that support business operations.

The basic definition of Operations is “The act of harvesting value from resources”.  More specific to IT I believe that IT operations represent our collective approach (strategy) and tactics (tasks, instructions, and programs) designed to prevent outages and interruptions to the IT services that existing business operations depend on.

Since the whole point of business operations at large is to produce profit (or achieve the mission for you non-profits) from its resources, when we perform IT operations successfully we are protecting business revenue generation.  I like to call it “protecting revenue” for short

You may notice that firefighting, support or outages did not appear in my definition or mission of operations.

This is very important.

When we are troubleshooting, or firefighting an outage, operations have ceased and we are now in recovery mode (attempting to recover operations).  Even if the issue is service impairment versus a full blown IT black-out we are attempting to recover from the situation and therefore are not in nominal operation. Both scenarios interrupt or affect business operations and can put revenue at risk in many ways.

So if you were really spending 70% of your time in operations..Do you think there would be so much chaos?

Next Post I will talk more about defining and measuring operations and the value it can provide. 

kb

I just received this email from my colleague Steve Spear whom I am working on several articles with at the moment.  Congrats Steve!  You deserve this award for some of the most profound work I can remember!

“Dear Friends and Colleagues,

I’m delighted that my book, Chasing the Rabbit: How Market Leaders Outdistance the Competition and What Great Companies Can Do to Catch Up and Win, has received welcome accolades in the last few days.  The book was awarded a Shingo Prize for Research Excellence, and it received a flattering appraisal in Harvard Business Review’s April issue.

The reviewer, Anand J. Raman, writes:

            Spear…has dazzled readers with his insights into what makes 

            Toyota tick and his understanding of how any organization can 

            use those ideas to improve its effectiveness. Not surprisingly, his 

            first tome was highly anticipated, and it’s probably an understatement 

            to say that it won’t disappoint.

He concludes…

            I have a dozen books on Toyota stacked on my shelf, in order from the 

            least read to the most referred to-and Chasing the Rabbit is probably 

            going to stay right on top of the pile.

Chasing the Rabbit is based on my research which was initially meant to answer the question: Why was Toyota doing so well despite (a) being in a hyper competitive market, (b) starting well behind its rivals, and (c) having been studied and imitated intensely?  

The answer was that for all the attention that outsiders had paid to particular production control tools, the company’s real genius was its management system that fostered and sustained high velocity, high endurance improvement, innovation, and invention across a broad range of work. I later found other organizations that had arrived at similar approaches, and with the generous help of myriad practitioners showed that this ‘high velocity’ approach has great impact across a broad range of situations.       

Based on this research, Chasing the Rabbit explains how competitive advantage can be generated in even the most arduous markets and illustrates its points with diverse examples from heavy and high tech manufacturing, new product development and production, commercial and military situations, and health care.

I’m much indebted to those who helped advance this work over many years and it is my hope that their efforts and mine prove to be useful to you and your colleagues as you attempt to generate far more value with far less effort than most in your fields even imagine possible.

With best wishes,

Steve Spear

Senior Lecturer, Massachusetts Institute of Technology

Senior Fellow, Institute for Healthcare Improvement

Please visit my blog: http://chasingtherabbitbook.com “

I highly recommend this book and keep an eye out for our articles on IT and healthcare!

This article angered me for several reasons.  To imply that IT has gone too far with process standardization is wholly inaccurate IMHO.  To write the article with zero actionable take aways is very annoying.  To do all of the above based on conjecture is irresponsible.  

I often take issue with the IT press’ assumptions regarding the current state of  IT management. But this is beyond issue for me.  This kind of thinking does nothing to foster the type of intentional system design that is so desperately needed in today’s IT organizations.

The symptom of one size fits all process adtopion is just that, a symptom of poor management.  To suggest that the solution for poor management is somehow to fuzzy the lines around process and end the mellow harshing, creativity killing approach of process standardization is just hollow.

Building effective high performing systems is focused 100% on achieving company goals and if that means different approaches for different regions so be it.  Why is this so hard to see?  

Besides is this really a conversation you could imagine having with your CEO?

I just pinged Steven Spear about this and want to write a response that draws attention to the pressing issues facing IT, based on over 10 years of research and empirical data.  

/end rant

This morning I was running 45 minutes behind my usual schedule.  I tried to reassure myself that I was still in over an hour before any of my staff, except for Rob my Deputy CIO.

As I walked towards my office I saw someone chatting with Mariah at her desk in front of my office.

I walked directly to my door and stepped inside my office to divest of my coat and briefcase.  I emerged an headed straight for the coffee cache in the back of Mariah’s cubicle.

Mariah was trying to schedule a meeting with the gentlemen from what I could overhear.   

“Well the first available slot is tomorrow at 11am.  Phil only has a half-hour available.  Otherwise I can get you one hour next week.  What will it be?” Mariah queried the young IT staffer.

“sheesh.  Well this is extremely important and confidential.  It is an information security issue that must be addressed and I need Mr. Chairs to be up to speed before anything is done.”

I decided that I had heard enough.  I walked around the front of the desk and set my coffee down on the return.

I turned to address the staffer with a warm look and extended my hand.

“Hi there, I am Phil chairs.  I don’t think we have met?”

“No sir, we haven’t.  I am Tom Lispon. I work in information security here.  Good to meet you.”

We shook hands and I picked my precious cup of liquid genius up from the desk.

“Tom, why don’t you come in to my office for a minute and tell me what is so important?”

“But Phil, you have almost no prep time before your meeting with our CEO at 9.” Mariah insisted.

“I know, Mariah but something tells me I need to hear what Mr. Lispon has to say.”

We both headed in to my office and Tom shut the door behind him as I sat behind my desk.

“So what’s the deal Tom? What is important for me to know?”

“Well sir one of my friends on the network team found some suspicious undocumented hardware in our first datacenter.  He tipped me thinking it was a top-secret infosec project but he was wrong.  The hardware was from the company we bought in the reverse merger several years back.  When my team dug in to what it was doing and why, let’s just say that we found several hacked servers.”

My stomach was not giving me a pleasant sensation at this point.  My mouth was dry.  I willed myself with my inner voice “Drink some coffee Phil”. I listened to myself and drank deeply from the cup.

“How long have they been compromised and who knows about this?” I asked.

“Well that’s the bad news..from the file dates and error logs it looks like these boxes have been owned for years. At this point exactly 5 people know of this the only two outside of infosec are you and my friend in the network group.”

“hmm.  I assume we need to preserve evidence and a chain of custody? I mean do you have any idea who may have done this and why?”

“Right now we have nothing.  It is a pretty cold trail.  I would like your permission to take the servers off-line.”

“You mean you haven’t done that yet?  If they are old and unaccounted for why are they even on in the first place?  Of course you can turn them off.”

I was sure that I was getting a bit flushed as all the sudden I felt quite warm in the face.

“Well that’s the other thing. See the servers are still in use.  The executive management team still uses the Exchange accounts on those servers.  If we shut them down we take out their email.”

Oh my goodness.  The boxes were compromised AND the mail accounts of my peers was as well.  

“Well I guess the agenda of my meeting with our CEO just changed” I pondered out loud.

Just then the phone rang. It was Mariah. I picked it up.

“Yes, Mariah.”

“It is time for your meeting with the CEO.”

“Great, thanks Mariah”

I turned back to Tom.

” I need a full update on what you recommend we do and how we can be sure there are no other machines that have been compromised.  Oh yeah, I need it in one hour. Ok?”

Tom nodded his head.

“You have whatever permission you need to take that gear down in an orderly fashion. Also announce it as an emergency maintenance procedure. DO NOT tell anyone else that this has happened ok?”

Tom nodded again

“I want you to gather everyone that knows about this here in my office this afternoon for a briefing ok?”

“Ok Mr. Chairs.”

“See you then”

To say I dreaded my next meeting was the understatement of my career.  I knew this would not reflect well on our organization and may launch the rest of my shark-peers in to blood-in-the-water-mode.

Bleep Bleep

I had just finished almost ten solid minutes of uninterrupted work on my bosses’ pet project.  I had three ssh sessions and about 25 other windows going on my two monitors.

The phone caller-ID read Tom Lispon, an infosec Ninja, and a great friend of mine from Information Security.

“What’s the haps Tom?” I sighed into the phone.

“Dude, don’t sound so enthusiastic just for me.  I am just delivering the friendly pre-atom-bomb-clue-stick to you.”

“Oh, yeah?” I managed while putting the finishing touches on a script and saving it.

“Yeah, remember those old servers you found in the datacenter last week?”

As I secure copied the script to a server, su-d and ran it I remembered, Tom was speaking of two servers that had no visible identification and had multiple nics in them.

At the time, I thought they were some sort of experiment by the security team.  They certainly fit the mold of a skunk works project, made of retired server hardware and located in the back corner rack of row one of the old data center sans any physical asset tags and or labels of any kind. So naturally I queried my buddy Tom and acted like I was in the know just to spook him.

Turns out nobody owned these boxes and they had been here forever.

They were Microsoft boxes and still had their own domain setup on them, we had migrated to Active Directory eons ago so either these boxes were for some app that was not happy with AD or they were just missed during the migration.

“Yeah, I remember.  Did you ever figure out who they belonged to or even what they are doing in there?”

“Um, yeah, and boatloads more. So check this out, remember we couldn’t find them in any docs? Well turns out when we did that whole reverse merger years back they were the original Exchange and Outlook Web Access servers for the company we bought.”

I started to laugh one of those ‘this-is-not-really-funny-but-everyday-here-is-like-a-masters-program-in-how-not-to-manage-anything’ laughs, but then it hit me.

If the servers were from the company we bought in the reverse merger many of the accounts on it would have been from the now senior executives of our company. Holy crap, this was getting interesting.

“Tell me the boxes are dormant dude.  Please tell me.” I quietly prayed. 

The other reason I was sure that the boxes were a security project had to do with the fact the multiple network cards and their connections actually connected to several different networks.  I was pretty sure that one of the NICs plugged directly into a DMZ or non-firewalled VLAN.

“I wish I could, but they are still being used. Only by about a dozen accounts as far as I can tell”

“You found some creds to get in?”

“Yep, actually we didn’t need them.”

“What?  Was the console just logged in with no time out after like 4 years?”

“No, the boxes had all been owned for, like, many years, dude. They had reactivated and modified very handy backup-administrator accounts, removed the passwords, and gave them full administrator privileges”

This was getting worse by the minute.

“Looks like they used an old IIS hack and from there they just owned the box.  It’s got all kinds of tar files in a temp directory.  Including a nice rootkit and a bunch of tools to explore the inside network from the servers. Oh yeah we found some old CDC stuff and a old Back-Orifice app listening for connections.”

God, the thought of someone opening and closing the CD-ROM drive of the old servers in our datacenter, not to mention all of the data… Oh man.

“Well at least not many users are on the box, right Tom?

“True, but it is the entire executive management team minus like two of our original execs.  But that list includes the CEO, CFO, CMO, COO, Chief legal counsel, and several VPs.  This is not pretty my friend.”

“Indeed.  Have you told anyone yet?” I asked.

“I had to.  I scheduled a quick meeting with our new interim CIO.  He was less than pleased.  I guess he has been on the job all of about four days now.  Poor guy, I actually felt bad for him, unlike the last guy who let this happen on his watch.”

“Yeah, I hear you.”  I chimed in.

I could count the CIOs I had met or even read about that understood security let alone operations with just a couple of fingers.

“So you wanna hear the real kicker we found?” Tom baited.

“Probably not, but go ahead.  Tell me.”

“The Outlook Web App box had a capture app intercepting the usernames and passwords as everyone was logging in.”

“Oh man they were harvesting creds from that box.  Good thing we have a policy mandating different creds than your internal ones for external facing web apps like that”

“Well… not back then we didn’t.  All of the account names and passwords were owned for years. When you consider that people usually recycle three different passwords for mandatory passwords over the course of a year they have a good chunk of our execs creds. We are still trying to figure out what you can see from these boxes.  Even though they are not part of the Active Directory they have access to a lot of stuff via legacy domain trust relationships with other old stuff. ”

This whole scenario was getting out of control.  There would be no way to put a lid on this, as we would need to interrupt the senior exec’s email to resolve the problem.

Mich Kabay wrote a very nice review of Visible Ops for Network World. 

He also wrote a great review of Visible Ops Security edition.

kevinbehr/rant

So this innocent question at Slashdot received so many completely ridiculous responses I simply HAD to say something for the record.

IMHO any IT organization that puts an IT worker in the line of fire like this just flat out SUCKS.  

It is Managements job to understand the goals of the company it works for.  Then it is Managements job to design and build a system that helps accomplish those goals.

To paraphrase Goldratt “Technology CAN (not will but can) only provide value if it diminishes or removes business constraints (See Must Read books and buy “The Goal” and “Beyond the Goal” from link to the right). 

I love these definitions of System from Merriam Webster

•  an organized set of doctrines, ideas, or principles usually intended to explain the arrangement or working of a systematic whole 
• an organized or established procedure 
• a manner of classifying, symbolizing, or schematizing<a taxonomic system> <the decimal system>4: harmonious arrangement or pattern : order

It is incumbent on IT management to create a system which includes a general flow or direction of work, a clear relationship between the goals of the system and how the system will achieve those goals, detailed specification of the roles of the workers, precisely detailed specification of the work to be performed by the workers (bearing in mind the implicit assumptions inside of each task that the level of specification must provide deterministic proof of as output of the task - such as a.) whether the worker is capable of the task and b.) whether the task will deliver the value expected), and an adequate system of controls to help management determine if the system as whole or in part is working as expected.

If you are complaining about having to document steps or tasks so some “less experienced or lower pay grade worker” can do the work I would bet a signed dollar you are in a low performing system that consists of little beyond intentions to arrive at a destination. But contains  zero actual instructions s on how to get there (where “there” even is), milestones along the way and constant interuptions by a cast of dozens to take different trips to unknown other destinations..in other words, IT hell.

Btw it is the cause of this very symptom that brings entire IT orgs, and the businesses that depend so dearly on them, to their collective knees.  

/end rant