Definitions Matter: Part Two - The Operating System

by admin on Dec.31, 2009, under IT Management

My definition (purpose) of IT Operations  ”The Operating System”

“An explicitly specified system of work that consists of a structured, controlled and measured approach to the continuous delivery of existing IT Services and Service Assets in order to achieve the goals of business.  The system accurately illustrates capacity, describes, manages and measures the flow of work in to operations via Service Transition and Standard Operations Requests and compares their actual resulting output to business expectations.

The Operations System collectively focuses on  preventive action first, detective controls that instrument variance from specified acceptable operating states, and lastly when operations are interrupted, specified structured  corrective measures that deliver root cause analysis with corresponding countermeasures to prevent reoccurrence are used to restore operation.”

I know you have operating systems in IT but do you have an IT Operations System? If you do, could it be better? Next post I will look in to the pillars of “The Operating System”

Kevin Behr

2 Comments :, , , , more...

IT Ops or IT Slops? Definitions Matter: Part One

by admin on Apr.07, 2009, under IT Management

 “I wish we had dedicated project resources. I am so busy with operations that I just don’t have time for projects.”

“Why does every IT issue get escalated to my top network and security people?”

“I don’t care if you have enough time. I need this stuff done now.”

“It takes me more time to fill out a change request than to make the actual change”

“We spend over 70% of our time doing operations which only leaves me 10-15% to work on projects after I read my email”

Sound Familiar?

I love the provocative statement that Goldratt made (I paraphrase):

“Technology CAN (not does) provide value IF and only IF it diminishes a business constraint.”

Before you go off emailing me that technology has many other values please reflect deeply on this statement.  Please reflect deeply on what business value IT provides. 

I love the notion of continuity that the “diminishes” brings to the statement.  In other words the constraint must be continually diminished as opposed to the word vanquished.

In order for the constraint to be continually diminished the technology must operate without ceasing or the constraint is re-introduced and the business is forced to deal with the issue once more, usually without any warning.

One could argue that not all applications in service deliver the best business value. But for the purposes of this discussion on IT operations let us assume that everything we are running in our datacenters provides the business with some critical constraint-busting capability.

By doing everything we can to ensure that those systems continue to operate free of interruption we are performing IT operations that support business operations.

The basic definition of Operations is “The act of harvesting value from resources”.  More specific to IT I believe that IT operations represent our collective approach (strategy) and tactics (tasks, instructions, and programs) designed to prevent outages and interruptions to the IT services that existing business operations depend on.

Since the whole point of business operations at large is to produce profit (or achieve the mission for you non-profits) from its resources, when we perform IT operations successfully we are protecting business revenue generation.  I like to call it “protecting revenue” for short

You may notice that firefighting, support or outages did not appear in my definition or mission of operations.

This is very important.

When we are troubleshooting, or firefighting an outage, operations have ceased and we are now in recovery mode (attempting to recover operations).  Even if the issue is service impairment versus a full blown IT black-out we are attempting to recover from the situation and therefore are not in nominal operation. Both scenarios interrupt or affect business operations and can put revenue at risk in many ways.

So if you were really spending 70% of your time in operations..Do you think there would be so much chaos?

Next Post I will talk more about defining and measuring operations and the value it can provide. 

 

kb

3 Comments :, , , , , more...

Steve Spear wins a Shingo Prize for Excellence

by admin on Mar.30, 2009, under IT Management, Worthy Reading!

I just received this email from my colleague Steve Spear whom I am working on several articles with at the moment.  Congrats Steve!  You deserve this award for some of the most profound work I can remember!

“Dear Friends and Colleagues,

 

I’m delighted that my book, Chasing the Rabbit: How Market Leaders Outdistance the Competition and What Great Companies Can Do to Catch Up and Win, has received welcome accolades in the last few days.  The book was awarded a Shingo Prize for Research Excellence, and it received a flattering appraisal in Harvard Business Review’s April issue.

 

The reviewer, Anand J. Raman, writes:

 

            Spear…has dazzled readers with his insights into what makes 

            Toyota tick and his understanding of how any organization can 

            use those ideas to improve its effectiveness. Not surprisingly, his 

            first tome was highly anticipated, and it’s probably an understatement 

            to say that it won’t disappoint.

 

He concludes…

 

            I have a dozen books on Toyota stacked on my shelf, in order from the 

            least read to the most referred to-and Chasing the Rabbit is probably 

            going to stay right on top of the pile.

 

Chasing the Rabbit is based on my research which was initially meant to answer the question: Why was Toyota doing so well despite (a) being in a hyper competitive market, (b) starting well behind its rivals, and (c) having been studied and imitated intensely?  

 

The answer was that for all the attention that outsiders had paid to particular production control tools, the company’s real genius was its management system that fostered and sustained high velocity, high endurance improvement, innovation, and invention across a broad range of work. I later found other organizations that had arrived at similar approaches, and with the generous help of myriad practitioners showed that this ‘high velocity’ approach has great impact across a broad range of situations.       

 

Based on this research, Chasing the Rabbit explains how competitive advantage can be generated in even the most arduous markets and illustrates its points with diverse examples from heavy and high tech manufacturing, new product development and production, commercial and military situations, and health care.

 

I’m much indebted to those who helped advance this work over many years and it is my hope that their efforts and mine prove to be useful to you and your colleagues as you attempt to generate far more value with far less effort than most in your fields even imagine possible.

 

With best wishes,

Steve Spear

 

Senior Lecturer, Massachusetts Institute of Technology

Senior Fellow, Institute for Healthcare Improvement

Please visit my blog: http://chasingtherabbitbook.com “

I highly recommend this book and keep an eye out for our articles on IT and healthcare!

2 Comments :, , more...

Wrong Bullet, Wrong Gun, Crooked Barrel with Finger, Wrong Argument, No Data

by admin on Mar.18, 2009, under /rant

This article angered me for several reasons.  To imply that IT has gone too far with process standardization is wholly inaccurate IMHO.  To write the article with zero actionable take aways is very annoying.  To do all of the above based on conjecture is irresponsible.  

I often take issue with the IT press’ assumptions regarding the current state of  IT management. But this is beyond issue for me.  This kind of thinking does nothing to foster the type of intentional system design that is so desperately needed in today’s IT organizations.

The symptom of one size fits all process adtopion is just that, a symptom of poor management.  To suggest that the solution for poor management is somehow to fuzzy the lines around process and end the mellow harshing, creativity killing approach of process standardization is just hollow.

Building effective high performing systems is focused 100% on achieving company goals and if that means different approaches for different regions so be it.  Why is this so hard to see?  

Besides is this really a conversation you could imagine having with your CEO?

I just pinged Steven Spear about this and want to write a response that draws attention to the pressing issues facing IT, based on over 10 years of research and empirical data.  

/end rant

3 Comments :, , , , more...

In Search of The Ultimate IT Robot (Ever been transformed by a trip to Piggly Wiggly?)

by @kevinbehr on Mar.12, 2009, under IT Management

When asked to improve, what is ITs obsession with the dream of total automation of everything?

Is it the ultimate exercise in black-and-white thinking? Given the reality of countless failed automation efforts marked by dead-bodies, is there a better middle road?

This basis of my ensuing posteriori-logic-trap becomes especially apparent when one begins to peel away at the thin veneer that obscures the complete failure of IT at large to operate in the realm of the scientific method.

In my dictionary, automation merely consists of  defined practices, values and expected outcomes committed to code. But I have found that in the crucial connective fascia between; Project Management, DEV, QA and Ops, IT often lacks substantive specification, accurate documentation which often yields non-deterministic untraceable outcomes (action x and y caused effect z).

Ok, this may be a pretty harsh statement, but one that is borne out of over 20 years in IT, working with hundreds of IT organizations that struggle to accurately articulate either the goal or the definition of IT operations.  If nearly every implicit destination defined below those two map coordinates is off by even single digit percentages and we consider the length and breadth of the IT journey, not to mention the height of the weeds we can get caught up in, well let’s just say chronic ITFAIL pain is both aft and on the horizon.

In the great book, Why Smart Executives Fail: And What You Can Learn from Their Mistakes the classic story of GMs “must replace labor with robots” fail is told in brutal hindsight.  The lessons are clear but are an order of magnitude harsher when one considers the lesson that history has now taught us.  Not only was the money they spent on those automation efforts lost to their deficient consitency of practice (automation just accelerated their rate of failure) but they could have purchased Nissan, Toyota, Honda and maybe even Mazda with the money they squandered on robots.

Only after years of trial and error have manufacturers struck a balance between automation and human involvement.  Shigeo Shingo, the first person to document the Toyota Production system author of many amazing books including, Kaizen and the Art of Creative Thinking - The Scientific Thinking Mechanism, formalized this approach by refining the Japanese concept of Jidoka or Autonomation.  Simply put, Autonomation is automation with human intelligence.  This is the direction we need to explore in IT for our command and control systems.

To help define just where the intersection and labor divisions should best occur there are several Toyota Production system terms that are worth investigating as a path to improving your shop’s performance

Muri - Overburden - Is every day an exercise in futility? The email piles up the issues escalated, phone calls from execs, standing daily or weekly outage conference calls? Is your IT organization behind or stuck on projects?  How many of these precious business projects are missing their commitment dates, over budget, under resourced because your team is overwhelmed with unplanned firefighting and drive-bys?

Mura- Inconsistency - Routine tasks and changes are like roulette with a two out of ten ending in unexplained failure that consumes your brightest staff for hours or days?  Is patching or upgrading a fearful event which is marked by all knocking on wooden or even wood veneered objects and the presence of a shaman or holy person to ward off evil fail spirits?

Muda -Waste-All of those operating expense dollars lost to firefighting, audit corrective action drive-bys, shadow IT projects, unauthorized changes and root cause analysis meetings that take weeks to recommend the same trifecta of we need more budget, more staff and more time to focus on proactive tasks?

Right now many IT organizations are looking at Muda or waste in order to drive down costs.  I posit that understanding Muri and Mura would be a much more valuable use of time and ultimately will reduce waste and increase IT throughput

Inevitably the solutions recommended by IT teams to these issues will involve automation or tools.  This is not all bad, but the focus should be on building more deterministic ways of working for humans and considering where automation may help humans interact with their IT infrastructure more consistently.

This process of self reflection or Hansei is important fuel for Kaizen (continuous improvement).  It becomes essential to distill all of the undesirable symptoms of overburden, inconsistency and waste and understand the few root causes that drive them all.

Did you know that the “Just In Time’ concept was pioneered by a group of Toyota Employees? These Toyota team members were lead by Taiichi Ono on a trip to the US in the 1950s to visit US auto manufacturers .  They journeyed to Michigan and walked through Ford plants and were generally unimpressed by the high amounts of inventory they required to operate and the variance in labor output from day to day.

During the visit they stopped by a Piggly Wiggly grocery store and were amazed by their inventory replenishment system that only requested new items when they were sold. From this focus on Kaizen and Hansei they developed what later became the famous Just-In-Time philosophy that has become a pillar of the Toyota Production System.

It is not merely enough to improve in this economy, we are faced with the imperative of only improving the most important functions so as to quickly improve execution and IT throughput. As we set out on our journeys and investigation of other practices let’s make sure we are attacking true root causes of overburden and rework not just merely their undesirable effects.

I think that the Toyota Production System offers us many valuable insights in to building better IT. I find the thinking behind the system to be more enlightening than the practices. I encourage you to view all TPS, Lean and “Best Practices” in this light. Often the answer to “why” is more important than the “what” IMHO.

I will be writing more about the intersections of TPS and IT.  I will be focusing on universal principles, that draw from Goldratt’s Theory of Constraints work, Steven Spear, Taiichi Ohno, Shingeo Shingo, Deming and the 10 years of research Gene Kim and I have done around IT high performance, in the coming weeks.

5 Comments :, , , , , , , , , , , , more...

The Adventures of Phil Chairs, Interim CIO-Day 4

by admin on Mar.05, 2009, under The Adventure of Phil Chairs - Interim CIO

This morning I was running 45 minutes behind my usual schedule.  I tried to reassure myself that I was still in over an hour before any of my staff, except for Rob my Deputy CIO.

As I walked towards my office I saw someone chatting with Mariah at her desk in front of my office.

I walked directly to my door and stepped inside my office to divest of my coat and briefcase.  I emerged an headed straight for the coffee cache in the back of Mariah’s cubicle.

Mariah was trying to schedule a meeting with the gentlemen from what I could overhear.   

“Well the first available slot is tomorrow at 11am.  Phil only has a half-hour available.  Otherwise I can get you one hour next week.  What will it be?” Mariah queried the young IT staffer.

“sheesh.  Well this is extremely important and confidential.  It is an information security issue that must be addressed and I need Mr. Chairs to be up to speed before anything is done.”

I decided that I had heard enough.  I walked around the front of the desk and set my coffee down on the return.

I turned to address the staffer with a warm look and extended my hand.

“Hi there, I am Phil chairs.  I don’t think we have met?”

“No sir, we haven’t.  I am Tom Lispon. I work in information security here.  Good to meet you.”

We shook hands and I picked my precious cup of liquid genius up from the desk.

“Tom, why don’t you come in to my office for a minute and tell me what is so important?”

“But Phil, you have almost no prep time before your meeting with our CEO at 9.” Mariah insisted.

“I know, Mariah but something tells me I need to hear what Mr. Lispon has to say.”

We both headed in to my office and Tom shut the door behind him as I sat behind my desk.

“So what’s the deal Tom? What is important for me to know?”

“Well sir one of my friends on the network team found some suspicious undocumented hardware in our first datacenter.  He tipped me thinking it was a top-secret infosec project but he was wrong.  The hardware was from the company we bought in the reverse merger several years back.  When my team dug in to what it was doing and why, let’s just say that we found several hacked servers.”

My stomach was not giving me a pleasant sensation at this point.  My mouth was dry.  I willed myself with my inner voice “Drink some coffee Phil”. I listened to myself and drank deeply from the cup.

“How long have they been compromised and who knows about this?” I asked.

“Well that’s the bad news..from the file dates and error logs it looks like these boxes have been owned for years. At this point exactly 5 people know of this the only two outside of infosec are you and my friend in the network group.”

“hmm.  I assume we need to preserve evidence and a chain of custody? I mean do you have any idea who may have done this and why?”

“Right now we have nothing.  It is a pretty cold trail.  I would like your permission to take the servers off-line.”

“You mean you haven’t done that yet?  If they are old and unaccounted for why are they even on in the first place?  Of course you can turn them off.”

I was sure that I was getting a bit flushed as all the sudden I felt quite warm in the face.

“Well that’s the other thing. See the servers are still in use.  The executive management team still uses the Exchange accounts on those servers.  If we shut them down we take out their email.”

Oh my goodness.  The boxes were compromised AND the mail accounts of my peers was as well.  

“Well I guess the agenda of my meeting with our CEO just changed” I pondered out loud.

Just then the phone rang. It was Mariah. I picked it up.

“Yes, Mariah.”

“It is time for your meeting with the CEO.”

“Great, thanks Mariah”

I turned back to Tom.

” I need a full update on what you recommend we do and how we can be sure there are no other machines that have been compromised.  Oh yeah, I need it in one hour. Ok?”

Tom nodded his head.

“You have whatever permission you need to take that gear down in an orderly fashion. Also announce it as an emergency maintenance procedure. DO NOT tell anyone else that this has happened ok?”

Tom nodded again

“I want you to gather everyone that knows about this here in my office this afternoon for a briefing ok?”

“Ok Mr. Chairs.”

“See you then”

To say I dreaded my next meeting was the understatement of my career.  I knew this would not reflect well on our organization and may launch the rest of my shark-peers in to blood-in-the-water-mode.

2 Comments :, , , , , , , , , more...

The IT Performance DOJO is no longer optional, but survival is

by @kevinbehr on Mar.03, 2009, under Hire Kevin Behr as a Consultant, IT Management

Have you ever read “The Mythical Man-Month: Essays on Software Engineering“?

If so you are a rare bird (or over 45) in today’s tech zoo.

I wish I could drop this book (among many others such as The Goal, Critical Chain, The Machine that Changed the World, Great Boss Dead Boss, and Chasing the Rabbit  - see the right side column to buy them!) out of airplanes and organize study groups with my IT; operations, security, audit, project management, executive management and yes especially development brethren.

I was reflecting on this text and several other seminal management books I have collected over the years and I decided I needed to write a series for my blog on this topic. As of late Gene Kim and I are working on some amazing projects for fun and exciting internet 2.0 companies. We have been writing a new IT management novel with George Spafford (think “The Goal” for IT) , and helping a bunch of IT shops solve the quality, reliability and cost reduction realities that have now become must-do (all at the same time mind you).

Just yesterday we began working with long time collaborator and joyful subversive, Eileen Forrester, from the Software Engineering Institute at Carnegie Mellon University.

We are shaping up a few writing projects for the newly released CMMI-Service model.  Eileen has been in charge of getting the new model released on time and has actually over-performed and it is out much earlier than expected for the first time in history.

I relish the opportunity to exchange ideas and approaches for improving the delivery of services with folks from all sectors of the economy (especially those disciplines outside of the IT world).

Many of these industries and business disciplines have a long history of systems design and improvement that reads like rich history.

Where is this similar lineage taught and practiced for IT today?

As I work with many IT executives and teams around the globe I often find myself wondering if the collective “we” has forgotten more than we have learned up to this point.

For a community that not only leans towards scientific interest our entire profession exists on the shoulders of science (knowledge and technology are the output) so little of the way we actually do IT is built on the scientific method or around what I have dubbed the “System Building and Continuous Improving Arts” (SBCIA).  We often are just reacting rather than treating every request as a hypothesis and making sure we have built a repeatable response that correctly satisfies all of the implicit assumptions in the request.

To better explain what I am talking about I will use the Japanese term RYU, which denotes the flow of water.  The idea represents formal Japanese traditions such as the Martial Arts.  In Karate fluidity and finding one’s path are compared.  Water will find its own path in balance with the terrain it is surrounded by, just as the Karate student must advance through his or her path by practice and training.

Bringing me to the point of my concern in this post…

Much of “the path” for IT practitioners and managers of all levels travels through the acquisition and appropriation of myopic technical domain knowledge (yes even auditors). In other words “what we must know” so “we” know how to react in a given technical situation”.

What is missing?  What we need to understand holistically (see) to decide how we should work together as a system .  This information is both timeless and too time consuming to amass through trial and error in our own life experience. The old adage applies here “Time may be the best teacher but it kills all of its pupils”. In order to amass the collective wisdom about how to see the work in order to build an effective system to do the work, we need the collective wisdom of our predecessors and our current community at large. We also need to look outside of our industry and find analogs to improve our execution, quality and safety with larger and scientifically validated levers. Stop chasing consensus based “Best Practices” and start purpose-building systems around scientifically proven practices.

There is much room in IT to build a new flow of knowledge about the “System Building and Continuous Improving Arts” (SBCIA).  This economy is shaping up to be the ultimate Dojo for us to do so.  Let’s just say performance is no longer optional but survival is.

So pick up a copy of “The Mythical Man-Month: Essays on Software Engineering” and start a study group and learn from your ancestors.  Create a culture of Kung-Fu, a Chinese term that means “individual accomplishment or skill cultivated through long and hard work” .  Set the tone with your team and define skill from a systemic and improvement standpoint rather than merely a technical one.

Check out the books in the far right column on this blog to find a handy DOJO compendium to get you started on this journey.  If you are serious about improving IT flow, reaping the drastic cost savings, increases in project execution and infrastructure reliability that result, feel free to contact me about mentoring and staff coaching.  It really does make a huge difference!

If you have stumbled upon a nugget on your journey please let me know I am @kevinbehr on Twitter - join the conversation!

 

5 Comments :, , , , , , , , , , , , , , , , , , , , , , , more...

The hidden sucking sound -Act4

by admin on Mar.02, 2009, under The Sucking Sound Saga

Bleep Bleep

I had just finished almost ten solid minutes of uninterrupted work on my bosses’ pet project.  I had three ssh sessions and about 25 other windows going on my two monitors.

The phone caller-ID read Tom Lispon, an infosec Ninja, and a great friend of mine from Information Security.

“What’s the haps Tom?” I sighed into the phone.

“Dude, don’t sound so enthusiastic just for me.  I am just delivering the friendly pre-atom-bomb-clue-stick to you.”

“Oh, yeah?” I managed while putting the finishing touches on a script and saving it.

“Yeah, remember those old servers you found in the datacenter last week?”

As I secure copied the script to a server, su-d and ran it I remembered, Tom was speaking of two servers that had no visible identification and had multiple nics in them.

At the time, I thought they were some sort of experiment by the security team.  They certainly fit the mold of a skunk works project, made of retired server hardware and located in the back corner rack of row one of the old data center sans any physical asset tags and or labels of any kind. So naturally I queried my buddy Tom and acted like I was in the know just to spook him.

Turns out nobody owned these boxes and they had been here forever.

They were Microsoft boxes and still had their own domain setup on them, we had migrated to Active Directory eons ago so either these boxes were for some app that was not happy with AD or they were just missed during the migration.

“Yeah, I remember.  Did you ever figure out who they belonged to or even what they are doing in there?”

“Um, yeah, and boatloads more. So check this out, remember we couldn’t find them in any docs? Well turns out when we did that whole reverse merger years back they were the original Exchange and Outlook Web Access servers for the company we bought.”

I started to laugh one of those ‘this-is-not-really-funny-but-everyday-here-is-like-a-masters-program-in-how-not-to-manage-anything’ laughs, but then it hit me.

If the servers were from the company we bought in the reverse merger many of the accounts on it would have been from the now senior executives of our company. Holy crap, this was getting interesting.

“Tell me the boxes are dormant dude.  Please tell me.” I quietly prayed. 

The other reason I was sure that the boxes were a security project had to do with the fact the multiple network cards and their connections actually connected to several different networks.  I was pretty sure that one of the NICs plugged directly into a DMZ or non-firewalled VLAN.

“I wish I could, but they are still being used. Only by about a dozen accounts as far as I can tell”

“You found some creds to get in?”

“Yep, actually we didn’t need them.”

“What?  Was the console just logged in with no time out after like 4 years?”

“No, the boxes had all been owned for, like, many years, dude. They had reactivated and modified very handy backup-administrator accounts, removed the passwords, and gave them full administrator privileges”

This was getting worse by the minute.

“Looks like they used an old IIS hack and from there they just owned the box.  It’s got all kinds of tar files in a temp directory.  Including a nice rootkit and a bunch of tools to explore the inside network from the servers. Oh yeah we found some old CDC stuff and a old Back-Orifice app listening for connections.”

God, the thought of someone opening and closing the CD-ROM drive of the old servers in our datacenter, not to mention all of the data… Oh man.

“Well at least not many users are on the box, right Tom?

“True, but it is the entire executive management team minus like two of our original execs.  But that list includes the CEO, CFO, CMO, COO, Chief legal counsel, and several VPs.  This is not pretty my friend.”

“Indeed.  Have you told anyone yet?” I asked.

“I had to.  I scheduled a quick meeting with our new interim CIO.  He was less than pleased.  I guess he has been on the job all of about four days now.  Poor guy, I actually felt bad for him, unlike the last guy who let this happen on his watch.”

“Yeah, I hear you.”  I chimed in.

I could count the CIOs I had met or even read about that understood security let alone operations with just a couple of fingers.

“So you wanna hear the real kicker we found?” Tom baited.

“Probably not, but go ahead.  Tell me.”

“The Outlook Web App box had a capture app intercepting the usernames and passwords as everyone was logging in.”

“Oh man they were harvesting creds from that box.  Good thing we have a policy mandating different creds than your internal ones for external facing web apps like that”

“Well… not back then we didn’t.  All of the account names and passwords were owned for years. When you consider that people usually recycle three different passwords for mandatory passwords over the course of a year they have a good chunk of our execs creds. We are still trying to figure out what you can see from these boxes.  Even though they are not part of the Active Directory they have access to a lot of stuff via legacy domain trust relationships with other old stuff. ”

This whole scenario was getting out of control.  There would be no way to put a lid on this, as we would need to interrupt the senior exec’s email to resolve the problem.

4 Comments :, , , , , , , , , , , , more...

How to make changes you can believe in!

by @kevinbehr on Feb.25, 2009, under Hire Kevin Behr as a Consultant, IT Management

*Put down the Blackberry. Step away from the Blackberry please.”

When I read this article about Obama’s Blackberry use, I rolled my eyes at some of the intellectual stretches it was taking. While I agree with the premise and I do certainly understand the Blackberry-jones (nervous separation anxiety and the compulsion to thumb it when the little red light blinks) premise, I was far more interested in the work that Gloria Mark was engaged in.

“When Gloria Mark, professor of informatics at the University of California, Irvine, shadowed employees at two high-tech firms, she found that the average worker spends only 11 minutes on any given project before being interrupted and asked to do something else. IT workers have it worse, switching attention every three minutes, on average.”  - Newsweek Magazine, 2/16/09

Over the last ten years, Gene Kim and I have been inside of and benchmarked hundreds of IT organizations, conducted numerous research studies and written a LOT about this very topic of unplanned work. We have come to believe that, largely, IT management as a whole has not built an effective system for “Doing IT.”  This study from the University of California dovetails with some of our findings from benchmarking nearly 1000 IT organizations.

Namely, the first finding that Unplanned Work, which by definition is the IT equivalent of task assignment by drive-by-shooting (fire-fighting outages, urgent security patching or compliance related work, essentially anything that takes a worker off of the planned task list of things that are important) is virtually non-existent in high performing shops compared to everyone else.  This is evident both by the low amount of it (expressed as a percentage of labor opex often as low as five to ten percent, compared to the rest at twenty to fifty percent of labor opex lost to heat) and by the advantage high performers have over everyone else in project execution. High performers get up to 8X the projects done, 14-16x times the infrastructure changes with one-half the change failure rate.  Also, their mean time to repair an outage is as low as 1/10 the time of low performers!  All of this creates higher availability and the ability to execute more projects in a more controlled manor with less negative business impact.

Oh yeah, did I mention that they do this with as little as 1/4 of the system administrator staff when compared to low performers?

By building an effective execution flow system it is not only possible, it is probable that you will outperform your competitors and actually help your company meet its goals.

Over the last ten years Gene and I have learned:

1.       Constant fire-fighting  and uncontrolled change weakens infrastructure and creates security problems,

2.       The hero culture that ensues accelerates the rate of outage entropy because little information and visibility in to the known good state exists for others to draw on,

3.       Your best and most talented people are stuck on the fire-fighting line instead of solving business problems,

4.       Your ability to demonstrate proof of compliance diminishes with the more outages you have,

5.       More fire-fighting = less project and planned work = less credibility with business peers = shadow IT = failure,

6.       This can all be fixed by building a system based on Flow and controls.  No, I am not speaking of a never-ending-death-march of process analysts and skies darkened by consultants. I am talking about building a deliberate discovery based system that illuminates what is actually happening versus obscuring it. This type of system learns from every mistake and seldom repeats the mistakes made in the past.

Sound interesting?  If this is the direction your organization wants to move - Dramatically Lower Cost structures, Drastically Increased Project Delivery capability, and Rock Solid Reliability, then contact me here and let’s talk about how we can help!

We are currently doing just this for some of the most advanced and largest IT operations in existence today.

2 Comments :, , , , , , , , , , , , , , more...

Article Gene Kim, George Spafford and I wrote for CMDB Magazine

by @kevinbehr on Feb.24, 2009, under Hire Kevin Behr as a Consultant, IT Management, Visible Ops, Worthy Reading!

You can download the PDF of the article here.  This is a great overview of Visible Ops for execs.  If you want to talk to your boss about adopting Visible Ops, ITIL or just some of the concepts in the book. This document is nice and portable and has pretty colors (hint: execs love color).

If you are interested in improving your IT organization’s ability to focus on and complete planned work, such as projects and proactive info sec work, feel free to contact me here to find out about the workshops, benchmarking and briefings I do to help get folks started!

2 Comments :, , , , , , , , more...

Looking for something?

Use the form below to search the site:

Still not finding what you're looking for? Drop a comment on a post or contact us so we can take care of it!

Archives

All entries, chronologically...