*Put down the Blackberry. Step away from the Blackberry please.”

When I read this article about Obama’s Blackberry use, I rolled my eyes at some of the intellectual stretches it was taking. While I agree with the premise and I do certainly understand the Blackberry-jones (nervous separation anxiety and the compulsion to thumb it when the little red light blinks) premise, I was far more interested in the work that Gloria Mark was engaged in.

“When Gloria Mark, professor of informatics at the University of California, Irvine, shadowed employees at two high-tech firms, she found that the average worker spends only 11 minutes on any given project before being interrupted and asked to do something else. IT workers have it worse, switching attention every three minutes, on average.”  - Newsweek Magazine, 2/16/09

Over the last ten years, Gene Kim and I have been inside of and benchmarked hundreds of IT organizations, conducted numerous research studies and written a LOT about this very topic of unplanned work. We have come to believe that, largely, IT management as a whole has not built an effective system for “Doing IT.”  This study from the University of California dovetails with some of our findings from benchmarking nearly 1000 IT organizations.

Namely, the first finding that Unplanned Work, which by definition is the IT equivalent of task assignment by drive-by-shooting (fire-fighting outages, urgent security patching or compliance related work, essentially anything that takes a worker off of the planned task list of things that are important) is virtually non-existent in high performing shops compared to everyone else.  This is evident both by the low amount of it (expressed as a percentage of labor opex often as low as five to ten percent, compared to the rest at twenty to fifty percent of labor opex lost to heat) and by the advantage high performers have over everyone else in project execution. High performers get up to 8X the projects done, 14-16x times the infrastructure changes with one-half the change failure rate.  Also, their mean time to repair an outage is as low as 1/10 the time of low performers!  All of this creates higher availability and the ability to execute more projects in a more controlled manor with less negative business impact.

Oh yeah, did I mention that they do this with as little as 1/4 of the system administrator staff when compared to low performers?

By building an effective execution flow system it is not only possible, it is probable that you will outperform your competitors and actually help your company meet its goals.

Over the last ten years Gene and I have learned:

1.       Constant fire-fighting  and uncontrolled change weakens infrastructure and creates security problems,

2.       The hero culture that ensues accelerates the rate of outage entropy because little information and visibility in to the known good state exists for others to draw on,

3.       Your best and most talented people are stuck on the fire-fighting line instead of solving business problems,

4.       Your ability to demonstrate proof of compliance diminishes with the more outages you have,

5.       More fire-fighting = less project and planned work = less credibility with business peers = shadow IT = failure,

6.       This can all be fixed by building a system based on Flow and controls.  No, I am not speaking of a never-ending-death-march of process analysts and skies darkened by consultants. I am talking about building a deliberate discovery based system that illuminates what is actually happening versus obscuring it. This type of system learns from every mistake and seldom repeats the mistakes made in the past.

Sound interesting?  If this is the direction your organization wants to move - Dramatically Lower Cost structures, Drastically Increased Project Delivery capability, and Rock Solid Reliability, then contact me here and let’s talk about how we can help!

We are currently doing just this for some of the most advanced and largest IT operations in existence today.

You can download the PDF of the article here.  This is a great overview of Visible Ops for execs.  If you want to talk to your boss about adopting Visible Ops, ITIL or just some of the concepts in the book. This document is nice and portable and has pretty colors (hint: execs love color).

If you are interested in improving your IT organization’s ability to focus on and complete planned work, such as projects and proactive info sec work, feel free to contact me here to find out about the workshops, benchmarking and briefings I do to help get folks started!

I loved the early hours at the office.  They were even better after I found out about my own private coffee machine and secret stash of amazing bean.  It was day 3 on the job in my new role and as I gazed out over the gray and cloudy horizon of my office windows, I felt a strange rush of adrenaline. Maybe it was just the coffee, but I felt a charge of voltage in my chest today.  I had entered the proverbial lion’s den yesterday and walked out with a pretty satisfying victory.  Not many execs have the facts to call each other’s behavior out.  I had not only caught my peers red-handed circumventing the rules, I had the data to prove that they were part of the problem that necessitated those very rules.

Still I knew that until we cleaned up the IT side of the street the cadre of spinning project-plates was in real danger of crashing to the ground all at once.  It’s one thing to call foul on my business peers; it’s another altogether to solve the problems that forced them to break the rules in the first place.

When IT gets in the way of the business as bad as this shop has, successful executives (especially the type-A variety) will circumnavigate the obstacles and go direct to the folks that deliver.  I had merely put everyone on notice of one part of the problem by stopping the pressure against the outer walls of IT.  Now I had to figure out why we had so much work in progress but so little being completed.

I picked up the phone and pressed the speed dial button for Rob, figuring I would leave him a voicemail to come see me when he arrived in a couple of hours.

“Good morning sunshine.” He answered.

“Ha-ha, Good Morning Rob!  What are you doing here so early?  I figured I would leave you a voicemail.”

“Since you started I have been here every day at 7am and plan on it until you start coming in later.  I overheard you walking through the office on day one muttering to yourself about how no one is here on time and everyone is gone at 4:59.  So if you are here, so am I.”

This delighted me beyond words.  I do not equate long hours with high performance, actually just the opposite, but I knew I needed at least two hours a day of undisturbed planning and reflecting time and I was usually out of the office by 6:00pm anyway.  I really liked this Rob guy and it felt good to have a partner that not only had my back but was at my side when I needed it.

“Great news Rob, you just made my day.  Now get down here stat.  We need to plan.”

“Be there in a sec Chief.” And we hung up.

Sixty seconds later Rob knocked on the open door of my office and walked in.

“What’s on your mind?” he queried as he pulled up a chair in front of my desk.

“I have been thinking about the meeting yesterday and how we had better come up with a plan to clean up our side of the street in this whole project situation.  I feel like I can hold back the water from my peers for a while but I need to get our execution up to par. ”

Rob was nodding with every word

“Yeah you were fun to watch in the board room yesterday, but if you don’t feed the sharks some meat they will turn on you out of pure hunger. By the way you did realize when I gave you the list of 100 active projects from my discovery interviews they were in addition to the 39 approved projects?”

“You have to be kidding!” I exclaimed.

“I wish I was, Phil.  I even combed the list and found any projects that were really just tasks or sub-projects that belonged to an approved capital project. So that list is fully in addition to the approved list.  Your team is working on 139 projects. ”

“Wow.  This is not a small number” I reclined in my chair and let out a loud exhale. 

“How do you let things get this out of control?” I wondered aloud.

“This place is amazing, I have never seen anywhere like it.” Rob chuckled.

“I wish that were true, Rob.  It is depressingly all too common.  Every shop I have been in has struggled with this very issue.  Some worse than others, I will admit but I think this number of projects may take the cake. I mean, how does a CIO get so far removed from understanding what is actually going on inside his own shop?  ”

“His door was closed a lot.  His two lieutenants were essentially yo-yo’s for his peers.  I mean they were the spinning end of the yo-yo and the VPs were working them.  You have some issues with these two guys.  I think they are underwater and have no idea what they need to be focusing on.  You need to deal with this quickly.  You have a clear vision of what needs to happen and in many cases they are telling your team to do things that fly in the face of your intentions.”

I knew Rob was spot on here.  Not only were they often clueless but they were doing a lot of harm to both the credibility of our department and themselves. I had already been confronted on day two of my post by two of my peers who made it clear they thought I should cut both of them loose to pursue other career options.

I really believed that my VP of IT ops and VP of Engineering meant well but I also knew that they were threatened by me, by the mere nature of my position, and plans to bring sunshine to the way we were working. In every conversation I had with them, I noticed the deer in the headlights look in their eyes as I further painted my plans.

George Spafford has built an amazing resource with his Yahoo news group.  Read by executives in; audit, IT managemnt, the C-suite and consultants world-wide he provides some great information and links thought-provoking articles.  Highly recommended!

Mich Kabay wrote a very nice review of Visible Ops for Network World. 

He also wrote a great review of Visible Ops Security edition.

I arrived at the boardroom five minutes early so I could get my thoughts together after the nearly 2 hour barrage of background information from Rob.  A week prior I had also asked him to account for each IT department’s workload across projects and support related activity.  While doing so, he had gathered some very interesting data.  From the budget worksheets I pulled a list of approved and funded IT capital projects, of which there were nearly forty, but the list of project activity from the IT team was alarming. If I was reading this right there must be over a hundred IT projects in progress.  My mind was instantly flooded with synapses. Where were these projects coming from?

I thought that having forty capital projects was overload, but over a hundred was just ridiculous.  I now had proof of some serious shadow IT going on.  The question wasn’t who in the boardroom was guilty, but rather, with this number, who was innocent! I started to make tick marks against projects outside of the approved list that were easily attributable to marketing, finance or operations just so I would have an idea of who the worst offenders were.  I did this until the room began to fill with my peers. 

We were all waiting on Paul, our CEO, when I overheard our CMO, Skip Sorrenson, complain about IT being in the way of his marketing data warehouse project. He wanted to hire new staff but I had told HR to hold off on any new IT hires until I could figure out who and what were needed.  According to Rob, this project had been a money pit since day one and had sucked up millions of dollars in capital and had distracted at least a dozen team members between marketing and IT for over five years.  The kicker was that it had delivered zero value to the company.  I knew this was a touchy subject with all and that I would need to get my arms around this project and rescue it fast.

“Ok everyone, sorry to keep you waiting, my call with the street went a little over.  I had to explain our crazy high EBITDA again for the third quarter in row. Thanks Skip.” He joked dryly to a chuckling-murmur in the room. Skip was obviously a rising star in the company and due to his marketing prowess we were enjoying unprecedented quarter over quarter profits.

I heard the conference room door close and noticed my Deputy CIO slip into a chair along the wall in the back of the boardroom.  I told him that he should be here in case I needed backup and to document any deliverables I might get assigned.

“None of us has any time to waste, so let’s talk about IT projects.  Everyone here knows Phil, our interim CIO, by now.  Phil I know you haven’t had much time but can you give us an update on the major capital projects?  I would like to know, first of all, how much of your capital allotment budget has been spent, where the completion status is for the projects, and then answer any questions anyone else might have.”

“Sounds reasonable” I replied confidently, although not sure why.

“Well it is now the third quarter and we have spent less than 25% of our capital budget.  I have some serious questions about the way IT projects are run and why our completion rate is so low. I certainly understand the concerns, regarding IT project effectiveness that many of you raised during my one on one interviews with you a few weeks ago.

It seems that we have far too many IT project planes taking off and landing with little to zero tower oversight. I compiled a list of 39 approved capital projects, most of which are major, by the way. I then sent Rob McNunzio on a bit of a skunk works project to see what my team was actually working on. The results are preliminary but it looks as if the actively worked project list numbers well over 100.”

*Crickets*

I made stern eye contact with all of my peers shooting the I-know-you-are-killing us-all look, followed by a smile.

“Truthfully I was going to come to you today and say that this list of 39 capital projects needs to be prioritized and paired down to 10 major projects and 5 mid size projects just to get things under some sort of control. But after finding out about over 100 projects being active I would say we have a fundamental problem here, and by here I mean in this room.”

I had just thrown down in my first senior leadership team meeting.  I had the worst performing department in terms of credibility and satisfaction ratings, I was in my post less than a week, my department was in the way of company strategy execution, and I had just went type-a on a bunch of sharks…in the shark tank.

kevinbehr/rant

So this innocent question at Slashdot received so many completely ridiculous responses I simply HAD to say something for the record.

IMHO any IT organization that puts an IT worker in the line of fire like this just flat out SUCKS.  

It is Managements job to understand the goals of the company it works for.  Then it is Managements job to design and build a system that helps accomplish those goals.

To paraphrase Goldratt “Technology CAN (not will but can) only provide value if it diminishes or removes business constraints (See Must Read books and buy “The Goal” and “Beyond the Goal” from link to the right). 

I love these definitions of System from Merriam Webster

•  an organized set of doctrines, ideas, or principles usually intended to explain the arrangement or working of a systematic whole 
• an organized or established procedure 
• a manner of classifying, symbolizing, or schematizing<a taxonomic system> <the decimal system>4: harmonious arrangement or pattern : order

It is incumbent on IT management to create a system which includes a general flow or direction of work, a clear relationship between the goals of the system and how the system will achieve those goals, detailed specification of the roles of the workers, precisely detailed specification of the work to be performed by the workers (bearing in mind the implicit assumptions inside of each task that the level of specification must provide deterministic proof of as output of the task - such as a.) whether the worker is capable of the task and b.) whether the task will deliver the value expected), and an adequate system of controls to help management determine if the system as whole or in part is working as expected.

If you are complaining about having to document steps or tasks so some “less experienced or lower pay grade worker” can do the work I would bet a signed dollar you are in a low performing system that consists of little beyond intentions to arrive at a destination. But contains  zero actual instructions s on how to get there (where “there” even is), milestones along the way and constant interuptions by a cast of dozens to take different trips to unknown other destinations..in other words, IT hell.

Btw it is the cause of this very symptom that brings entire IT orgs, and the businesses that depend so dearly on them, to their collective knees.  

/end rant

It’s a gorgeous orange and blue hued morning as I look out the west facing glass of my spacious corner office.  I walk behind the cherry desk and return and sit in the Herman Miller chair. After fuddling with the controls I get the chair to fit me correctly.

I open up my laptop and begin to connect the power adapter and arrange the desk.

“Hello?” I am startled by a voice from the door, which I had apparently failed to close.

“Good Morning!” I replied, if not a bit too enthusiastically, I thought.

“My name is Mariah Hansen and I am your assistant.  Can I get you anything?”

Mariah was certainly not hard to look at and that was a perk.  Now I needed to figure out if I could confide in her.  The role of interim executive is often not a popular one.  When a senior leader moves on from a company it can create some serious uncertainty amongst the staff. I made sure that everyone knew that the previous CIO had asked me to take over after he left.  I had met with every peer of the CIO to establish a social rapport and let them know I was here to serve.  I had attended company holiday functions and been entertained by the President and also by the divisional CEOs.  But I have found that IT staffs are often the hardest to win over. I have found that many feel a new CIO will fire them and bring in their own dream team. I knew that the only way to convince these folks was by painting the path we needed to get on and earning their credibility day by day. 

I walked over to the door to shake Mariah’s hand.

“Great to meet you Mariah, I am Phil Chairs.  I would love to get some coffee, but I haven’t been able to find it.”

She laughed musically “We have our own machine.”

I figured an IT department of nearly 800 people, in this location alone, would have its own coffee break room.

“Could you point me in the right direction? I need this stuff to get going.” I smiled.

“No, I meant you and I have our own machine! I have one of those pod machines that makes amazing coffee one cup at a time.  It’s in my cubicle towards the back next to the color laser printer.  The coffee pods are in the locking shelf above.  You can just email me or call and I can bring you a cup anytime you want.”

“Wow, I love that idea Mariah. Consider yourself emailed then.  I need two cups, and do we have any half and half?”

“Coming right up!  Oh and we should talk before your Leadership Team meeting this morning.  Paul, your boss wants to get an update on all IT projects and their status.  Now before you panic I have already got all of that information for you.  I put a folder in your basket that has the printouts.  I have also had your new deputy CIO, Rob McNunzio, put the data into power point slides that are waiting in your email.  Look all of that over while I get you coffee and we can chat about your scheduling this week when I bring it to you.”

“Sounds great.  Thanks so much!  See you in about fifteen then.”

As I walk back to my sleek cherry desk I find the bulging folder of project updates and begin to leaf through it.

As I sit down I wonder if this data is accurate.  When was any of this was audited last?  What is our project management process, and who was in charge?  Who managed the budgets and performed the plan updates?  What kind of oversight was currently in place?  I needed my right hand man. I reached for the phone and buzzed Mariah.

“Hello, office of the CIO this is Mariah”, she answered.

“Hey it’s me again, could you get a hold of Rob and have him meet with me after you and I are finished?  I need to get answers to some questions before the leadership team meeting.”

“It would be my pleasure.  Is that all?”

“Yep, thanks Mariah.”

After going through the project Gant charts and looking at the slides Rob had created I had many more questions swimming in my head.  I pulled out a yellow legal pad from my briefcase and began listing them out as they came.

1.       Is there an established PMO?

2.       Who are our project managers?

3.       Who controls the purchasing for capital projects?

4.       Who reports on status?

5.       How many projects are on the docket?

6.       Who prioritizes them and how?

7.       Do we have a project methodology?

8.       Who is in charge of resourcing and time tracking?

9.       Who is in charge of the financial side (tracking capitalized labor, P&L reconciliation, etc.)?

10.   Who manages the project risk issues?

11.   Why are so many of the large projects at 80% completion with more time left than has elapsed?

Just then Rob knocked on my door sporting a huge grin.

“Good morning, Chief.” He joked dryly

“Morning Rob” I cracked a smile.

“I guess we should talk about this circus of projects. Mariah just called me and I was relieved.  I didn’t want you going in to a bloodbath this morning.  You need some background and history on the key projects or else you will look like a wounded fawn in there to those jackals.  Believe me, you have a couple of guys, namely the CFO, Dave Williams, and the CMO, Skip Sorrenson, that will go type-a on you and try to assert dominance so they can horse-collar you like they did the last CIO.”

“Nothing like a friendly meeting of the leadership team” I thought aloud.

Follow the high-paced adventures of international interim CIO for hire, Phil Chairs and catch a rare glimpse inside the corner office.  Become a fly-on-the-wall in the boardroom, carefully contain auditors, watch Phil masterfully lower marketing’s expectations and attempt to coach his CEO through cutting the capital project list in half while single handedly gaining funding for IT test and staging infrastructure.

This irreverent weekly blog novella from Kevin Behr, coauthor of Visible Ops and consultant to CEOs, CIOs and CTOs, will make you laugh, cry and hopefully, feel some sympathy for the IT-big-guy.

Make sure and RSS this blog so you won’t miss a single installment!

Starts tomorrow!

Kb

I rounded the corner  adjacent to the break room and overheard what started as a distant, dull roar and bloomed into a full on verbal cacophony as I peered inside the door.  My nose immediately detected the smell of burnt coffee and some sort of Indian food in the microwave.  It was not an appealing combination.  The noise was emanating from one our helpdesk managers, a guy that I recognized from an audit, and two folks from information security. 

I could not make heads or tails of the conversation except that the auditors wanted to know why there were so many helpdesk workers with the ability to grant administrative privileges to IT staffers without a clearly defined approval and decommissioning process. Apparently there were more exceptions to the policy than actual IT workers.  Man, I did not want any part of this.  If only they knew how many shared admin accounts I had with the creds tucked away in my cranium, they would all have heart palpitations so bad they wouldn’t even be able to steady their little hands to write an urgent doomsday opinion letter to management.

I decided I would actually rather get robo-coffee from the vending machine than even walk through that fray so I backtracked and headed downstairs to the vending room.

As I approached the row of machines I spied Dick Majors arguing with the candy machine.  Dick was our VP of Marketing and a major pain in our collective IT posterior.

“Hey you’re an IT-guy maybe you can make this thing give me the Baby Ruth I paid for”, he vented.

“Well let’s just say we have both met our match.  I call that machine the bully. It has stolen more lunch money from me than any actual bully I ever knew. You are pretty much screwed, I’d say.”

“Hey, so how are you guys coming on the new web property launch for corporate? I keep getting told there are all kinds of technical issues and they are being blamed on network and security engineering.  What’s your take?”

*Quagmire alert*

My internal klaxons are ringing at 120db and my prefrontal cortex just went to maximum self control status to prevent a major career-threatening verbal miscue of stupendous proportions.

“You got me on that one sir.  I just found out about it two days ago when the developers went rogue and somehow overran data center security and were actually installing apps on servers without any change approval or ops guys in the loop.  The only reason I know that the roll out was happening is because they called my desk when they ran in to problems.”

“Really?” He shrieked.

“Your boss said he had you clear the way for them and that you had set all of the firewalls and other network gear up for them over a week ago and that the servers were done three days ago. Are you telling me this is not the case?”

Crap. My prefrontal cortex had just panicked necessitating a cranial reboot.  This was not the time to tax my caffeine deprived brain.

I managed

“Well the network gear was set up, although I did not realize it was for this project.  I still have the servers to complete. They should be easy as soon as the dev and staging team tell me what they need them to do.”

In the back of my head a growing sense of foreboding met up with a wave of nausea as I realized there was no way the launch would work.  No one had ever told me what I was setting the gear up for.  All I had done was set up interfaces, vlans, routing, and trunking.  There were no rules permitting the traffic to and from the various segments the dev guys had mentioned. I had to get out of here…… and fast.