/kevinbehr/home

When asked to improve, what is ITs obsession with the dream of total automation of everything?

Is it the ultimate exercise in black-and-white thinking? Given the reality of countless failed automation efforts marked by dead-bodies, is there a better middle road?

This basis of my ensuing posteriori-logic-trap becomes especially apparent when one begins to peel away at the thin veneer that obscures the complete failure of IT at large to operate in the realm of the scientific method.

In my dictionary, automation merely consists of  defined practices, values and expected outcomes committed to code. But I have found that in the crucial connective fascia between; Project Management, DEV, QA and Ops, IT often lacks substantive specification, accurate documentation which often yields non-deterministic untraceable outcomes (action x and y caused effect z).

Ok, this may be a pretty harsh statement, but one that is borne out of over 20 years in IT, working with hundreds of IT organizations that struggle to accurately articulate either the goal or the definition of IT operations.  If nearly every implicit destination defined below those two map coordinates is off by even single digit percentages and we consider the length and breadth of the IT journey, not to mention the height of the weeds we can get caught up in, well let’s just say chronic ITFAIL pain is both aft and on the horizon.

In the great book, Why Smart Executives Fail: And What You Can Learn from Their Mistakes the classic story of GMs “must replace labor with robots” fail is told in brutal hindsight.  The lessons are clear but are an order of magnitude harsher when one considers the lesson that history has now taught us.  Not only was the money they spent on those automation efforts lost to their deficient consitency of practice (automation just accelerated their rate of failure) but they could have purchased Nissan, Toyota, Honda and maybe even Mazda with the money they squandered on robots.

Only after years of trial and error have manufacturers struck a balance between automation and human involvement.  Shigeo Shingo, the first person to document the Toyota Production system author of many amazing books including, Kaizen and the Art of Creative Thinking - The Scientific Thinking Mechanism, formalized this approach by refining the Japanese concept of Jidoka or Autonomation.  Simply put, Autonomation is automation with human intelligence.  This is the direction we need to explore in IT for our command and control systems.

To help define just where the intersection and labor divisions should best occur there are several Toyota Production system terms that are worth investigating as a path to improving your shop’s performance

Muri - Overburden - Is every day an exercise in futility? The email piles up the issues escalated, phone calls from execs, standing daily or weekly outage conference calls? Is your IT organization behind or stuck on projects?  How many of these precious business projects are missing their commitment dates, over budget, under resourced because your team is overwhelmed with unplanned firefighting and drive-bys?

Mura- Inconsistency - Routine tasks and changes are like roulette with a two out of ten ending in unexplained failure that consumes your brightest staff for hours or days?  Is patching or upgrading a fearful event which is marked by all knocking on wooden or even wood veneered objects and the presence of a shaman or holy person to ward off evil fail spirits?

Muda -Waste-All of those operating expense dollars lost to firefighting, audit corrective action drive-bys, shadow IT projects, unauthorized changes and root cause analysis meetings that take weeks to recommend the same trifecta of we need more budget, more staff and more time to focus on proactive tasks?

Right now many IT organizations are looking at Muda or waste in order to drive down costs.  I posit that understanding Muri and Mura would be a much more valuable use of time and ultimately will reduce waste and increase IT throughput

Inevitably the solutions recommended by IT teams to these issues will involve automation or tools.  This is not all bad, but the focus should be on building more deterministic ways of working for humans and considering where automation may help humans interact with their IT infrastructure more consistently.

This process of self reflection or Hansei is important fuel for Kaizen (continuous improvement).  It becomes essential to distill all of the undesirable symptoms of overburden, inconsistency and waste and understand the few root causes that drive them all.

Did you know that the “Just In Time’ concept was pioneered by a group of Toyota Employees? These Toyota team members were lead by Taiichi Ono on a trip to the US in the 1950s to visit US auto manufacturers .  They journeyed to Michigan and walked through Ford plants and were generally unimpressed by the high amounts of inventory they required to operate and the variance in labor output from day to day.

During the visit they stopped by a Piggly Wiggly grocery store and were amazed by their inventory replenishment system that only requested new items when they were sold. From this focus on Kaizen and Hansei they developed what later became the famous Just-In-Time philosophy that has become a pillar of the Toyota Production System.

It is not merely enough to improve in this economy, we are faced with the imperative of only improving the most important functions so as to quickly improve execution and IT throughput. As we set out on our journeys and investigation of other practices let’s make sure we are attacking true root causes of overburden and rework not just merely their undesirable effects.

I think that the Toyota Production System offers us many valuable insights in to building better IT. I find the thinking behind the system to be more enlightening than the practices. I encourage you to view all TPS, Lean and “Best Practices” in this light. Often the answer to “why” is more important than the “what” IMHO.

I will be writing more about the intersections of TPS and IT.  I will be focusing on universal principles, that draw from Goldratt’s Theory of Constraints work, Steven Spear, Taiichi Ohno, Shingeo Shingo, Deming and the 10 years of research Gene Kim and I have done around IT high performance, in the coming weeks.

I loved the early hours at the office.  They were even better after I found out about my own private coffee machine and secret stash of amazing bean.  It was day 3 on the job in my new role and as I gazed out over the gray and cloudy horizon of my office windows, I felt a strange rush of adrenaline. Maybe it was just the coffee, but I felt a charge of voltage in my chest today.  I had entered the proverbial lion’s den yesterday and walked out with a pretty satisfying victory.  Not many execs have the facts to call each other’s behavior out.  I had not only caught my peers red-handed circumventing the rules, I had the data to prove that they were part of the problem that necessitated those very rules.

Still I knew that until we cleaned up the IT side of the street the cadre of spinning project-plates was in real danger of crashing to the ground all at once.  It’s one thing to call foul on my business peers; it’s another altogether to solve the problems that forced them to break the rules in the first place.

When IT gets in the way of the business as bad as this shop has, successful executives (especially the type-A variety) will circumnavigate the obstacles and go direct to the folks that deliver.  I had merely put everyone on notice of one part of the problem by stopping the pressure against the outer walls of IT.  Now I had to figure out why we had so much work in progress but so little being completed.

I picked up the phone and pressed the speed dial button for Rob, figuring I would leave him a voicemail to come see me when he arrived in a couple of hours.

“Good morning sunshine.” He answered.

“Ha-ha, Good Morning Rob!  What are you doing here so early?  I figured I would leave you a voicemail.”

“Since you started I have been here every day at 7am and plan on it until you start coming in later.  I overheard you walking through the office on day one muttering to yourself about how no one is here on time and everyone is gone at 4:59.  So if you are here, so am I.”

This delighted me beyond words.  I do not equate long hours with high performance, actually just the opposite, but I knew I needed at least two hours a day of undisturbed planning and reflecting time and I was usually out of the office by 6:00pm anyway.  I really liked this Rob guy and it felt good to have a partner that not only had my back but was at my side when I needed it.

“Great news Rob, you just made my day.  Now get down here stat.  We need to plan.”

“Be there in a sec Chief.” And we hung up.

Sixty seconds later Rob knocked on the open door of my office and walked in.

“What’s on your mind?” he queried as he pulled up a chair in front of my desk.

“I have been thinking about the meeting yesterday and how we had better come up with a plan to clean up our side of the street in this whole project situation.  I feel like I can hold back the water from my peers for a while but I need to get our execution up to par. ”

Rob was nodding with every word

“Yeah you were fun to watch in the board room yesterday, but if you don’t feed the sharks some meat they will turn on you out of pure hunger. By the way you did realize when I gave you the list of 100 active projects from my discovery interviews they were in addition to the 39 approved projects?”

“You have to be kidding!” I exclaimed.

“I wish I was, Phil.  I even combed the list and found any projects that were really just tasks or sub-projects that belonged to an approved capital project. So that list is fully in addition to the approved list.  Your team is working on 139 projects. ”

“Wow.  This is not a small number” I reclined in my chair and let out a loud exhale. 

“How do you let things get this out of control?” I wondered aloud.

“This place is amazing, I have never seen anywhere like it.” Rob chuckled.

“I wish that were true, Rob.  It is depressingly all too common.  Every shop I have been in has struggled with this very issue.  Some worse than others, I will admit but I think this number of projects may take the cake. I mean, how does a CIO get so far removed from understanding what is actually going on inside his own shop?  ”

“His door was closed a lot.  His two lieutenants were essentially yo-yo’s for his peers.  I mean they were the spinning end of the yo-yo and the VPs were working them.  You have some issues with these two guys.  I think they are underwater and have no idea what they need to be focusing on.  You need to deal with this quickly.  You have a clear vision of what needs to happen and in many cases they are telling your team to do things that fly in the face of your intentions.”

I knew Rob was spot on here.  Not only were they often clueless but they were doing a lot of harm to both the credibility of our department and themselves. I had already been confronted on day two of my post by two of my peers who made it clear they thought I should cut both of them loose to pursue other career options.

I really believed that my VP of IT ops and VP of Engineering meant well but I also knew that they were threatened by me, by the mere nature of my position, and plans to bring sunshine to the way we were working. In every conversation I had with them, I noticed the deer in the headlights look in their eyes as I further painted my plans.

I arrived at the boardroom five minutes early so I could get my thoughts together after the nearly 2 hour barrage of background information from Rob.  A week prior I had also asked him to account for each IT department’s workload across projects and support related activity.  While doing so, he had gathered some very interesting data.  From the budget worksheets I pulled a list of approved and funded IT capital projects, of which there were nearly forty, but the list of project activity from the IT team was alarming. If I was reading this right there must be over a hundred IT projects in progress.  My mind was instantly flooded with synapses. Where were these projects coming from?

I thought that having forty capital projects was overload, but over a hundred was just ridiculous.  I now had proof of some serious shadow IT going on.  The question wasn’t who in the boardroom was guilty, but rather, with this number, who was innocent! I started to make tick marks against projects outside of the approved list that were easily attributable to marketing, finance or operations just so I would have an idea of who the worst offenders were.  I did this until the room began to fill with my peers. 

We were all waiting on Paul, our CEO, when I overheard our CMO, Skip Sorrenson, complain about IT being in the way of his marketing data warehouse project. He wanted to hire new staff but I had told HR to hold off on any new IT hires until I could figure out who and what were needed.  According to Rob, this project had been a money pit since day one and had sucked up millions of dollars in capital and had distracted at least a dozen team members between marketing and IT for over five years.  The kicker was that it had delivered zero value to the company.  I knew this was a touchy subject with all and that I would need to get my arms around this project and rescue it fast.

“Ok everyone, sorry to keep you waiting, my call with the street went a little over.  I had to explain our crazy high EBITDA again for the third quarter in row. Thanks Skip.” He joked dryly to a chuckling-murmur in the room. Skip was obviously a rising star in the company and due to his marketing prowess we were enjoying unprecedented quarter over quarter profits.

I heard the conference room door close and noticed my Deputy CIO slip into a chair along the wall in the back of the boardroom.  I told him that he should be here in case I needed backup and to document any deliverables I might get assigned.

“None of us has any time to waste, so let’s talk about IT projects.  Everyone here knows Phil, our interim CIO, by now.  Phil I know you haven’t had much time but can you give us an update on the major capital projects?  I would like to know, first of all, how much of your capital allotment budget has been spent, where the completion status is for the projects, and then answer any questions anyone else might have.”

“Sounds reasonable” I replied confidently, although not sure why.

“Well it is now the third quarter and we have spent less than 25% of our capital budget.  I have some serious questions about the way IT projects are run and why our completion rate is so low. I certainly understand the concerns, regarding IT project effectiveness that many of you raised during my one on one interviews with you a few weeks ago.

It seems that we have far too many IT project planes taking off and landing with little to zero tower oversight. I compiled a list of 39 approved capital projects, most of which are major, by the way. I then sent Rob McNunzio on a bit of a skunk works project to see what my team was actually working on. The results are preliminary but it looks as if the actively worked project list numbers well over 100.”

*Crickets*

I made stern eye contact with all of my peers shooting the I-know-you-are-killing us-all look, followed by a smile.

“Truthfully I was going to come to you today and say that this list of 39 capital projects needs to be prioritized and paired down to 10 major projects and 5 mid size projects just to get things under some sort of control. But after finding out about over 100 projects being active I would say we have a fundamental problem here, and by here I mean in this room.”

I had just thrown down in my first senior leadership team meeting.  I had the worst performing department in terms of credibility and satisfaction ratings, I was in my post less than a week, my department was in the way of company strategy execution, and I had just went type-a on a bunch of sharks…in the shark tank.