/kevinbehr/home

When asked to improve, what is ITs obsession with the dream of total automation of everything?

Is it the ultimate exercise in black-and-white thinking? Given the reality of countless failed automation efforts marked by dead-bodies, is there a better middle road?

This basis of my ensuing posteriori-logic-trap becomes especially apparent when one begins to peel away at the thin veneer that obscures the complete failure of IT at large to operate in the realm of the scientific method.

In my dictionary, automation merely consists of  defined practices, values and expected outcomes committed to code. But I have found that in the crucial connective fascia between; Project Management, DEV, QA and Ops, IT often lacks substantive specification, accurate documentation which often yields non-deterministic untraceable outcomes (action x and y caused effect z).

Ok, this may be a pretty harsh statement, but one that is borne out of over 20 years in IT, working with hundreds of IT organizations that struggle to accurately articulate either the goal or the definition of IT operations.  If nearly every implicit destination defined below those two map coordinates is off by even single digit percentages and we consider the length and breadth of the IT journey, not to mention the height of the weeds we can get caught up in, well let’s just say chronic ITFAIL pain is both aft and on the horizon.

In the great book, Why Smart Executives Fail: And What You Can Learn from Their Mistakes the classic story of GMs “must replace labor with robots” fail is told in brutal hindsight.  The lessons are clear but are an order of magnitude harsher when one considers the lesson that history has now taught us.  Not only was the money they spent on those automation efforts lost to their deficient consitency of practice (automation just accelerated their rate of failure) but they could have purchased Nissan, Toyota, Honda and maybe even Mazda with the money they squandered on robots.

Only after years of trial and error have manufacturers struck a balance between automation and human involvement.  Shigeo Shingo, the first person to document the Toyota Production system author of many amazing books including, Kaizen and the Art of Creative Thinking - The Scientific Thinking Mechanism, formalized this approach by refining the Japanese concept of Jidoka or Autonomation.  Simply put, Autonomation is automation with human intelligence.  This is the direction we need to explore in IT for our command and control systems.

To help define just where the intersection and labor divisions should best occur there are several Toyota Production system terms that are worth investigating as a path to improving your shop’s performance

Muri - Overburden - Is every day an exercise in futility? The email piles up the issues escalated, phone calls from execs, standing daily or weekly outage conference calls? Is your IT organization behind or stuck on projects?  How many of these precious business projects are missing their commitment dates, over budget, under resourced because your team is overwhelmed with unplanned firefighting and drive-bys?

Mura- Inconsistency - Routine tasks and changes are like roulette with a two out of ten ending in unexplained failure that consumes your brightest staff for hours or days?  Is patching or upgrading a fearful event which is marked by all knocking on wooden or even wood veneered objects and the presence of a shaman or holy person to ward off evil fail spirits?

Muda -Waste-All of those operating expense dollars lost to firefighting, audit corrective action drive-bys, shadow IT projects, unauthorized changes and root cause analysis meetings that take weeks to recommend the same trifecta of we need more budget, more staff and more time to focus on proactive tasks?

Right now many IT organizations are looking at Muda or waste in order to drive down costs.  I posit that understanding Muri and Mura would be a much more valuable use of time and ultimately will reduce waste and increase IT throughput

Inevitably the solutions recommended by IT teams to these issues will involve automation or tools.  This is not all bad, but the focus should be on building more deterministic ways of working for humans and considering where automation may help humans interact with their IT infrastructure more consistently.

This process of self reflection or Hansei is important fuel for Kaizen (continuous improvement).  It becomes essential to distill all of the undesirable symptoms of overburden, inconsistency and waste and understand the few root causes that drive them all.

Did you know that the “Just In Time’ concept was pioneered by a group of Toyota Employees? These Toyota team members were lead by Taiichi Ono on a trip to the US in the 1950s to visit US auto manufacturers .  They journeyed to Michigan and walked through Ford plants and were generally unimpressed by the high amounts of inventory they required to operate and the variance in labor output from day to day.

During the visit they stopped by a Piggly Wiggly grocery store and were amazed by their inventory replenishment system that only requested new items when they were sold. From this focus on Kaizen and Hansei they developed what later became the famous Just-In-Time philosophy that has become a pillar of the Toyota Production System.

It is not merely enough to improve in this economy, we are faced with the imperative of only improving the most important functions so as to quickly improve execution and IT throughput. As we set out on our journeys and investigation of other practices let’s make sure we are attacking true root causes of overburden and rework not just merely their undesirable effects.

I think that the Toyota Production System offers us many valuable insights in to building better IT. I find the thinking behind the system to be more enlightening than the practices. I encourage you to view all TPS, Lean and “Best Practices” in this light. Often the answer to “why” is more important than the “what” IMHO.

I will be writing more about the intersections of TPS and IT.  I will be focusing on universal principles, that draw from Goldratt’s Theory of Constraints work, Steven Spear, Taiichi Ohno, Shingeo Shingo, Deming and the 10 years of research Gene Kim and I have done around IT high performance, in the coming weeks.

Have you ever read “The Mythical Man-Month: Essays on Software Engineering“?

If so you are a rare bird (or over 45) in today’s tech zoo.

I wish I could drop this book (among many others such as The Goal, Critical Chain, The Machine that Changed the World, Great Boss Dead Boss, and Chasing the Rabbit  - see the right side column to buy them!) out of airplanes and organize study groups with my IT; operations, security, audit, project management, executive management and yes especially development brethren.

I was reflecting on this text and several other seminal management books I have collected over the years and I decided I needed to write a series for my blog on this topic. As of late Gene Kim and I are working on some amazing projects for fun and exciting internet 2.0 companies. We have been writing a new IT management novel with George Spafford (think “The Goal” for IT) , and helping a bunch of IT shops solve the quality, reliability and cost reduction realities that have now become must-do (all at the same time mind you).

Just yesterday we began working with long time collaborator and joyful subversive, Eileen Forrester, from the Software Engineering Institute at Carnegie Mellon University.

We are shaping up a few writing projects for the newly released CMMI-Service model.  Eileen has been in charge of getting the new model released on time and has actually over-performed and it is out much earlier than expected for the first time in history.

I relish the opportunity to exchange ideas and approaches for improving the delivery of services with folks from all sectors of the economy (especially those disciplines outside of the IT world).

Many of these industries and business disciplines have a long history of systems design and improvement that reads like rich history.

Where is this similar lineage taught and practiced for IT today?

As I work with many IT executives and teams around the globe I often find myself wondering if the collective “we” has forgotten more than we have learned up to this point.

For a community that not only leans towards scientific interest our entire profession exists on the shoulders of science (knowledge and technology are the output) so little of the way we actually do IT is built on the scientific method or around what I have dubbed the “System Building and Continuous Improving Arts” (SBCIA).  We often are just reacting rather than treating every request as a hypothesis and making sure we have built a repeatable response that correctly satisfies all of the implicit assumptions in the request.

To better explain what I am talking about I will use the Japanese term RYU, which denotes the flow of water.  The idea represents formal Japanese traditions such as the Martial Arts.  In Karate fluidity and finding one’s path are compared.  Water will find its own path in balance with the terrain it is surrounded by, just as the Karate student must advance through his or her path by practice and training.

Bringing me to the point of my concern in this post…

Much of “the path” for IT practitioners and managers of all levels travels through the acquisition and appropriation of myopic technical domain knowledge (yes even auditors). In other words “what we must know” so “we” know how to react in a given technical situation”.

What is missing?  What we need to understand holistically (see) to decide how we should work together as a system .  This information is both timeless and too time consuming to amass through trial and error in our own life experience. The old adage applies here “Time may be the best teacher but it kills all of its pupils”. In order to amass the collective wisdom about how to see the work in order to build an effective system to do the work, we need the collective wisdom of our predecessors and our current community at large. We also need to look outside of our industry and find analogs to improve our execution, quality and safety with larger and scientifically validated levers. Stop chasing consensus based “Best Practices” and start purpose-building systems around scientifically proven practices.

There is much room in IT to build a new flow of knowledge about the “System Building and Continuous Improving Arts” (SBCIA).  This economy is shaping up to be the ultimate Dojo for us to do so.  Let’s just say performance is no longer optional but survival is.

So pick up a copy of “The Mythical Man-Month: Essays on Software Engineering” and start a study group and learn from your ancestors.  Create a culture of Kung-Fu, a Chinese term that means “individual accomplishment or skill cultivated through long and hard work” .  Set the tone with your team and define skill from a systemic and improvement standpoint rather than merely a technical one.

Check out the books in the far right column on this blog to find a handy DOJO compendium to get you started on this journey.  If you are serious about improving IT flow, reaping the drastic cost savings, increases in project execution and infrastructure reliability that result, feel free to contact me about mentoring and staff coaching.  It really does make a huge difference!

If you have stumbled upon a nugget on your journey please let me know I am @kevinbehr on Twitter - join the conversation!

*Put down the Blackberry. Step away from the Blackberry please.”

When I read this article about Obama’s Blackberry use, I rolled my eyes at some of the intellectual stretches it was taking. While I agree with the premise and I do certainly understand the Blackberry-jones (nervous separation anxiety and the compulsion to thumb it when the little red light blinks) premise, I was far more interested in the work that Gloria Mark was engaged in.

“When Gloria Mark, professor of informatics at the University of California, Irvine, shadowed employees at two high-tech firms, she found that the average worker spends only 11 minutes on any given project before being interrupted and asked to do something else. IT workers have it worse, switching attention every three minutes, on average.”  - Newsweek Magazine, 2/16/09

Over the last ten years, Gene Kim and I have been inside of and benchmarked hundreds of IT organizations, conducted numerous research studies and written a LOT about this very topic of unplanned work. We have come to believe that, largely, IT management as a whole has not built an effective system for “Doing IT.”  This study from the University of California dovetails with some of our findings from benchmarking nearly 1000 IT organizations.

Namely, the first finding that Unplanned Work, which by definition is the IT equivalent of task assignment by drive-by-shooting (fire-fighting outages, urgent security patching or compliance related work, essentially anything that takes a worker off of the planned task list of things that are important) is virtually non-existent in high performing shops compared to everyone else.  This is evident both by the low amount of it (expressed as a percentage of labor opex often as low as five to ten percent, compared to the rest at twenty to fifty percent of labor opex lost to heat) and by the advantage high performers have over everyone else in project execution. High performers get up to 8X the projects done, 14-16x times the infrastructure changes with one-half the change failure rate.  Also, their mean time to repair an outage is as low as 1/10 the time of low performers!  All of this creates higher availability and the ability to execute more projects in a more controlled manor with less negative business impact.

Oh yeah, did I mention that they do this with as little as 1/4 of the system administrator staff when compared to low performers?

By building an effective execution flow system it is not only possible, it is probable that you will outperform your competitors and actually help your company meet its goals.

Over the last ten years Gene and I have learned:

1.       Constant fire-fighting  and uncontrolled change weakens infrastructure and creates security problems,

2.       The hero culture that ensues accelerates the rate of outage entropy because little information and visibility in to the known good state exists for others to draw on,

3.       Your best and most talented people are stuck on the fire-fighting line instead of solving business problems,

4.       Your ability to demonstrate proof of compliance diminishes with the more outages you have,

5.       More fire-fighting = less project and planned work = less credibility with business peers = shadow IT = failure,

6.       This can all be fixed by building a system based on Flow and controls.  No, I am not speaking of a never-ending-death-march of process analysts and skies darkened by consultants. I am talking about building a deliberate discovery based system that illuminates what is actually happening versus obscuring it. This type of system learns from every mistake and seldom repeats the mistakes made in the past.

Sound interesting?  If this is the direction your organization wants to move - Dramatically Lower Cost structures, Drastically Increased Project Delivery capability, and Rock Solid Reliability, then contact me here and let’s talk about how we can help!

We are currently doing just this for some of the most advanced and largest IT operations in existence today.

Mich Kabay wrote a very nice review of Visible Ops for Network World. 

He also wrote a great review of Visible Ops Security edition.

It’s 1999 about seven o’clock on a rainy evening and Gene Kim and I are sitting in the corner at Pazzos in Portland Oregon. By now the windows are fogging we are on our second or maybe fifth round of Widmer Hefewizens. Our voices are very animated and we are getting more excited by the minute. I remember feeling like I had met my mental twin.What were we talking about? Mostly about how screwed up IT was as a corporate function. Management made little sense at times. We traded IT disaster stories and even what we thought were some solutions. I parted feeling like I had made an important friend but not knowing where the friendship would lead.

Little did I know the significance of that evening in terms of the research projects, books and breakthroughs we would spearhead as a result of our friendship and collaboration over the next decade. We were put together by a mutual friend, Jon Speer. Jon felt like Gene and I spoke the same language and felt we needed to meet. Thanks Jon, you were dead on!

Over the course of several meetings we performed many brain and book dumps on each other. We both described the work we were doing for our respective CEOs and found a important vein to begin working on together.

In both of our day jobs we had customers that were stand outs because they behaved differently and spoke a different language. In my business these clients were more profitable and required less man hours to support. In Gene’s business these customers were using his software (Tripwire) for operations as opposed to security. We began to study what we suspected were high performing organizations and see what made them so radically different than the others.

Two years passed and we progressed from suspicions to deeply held beliefs and developed what is now known as the Visible Ops methodology. We found four areas of focus that separated these special teams from the rest. We normalized all of their self derived playbooks and terminology back to the common parlance and taxonomy of ITIL. We then laid out an easy-to-follow approach to stemming IT chaos and firefighting with four simple projects. After completion of the methodology we co-founded the IT Process Institute to continue our research work and teamed up with George Spafford to write the Visible Ops Guide.

Since the publishing of Visible Ops it has sold nearly one hundred thousand copies. The second book in the series “Visible Ops Security” is selling very well. Our empirical research and benchmarking has helped hundreds of IT organizations answer the questions of “where should we start?” when it comes to process improvement and ITIL adoption. We have isolated the 80/20 of IT controls and processes and know just what high performers do to get their amazing results.

Over the last decade Gene and I have spoken at dozens of conferences, hosted countless roundtables with CIOs, CSOs and Chief Audit Execs, consulted CIOs, VPs, taken interim executive roles, consulted large IT turn-around efforts, written dozens of articles and benchmarked hundreds of IT organizations. It has been an amazing ten years but I am most excited at what lies ahead. We have not only proven our deepest beliefs and confirmed our restless suspicions but we have done it with empirical research and science.

In the next blog entry I am going to examine a topic all too familiar to IT executives and staff. It is the “hidden sucking sound” present in all IT shops whether large or small. High Performing shops have much less of it and manage it very carefully. What am I talking about? Unplanned Work. You may not know it by this name but if you have worked in IT you will recognize this all too well.

till then-

kb