Can IT rise from the ashes of a bad reputation?

I’m delighted, this week, to bring you an interview with Gene Kim. Gene will be presenting one of the opening keynotes at the Innovating IT Service conference, and a workshop. He has been a founder, CTO, and author. Gene loves finding and fixing bottlenecks which impede and frustrate the entire organization, enabling management from each tribe to achieve the greater organizational goals.

1. Your keynote for the upcoming conference asserts that everyone needs DevOps. How do you explain what DevOps is to an IT manager working in a traditional enterprise IT environment?


My definition of DevOps is the following: it is the set of cultural norms and technical practices that enable organizations to have a fast flow of work from Development through Test and deployment, while preserving world-class reliability, availability, and security.

These norms and practices are what enable organizations to do hundreds, thousands, or even tens of thousands of deployments per day. This used to be associated with “unicorn” organizations such as Amazon, Netflix, Google, and so forth. But increasingly, large, complex organizations such as Nordstrom, Macy’s, GE, Raytheon, and even the US Department of Homeland Security are adopting these practices and replicating the unicorn-like outcomes.

Many of us, especially in the service management community, believed very deeply that you couldn’t be agile and reliable at the same time. And yet what we found in our benchmarking of over 14,000 organizations, is that not only is this possible, the only way that you can have high reliability is to be doing smaller deployments, far more frequently.

Here’s what we found in the benchmarking (citation: http://www.slideshare.net/realgenekim/2014-state-of-devops-findings-velocity-conference):

 

* Agility metrics

* 30x more frequent code deployments

* 8,000x faster code deployment lead time

* Reliability metrics

* 2x the change success rate

* 12x faster MTTR

* Organizational performance metrics

* 2x more likely to exceed productivity, market share, and profitability goals

* 50% higher market capitalization growth over three years

DevOps transforms how we work, whether we are in Development, Test, Operations, or even Information Security.

I’ve been studying high-performing technology organizations since 1999, and there is no doubt in my mind that DevOps is something genuinely transformative.

 

2. In The Phoenix Project, Bill Palmer and John Pesche are representative of the ongoing struggle between rapid deployment and security management. What kind of work do organisations need to do to make that partnership work?


This is a great question, Aprill. There are two chronic struggles that we tried to portray in The Phoenix Project: the first was the constant battle between Development and IT Operations, where Development would always want to go faster, but would often cause chaos and destruction downstream. The natural reaction, of course, is that Bill (the VP of IT Operations) wants to slow down the rate of change. Now you have Development and IT Operations at odds with each other.

The other chronic struggle is between the entire organization and Information Security, as embodied by John (the Chief Information Security Officer). John believes, often rightly so, that everybody is more worried about their own work, and never properly integrates security requirements or testing into daily work. Unfortunately, the outcome is that John is always viewed as being in the way, trying to slow everybody down, creating meaningless bureaucracies that sucked the will to live out of everybody in their path.

We all laugh at these situations, but I think for many of us, these situations are all too real and all too commonplace.

The reason why I’m so excited about DevOps is that it allows everybody to achieve goals and outcomes that we didn’t think possible—even five years ago.

3. Speaking of John Pesche, what thoughts did you have to explain his reappearing clean-shaven and helpful after going so far off the rails?

Haha! I’ve actually gotten some e-mails from people who scold me for putting Information Security in such an unflattering light. A friend of mine actually even wrote, “How dare you humiliate our profession. Whether you like it or not, Gene, you are still in information security practitioner.”

In actuality, John is my favorite character. A friend of mine, Jez Humble, said the real “phoenix” in The Phoenix Project is John. He transforms, seemingly overnight, from a shrill, hysterical, bottom-up controls person, to a person who seems to be willing to take risks that the rest of the organization is too scared to make.

For example, John proposes to outsource all of their cafeteria point-of-sale systems, so that there will be no cardholder data for them to protect. He actually reduces the number of security controls, because he realizes that there are downstream manual controls that can achieve the control objectives.

Frankly, I love the fact that John has such a dramatic visual transformation, as well.

 

4. You talk about the need for various IT teams within an enterprise to come together and build “super-tribes” to maximise throughput. To quote Seth Godin, “A tribe is a group of people connected to one another, connected to a leader, and connected to an idea… A group needs only two things to be a tribe: a shared interest and a way to communicate.” What is it about our workplaces that can make these two ingredients so hard to find, and is that a problem limited to IT?

Especially in IT Operations, we tend to have a very functional orientation—departments and silos to concentrate our specialties. For example, we may have a database team, a storage team, a networking team, a server team. And whenever we want to do significant work, like a code deployment, we may have to do 50 different handoffs. Functional orientation is typically done to “optimize for cost.”

The opposite is what we call “market orientation,” that allows us to “optimized for speed.” These organizations tend to be flat and composed of multiple, cross-functional disciplines (e.g., marketing, engineering, etc.), which often leads to potential redundancies but allows us to respond quickly to customer needs. This is how many DevOps organizations operate. In extreme examples, each service team is simultaneously responsible for feature delivery and service support.

You can see the DevOps bias towards “market orientation” in these pithy quotes from two friends of mine:

“The root of most DevOps problems comes from silos: siloed groups, siloed thinking, siloed culture, siloed tools.” — Damon Edwards

“DevOps means caring about your job enough to not pass the buck, wanting to learn all the parts as a whole, and not just your little world. Developers need to understand the infrastructure, Operations people need to understand code, people need to actually work with each other and not just occupy space next to each other.” — John Vincent

 

5. When I talk to devs and ITIL comes up, they roll their eyes and speak of the bureaucratic bottleneck of change management. Is there a good place for ITIL in these conversations or is it time to move on?

Ha! For over a decade, I’ve been a fan of ITIL (IT Infrastructure Library). It describes extremely well the underpinning processes we all need to deliver reliable service. But we all know people who will take ITIL very literally, and put in all sorts of bureaucracies that burdened everybody, just like John the CISO.

Although many people view DevOps as backlash to ITIL or ITSM (IT Service Management), I take a different view. ITIL and ITSM still are best codifications of the business processes that underpin IT Operations, and they actually describe many of the capabilities needed into order for IT Operations to support a DevOps-style work stream.

Agile and continuous integration and release are the outputs of Development, which are the inputs into IT Operations. In order to accommodate the faster release cadence associated with DevOps, many areas of the ITIL processes require automation, specifically around the change, configuration, and release processes.

The goal of DevOps is not just to increase the rate of change, but to successfully deploy features into production without causing chaos and disrupting other services, while quickly detecting and correcting incidents when they occur. This brings in the ITIL disciplines of service design and incident and problem management.

Think of DevOps as the service managers dream: releases are almost completely automated, there is rigorous automated testing that gives us confidence that errors will be caught long before it gets into production, and when errors do occur, we can detect and correct for them quickly.

Moreover, our configuration management database (CMDB) is created for every application and service automatically, and it is always up-to-date, and Development and Operations are working together to fix known errors and pay down technical debt.

My advice: Let’s not get hung up on literal definitions. Let’s figure out how to get fast flow and painless and successful releases.

 

6. How do you recommend organisations transitioning into a DevOps environment keep their support teams up to date on deploys?

All too often in software development projects, Development will use up all the time in the schedule on feature development. This leaves insufficient time to adequately address IT Operations issues. Shortcuts are then taken in defining, creating, testing—everything that the code relies upon, which includes the databases, operating systems, network, virtualization….

This is certainly one of the primary causes for perpetual tension and suboptimal outcomes between Development and IT Operations. The consequences of this are well known: inadequately defined and specified environments, no repeatable procedures to deploy them, incompatibilities between deployed code and the environment, and so forth.

In this pattern, we will make environments early in the Development process, and enforce a policy that the code and environment be tested together. When Development is using an Agile process, we can do something very simple and elegant.

According to Agile, we’re supposed to have working, shippable code at the end of each sprint interval (typically every two weeks). We will modify the Agile sprint policy so that instead of having at the end of each sprint just shippable viable code, you also have to have the environment that it deploys into—at the earliest sprint, so we’re talking sprint 0 and sprint 1.

Instead of having IT Operations responsible for creating the specifications of the production environment, they will build an automated environment creation process. This mechanism will create the production environment, but also the environments for Dev and QA.

By making environments (and the tools that create them) available early, perhaps even before the software project begins, developers and QA can run and test their code in consistent and stable environments, with controlled variance from the production environment.

Furthermore, by keeping variance between the different stages (e.g, Development, QA, Integration Test, Production) as small as possible, we will find and fix interoperability issues between the code and environment long before production deployment.

 

Thank you, Gene!

Archives