Hello, all my fellow DevOps practitioners and enthusiasts! I recently wrote a book review about The Phoenix Project, a fictional novel that follows an organization’s DevOps transformation, thereby enabling it to escape an impending doom (going out of business) and begin winning in their market once again. Towards the end of the novel, one of the characters asks the VP of technology to do him one favor–to “write a book, describing the Three Ways and Parts Unlimited. Call it The DevOps Handbook and show how IT can regain the trust of the business and end decades of intertribal warfare.” And wouldn’t you know it?! Gene Kim, Jez Humble, Patrick Debois, and John Willis were already writing The DevOps Handbook! Where The Phoenix Project was a fictional novel, The DevOps Handbook starts with a technical breakdown of the underpinning principles of DevOps (“The Three Ways”). It continues with a prescriptive method for implementing DevOps within your organization.
The core chronic conflict
Before discussing the principles of DevOps, the Handbook first covers a significant problem with many organizations that directly impedes their ability to continue winning in their industry. As it turns out, there is a core, chronic conflict within the majority of Technology organizations that most businesses don’t even realize they’ve created. Development teams are incentivized to be nimble and rapidly respond to customer needs by regularly developing and deploying new services and features. However, the IT Operations teams are incentivized to provide stability and reliability for the customers and consumers of production software and services. Without realizing it, the business has created a disastrous recipe where the Development and Operations teams are fundamentally at odds with each other. Devs want to increase the speed of change (by deploying new features), and Ops wants to prevent instability in production (which happens when things change). When a Dev team misses a delivery deadline because a new feature isn’t deployed to production successfully, they almost always blame the Ops team for blocking their progress. Whenever the Ops team is firefighting production incidents that directly impact customers (at all hours of the day or night), they will blame the Dev teams for pushing new changes to production without proper testing.
This culture of blame only causes further distancing between the teams, and silos are quickly put in place to try and “protect” either of the teams from blame. The Dev teams will begin to increase their velocity to prove that it isn’t their fault new features aren’t being delivered to the customer. And Ops teams will devise layers of process and change management to ensure that ANY outage or customer impact can be directly tied back to the new features that were deployed. Before you know it, the entire organization begins in a downward spiral, and the company’s goals are continually being missed because the Technology value stream isn’t optimized and prioritized. And it has been proven that when IT fails, the entire organization fails because this day in age, every organization needs to function as a technology company. So, how do we mitigate this downward spiral and implement the practices required to achieve the organization’s desired outcome?
The three ways of DevOps
The First Way revolves around the principles of the flow of work and emphasizes systems thinking to achieve the global goal of delivering value to the customer. In most cases, this means that a product, service, or new feature is deployed to production so a customer can evaluate it. Since workflows from Dev teams to Ops teams, it is important to think about the entire delivery process and not only optimize any single work center. For instance, if a Dev team can produce more features than the Ops team can deploy, then the customer doesn’t get any added value from the entire system. Systems thinking focuses on optimizing each of the work centers so that flow is increased and requires that silos are broken down. This typically means that the Dev and Ops teams need to begin implementing a close-knit collaboration with a genuine interest in understanding the entire system.
When the Dev and Ops teams begin collaborating, they can quickly identify hardships in each other’s daily work. These hardships create waste, whether that be wasted resources, time, or work, that adds no additional value to the customer. Making these visible makes it possible to put in mitigations to prevent these hardships, which ultimately optimizes the flow of work so the value can be delivered more rapidly.
The Second Way targets the importance of implementing and amplifying feedback loops throughout your entire system. Where the goal of The First Way is to understand the system, the purpose of The Second Way is to improve the overall quality of the products and services that are developed. This is the foundation for continuous improvement and “shifting-left” in a Technology organization, where developers are immediately notified if a new code change has caused an issue. This is done by implementing quality gates in the form of a pipeline of automated processes, where any failure is raised so it can be corrected before ever moving further in the delivery lifecycle.
This is very similar to Toyota implementing the Andon Cord at each of its manufacturing work centers that can be pulled if any worker spots a problem or has an idea for improving the overall system. When this occurs, the issue is swarmed and fixed immediately to prevent that issue from flowing downstream to other work centers, and the worker who pulled the cord is thanked! Applying this concept to software development, when a pipeline stage (or quality gate) has an issue, the entire build process breaks, and the issue must be fixed. This increases everyone’s confidence that the same problem will not arise in a production environment. In theory, this will not only improve the quality of software being developed but should also reduce or even prevent production outages that are directly related to software changes being released, since most issues are detected much earlier in the development process.
The Third Way focuses on organizational culture and enabling continuous experimentation, which leads to improvements built on the learning and knowledge gained from those experiments. Let’s face it; we all work with highly complex systems, and understanding the impact of any change in complex systems is almost impossible. But, that doesn’t mean we can’t create some safety while working within such complicated systems. This is where an organization needs to create a culture that doesn’t reprimand failures but views failing as an opportunity to improve the reliability of the overall system (if we’re going to fail, let’s fail fast). Previously, Dr. Ron Westrum defined a model with three distinct types of organizational cultures. When applied to technology, it has been determined that an organization that has implemented and adopted a “Generative (performance-oriented)” culture fosters continuous experimentation and improvement because blame is eliminated, and fear of failure is abolished. When an outage occurs, instead of seeking someone to blame, a genuine inquiry can help produce countermeasures to prevent the issue from happening again. This is bubbled up through the entire organization so that everyone can learn from failure.
This unlocks an organization’s potential for innovation by allowing people to take risks, test out hypotheses, and, most importantly, to practice regularly within a safe environment without the fear of failure. New ideas are actively sought and tested. Teams are encouraged to introduce chaos into their running systems to discover the potential for new types of failures and fill those gaps to increase the overall resiliency of their complex systems. But none of this can be accomplished unless the value is placed on organizational learning, and the entire workforce feels a high sense of trust while trying to improve their daily routines. As we all know, failure is inevitable. But creating a sense of safety and viewing failure as an opportunity for global learning will ensure that your organization can continue to evolve and keep pace with the continually changing landscape of any market.
DevOps transformation
After understanding the underlying principles that create the foundation for the DevOps movement, many are left wondering how they should get started. The DevOps Handbook covers exactly how and where to start, including how to visualize your value streams and how to identify innovative teams to gain some traction with small successes that will help expand the DevOps philosophy across the organization, building critical mass. Of course, in every organization, you’ll have the early adopters of new technologies as well as the laggards or skeptics, which will need to be converted after first creating the silent majority within the organization. The Handbook also provides guidance on the different organizational archetypes and why most organizations need to transition from optimizing for cost and move towards optimizing for speed (enabling market-oriented teams). This valuable information lays out the path for your DevOps Transformation, including how Conway’s Law effects team communication and coordination, as well as how each team member should avoid overly specializing and move towards being “T-Shaped” generalists.
One of the strengths of The DevOps Handbook is the real-life case studies embedded throughout the book. Companies like Nordstrom, Etsy, Amazon, Target, and CSG, just to name a few, have adopted and implemented DevOps values to turn their organizations around. At CSG, after their transformation was complete, deployments became so frictionless and routine that Ops resources would spend free time playing video games, and the value was constantly being delivered to the customer in half the time. Changes weren’t feared because they worked to create a safe system where deployments could take place easily, and rollbacks worked seamlessly.
In addition to case studies that prove how DevOps can benefit your organization, The DevOps Handbook discusses the technical practices of setting up your continuous delivery pipelines, the benefits of automated testing and trunk-based development, and even the differences between deployments and releases using feature flags. From direction on database changes to blue/green deployments to the use of immutable infrastructure to create production-like environments, The DevOps Handbook provides an invaluable exploration into IT organizations and delivers the resources needed to transform your workforce and become a world-class technology organization completely.
The one question that remains, however, is, “Why should we adopt DevOps in the first place?” The Handbook makes it abundantly clear that technology has already proven to be a differentiator, and the companies that don’t embrace technology as a competitive advantage will fall victim to disruption in their industry. Blockbuster was disrupted by Netflix. The book store and retail industry have been and continue to be disrupted by Amazon. Uber built an app and disrupted the Taxi industry. There is an ongoing struggle for survival, and those organizations that adopt DevOps principles and patterns correctly will have adapted themselves to be able to out-innovate and out-perform the competition whenever customer needs and market demands illicit change. Which, as we all know, happens ALL THE TIME.
Get in touch with us to learn more about how EveryIT can help your organization accelerate innovation and begin your DevOps Transformation.
