What Is Continuous Delivery?
Continuous delivery is the ongoing process of building, testing, and delivering improvements to software code and user environments with the help of automation tools. Here’s a look at the Continuous Delivery (CD) pipeline and the stages involved.
Continuous Delivery and the Future of DevOps
As companies struggle under the human capital expense of manual software releases, the drive to automate processes and achieve CD models takes on increased urgency. Closing the gap between ideas for new software and putting it in users’ hands has become critical to business survival.
Moving to a CD model brings opportunities for faster customer interaction along with reduced inefficiencies and costs. But with this increased speed comes complexity. DevOps teams must consider a plethora of factors – like software dependencies and security relationships – before a CD process can work. Chief among these concerns is discovering the many ways a release won’t work through exhaustive failure testing prior to release.
Below is a look at key factors to consider when transitioning from current software release state to future CD state, and how to plan to fail so you can ensure success.
In their seminal work, The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations, author Gene Kim and his peers assembled the working manifesto for modern, forward-leaning continuous delivery. The authors outline the “Three Ways” of constantly improving, testing, and delivering software improvement.
The First Way – Principles of Flow
The Principles of Flow focus on the quick, efficient flow of work from Development to Operations. Flow is derived from Lean methodology and centers of methods for speeding work within and between teams and achieving the DevOps mantra of automating more and delivering quickly.
Understanding the Principles of Flow is key to achieving and improving continuous delivery. The DevOps Handbook stresses these keys to modernizing your workflow processes to improve CD.
- Make Work Visible. Identify and resolve flow bottlenecks through visualization tools like a kanban board.
- Limit Work in Progress (WIP). Reduce the amount of ongoing projects to the barest minimum, focusing more efforts on resolving problems keeping a project in the WIP phase. The Handbook quotes author David Anderson’s famous insight, “Stop starting. Start finishing.”
- Reduce Batch Sizes. To illustrate why small batch sizes are faster and more efficient. The Handbook cites the example of envelope-stuffing outlined in Lean Thinking: Banish Waste and Create Wealth in Your Corporation. Say there are 30 envelopes to stuff, seal, and send.
- A large batch size approach involves first folding and stuffing 30 pages of paper, then sealing 30 envelopes, then sending each. But an error in any of those phases can derail the entire delivery process.
- A small batch approach works on a per-envelope basis. In other words, complete the process for the first envelope, and move on and fully complete the second, third, etc. By reducing batch sizes, CD speeds up dramatically.
- Reduce the number of handoffs. Integrate teams as thoroughly as possible to reduce interdepartmental handoffs, where critical insights can get lost in the shuffle.
- Continual Improvement. A constant, aggressive cycle of identifying system constraints and throwing every effort at resolving them unclogs bottlenecks in your CD pipeline.
- Eliminate hardships and waste in the value stream. Inefficiencies like incompleted tasks, extra and unnecessary steps, defects in production all slow or halt the CD process. The Handbook cites models for identifying and removing these barriers.
The Second Way – Principles of Feedback
While the First Way focuses on creating work and delivering to end users, the Second Way stresses the inverse need to let the workers know how their product is being received through feedback. Telemetry – the automated process of collecting, ingesting, and applying data (from logs, events, and other metrics) – is critical for accurate, continuous feedback in real-time.
The core technical practices of feedback include:
- Create telemetry to enable fast feedback, see and solve problems;
- Analyze telemetry to anticipate problems;
- Integrate feedback to improve quality and deploy safely;
- Use hypothesis and A/B testing to achieve the outcome your customers want;
- Review and coordinate to reduce risk.
Telemetry can collect enough raw data to swamp DevOps teams, but with these approaches and tools it can be put to invaluable use.
Apply your telemetry practices to accelerate CD with these feedback principles.
- See problems as they occur. A fix can’t begin until a defect is identified, and since DevOps teams often work on different, interlocking aspects of software releases the problems can stay hidden. Ensuring active feedback and feedforward loops between ownership parties is key.
- Swarm problems to build knowledge. Even if a fix is minor, resolve it before moving on. By throwing every resource at eliminating problems teams develop a more intimate understanding of the product and the CD cycle.
- Keep pushing quality closer to the source. By this the authors mean that workgroups should stay as physically and operationally close to their work as possible. A feedback flow that requires approval from distant authorities before changes can be made will be slow and inefficient. Organize and empower teams to keep close ownership of quality control of their aspects of a project.
- Enable optimizing for downstream work. Avoid compartmentalization where teams divvy up critical product factors like stability, security, and others. Focus on coming as close to perfection across all these markers before passing work on. This will ensure that foundational cracks aren’t revealed in later pipeline stages, interrupting CD.
The Third Way – Principles of Continual Learning and Experimentation
Encourage experimentation and constant testing and refinement at the individual level to promote continuous learning. Injecting problems into the system primarily serves to bolster resilience, however it’s also a common means of experimenting and. Doing so will then carry over to the team level where the combination of improved resilience and the collected knowledge benefits the whole organization. Critical focus areas include:
- Enabling a learning safety culture. In worst cases organizations can punish testing and failure, resulting in a culture of fear that impedes workflow, feedback, and delivery. A ‘generative’ organizational culture encourages and rewards teams for actively seeking problems and banding together to fix them.
- Institutionalizing daily improvement. The natural impetus to avoid change often slows CD systems. Developing an atmosphere that stresses daily improvement as equally important to daily work leads to teams that work better and faster.
- Turning local discoveries into global improvements. Knowledge and skills, no matter how remote or small they seem in the CD pipeline, must be compiled and shared across the entire system. This prevents problems from duplicating in other points of the CD process.
- Injecting resilience patterns into daily work. Fail early and fail often, as the DevOps saying goes. Simulate disaster by subjecting systems to fatal stresses to learn better how to react to failures. The Handbook cites the Netflix ‘chaos monkey,’ which intentionally kills processes and servers during production hours to force teams to build resilience into their work approach.
- Reinforcing a learning culture through leadership. Leaders must set the example for curiosity and knowledge growth. By taking an active role in finding and overcoming obstacles and sharing knowledge with employees, leaders build a culture of continual learning and experimentation.
The Benefits of Experimentation and Continual Learning
Focusing on these technical practices at the individual level will result in
- Stronger products
- Quicker responses to problems
- More resilient teams
- More satisfied team members
- Improved daily work
The Continuous Delivery Pipeline
The continuous delivery pipeline is process of building, testing, and deploying software in short, fast cycles. A software package proceeds through the stages of the pipeline, being treated at each as a ‘release candidate’ and thoroughly evaluated for weaknesses before advancing to the next stage. Let’s take a look at four key segments of the CD pipeline.
1) Commit Stage
The first stage of the pipeline acts as an assembly and gatekeeping space. During commit, code is checked out from its longer term repository and placed in a temporary self-contained filesystem, where it is compiled and then undergoes a series of basic tests to make sure it’s ready to move ahead. A successful commit stage ends once these tests are passed. The code is then converted into a self-contained filesystem and passed to the next stage in the pipeline: automated testing.
2) Automated Testing
During this phase automated scripts build an environment that replicates production and launching secondary testing to stress test new builds.
Sometimes called sanity testing, this pipeline segment examines basic functionality like the ability to launch the app or how buttons respond in a near-production environment, and the app is deemed ready or not to move on to the next pipeline phase. Automated testing can quickly identify flaws that would otherwise take extensive human time to unearth.
3) Exploratory Testing
Once the package has made it this far in the CD pipeline it undergoes its real trials. In this phase the developers undergo an evolving cycle of interacting with the software, testing how it behaves and making improvements by refining code.
While basic functionality is tested in earlier phases, this is where DevOps teams really see their ideas in action and ensure the user experience matches their vision. When exploratory testing has been satisfactorily completed the package is ready for the final phase.
4) Production Deployment
The big show. The go-live. The CD pipelines ends once your users first experience the new deployment. Here’s where all your work, planning, and teamwork will be put to the ultimate test.
Focusing on these principles and structuring them in the Three Ways ensures your CD pipeline includes the most progressive, forward-leaning approaches.
What Continuous Delivery Does for You
For Manuel Edwards, the chief build engineer and deployment manager for E-Trade, the challenges of continuous delivery were even more complex. As one of the world’s oldest and largest online brokerage firms, DevOps teams at E-Trade are up against the strictest regulation in the industry and constant, real-time change that impacts delivery.
Edwards constructed a CD pipeline through four continual focus areas:
1) Configuration Management
Edwards stresses the importance of configuring the many environmental components, but not developing overly restrictive policies that slow change. He dubs this dilemma ‘configuration-itis.’ Using a mountain climbing analogy he compares good configuration management to a base camp that prepares and equips climbers to ascend.
2) Change Propagation
Focusing on avoiding too complex a value stream mapping process, E-Trade stressed deep analysis of actual user flow and intent. Manual helped build a change testing environment that mirrored as closely as possible the environment and stresses of previous production lessons. In DevOps only production matters to the end user, so change testing and propagation should strive to look just like production.
3) Software Engineering Disciplines
Edwards cautions that the right engineering approach to CD isn’t just a matter of human resources creating a new team and staffing it. Though fresh ideas may help, CD organization comes from linking the necessary parties with the right communication links.
Edwards recommends consistent contact and the development of a Do’s and Don’ts Manifesto – but stresses the importance of letting that document grow organically from the DevOps team so it captures their knowledge rather than telling them how to do their work.
Automating potentially time-intensive tasks like load balancing configuration, reporting, and other details will speed your CD pipeline. Edwards developed an automation workflow that is wide with human verification at the bottom but increasingly employs automation to shorten deployment times.
By developing the above process and disciplines, Edwards and his team were able to increase delivery to more than 200 successful deployments per month by September of 2016, representing an increase of over 1200 percent.
From IT Ops to DevOps: Transforming Processes and Teams
Not all DevOps communities know the computer scientist Melvin Conway, but most are familiar with his maxim: “Any organization that designs a system will inevitably produce a design whose structure is a copy of the organization’s communication structure.”
Put simply, Conway’s Law states that a flawed delivery process will produce a flawed product. Plan carefully and holistically to build teams and workflows aimed at overcoming any obstacles, especially the unforeseen ones.
Five patterns for highly successful DevOps teams.
- Plan for failure. As mentioned above, failure points are inevitable. The art of CD is planning to fail early, quickly, and often in code testing phases, replicating every possible scenario to stress test new releases before integrating them into a live environment. Software builds released with few or no failures likely haven’t been tested strenuously enough.
- Automate as much as possible. By using programming loops to simulate user experience, some of the most time-consuming aspects of CD can be almost eliminated. Test basic functionality, software interdependencies and other fine details with smart use of automation. But note that at this point full automation is still a dream, and that human intelligence plays the essential role in seamless rollouts and quality assurance.
- Use telemetry to gather feedback. So many complex interactions occur in a CD model that tracking one of them gone awry is impossible without the right tools and tracking approach. In other words, telemetry and feedback. Hidden needles in haystacks of data are among the most common clogs in CD pipelines. Troubleshoot effectively by ingesting all relevant log data and structuring it in a way that’s accessible and allows you to act on intelligence quickly.
- Build your team. Changing systems never comes without resistance. Help your team evolve by incorporating the mix of tenacity and creativity. Automatic tasks will help you perform routine and tedious tasks at high speed, but when problems arise they will need solving by real people. Make sure yours mesh together and pool their skills to ensure joint ownership of your CD model.
- Security as a constant. Interdependencies between apps, services and components should be considered before a CD process begins, not at the end. This will save countless problems that arise from security inconsistencies in the build processes, reduce the amount of telemetry (error reporting) to analyze, and make it easy for your team to focus on progress.
Set Continuous Delivery into Action
Continuous Delivery is a complex, intimidating process, one until recently thought to be important only for massive companies with millions or more interactions per day. But in an evolving world where speed and experience are essential, CD is increasingly necessary to remain competitive. If your organization isn’t constantly delivering faster, better product it is already falling behind!