How to Run a Successful Continuous Delivery

Did you know that Amazon can deploy new software over 1.000 times per hour?

And this fact has nothing to do with the size of their development and testing teams. Their not-so-secret weapon is embracing the continuous delivery model, and more specifically being early adopters of the DevOps approach to software development.

You may have heard the terms, but can’t put your finger on it?

Well, continuous delivery and DevOps terms do carry similar meanings, but actually, stand for different concepts. DevOps is a wider notion that incorporates all scope of changes in the development and delivery process, communication and collaboration between product management, development, and operational teams, as well as introducing automation for certain software delivery processes.

As Gartner puts it: “DevOps — Not a Market, but a Tool-Centric Philosophy That Supports a Continuous Delivery Value Chain”.

Continuous delivery only tackles the delivery aspect and focuses on introducing more automation in the routine processes to execute them faster, with higher accuracy and higher frequency.

So What is a Continuous Delivery Model?

In simple words, continuous delivery enables you to make changes within your product at any stage without harming the user experience or hampering the product quality. This includes delivering new features, fixing bugs, running experiments or making any server-side configuration changes.

Continuous delivery assumes having the next well-honed processes in place:

• First, the developers break down each task into smaller updates/changes that need to be deployed to the production-like environment. Those are made in version control.
• Next, each time the change is committed, the developer receives instant feedback from automated tests (regression testing, acceptance testing, performance testing and so on). In case, an automated test fails, the developer can manually look into the problem and “patch the hole”.
• After the build passes all the initial tests, additional automated testing may be triggered to ensure further product quality before the feature is released.

So, in a nutshell, continuous delivery assumes creating such an environment that allows deploying software automatically to the production-like environment from system/application configuration stored in the version control and makes it ready for the deployment to production.

Testing and development are no longer two separate stages. These processes go hand in hand and are organically integrated into the delivery.

However, it’s important to understand the difference here between continuous delivery and continuous deployment.

Specifically, continuous deployment is a separate process, which incorporates the practice of deploying every change automatically to the production. New changes are deployed after testing that is typically conducted automatically, without any manual interference from a DevOps engineer.

Continuous delivery, however, assumes that all new changes can be deployed automatically, but not necessarily deployed this way all the time for business reasons.

Source: puppet.com

In simple words, you can practice continuous delivery without continuous deployment, but not the other way around.

For instance, Romexsoft’s continuous delivery pipeline is based on the two-weeks Scrum sprints, meaning the developers can make deployments (including automation testing for new features) every two weeks. The particular benefit here is that the size of your product doesn’t matter – on-demand, you can add new features to enterprise-sized apps, embedded systems or mobile apps.

Yet, the continuous delivery benefits don’t end just here. Let’s explore the matter further.

Why Continuous Delivery: Improving The Product Delivery Lifecycle

Perforce recently conducted a survey of some 600 IT specialists asking them about the benefits of implementing continuous delivery. The following statements yielded the highest scores:

Faster Time To Market. You know how things can be – the Ping-Pong between the testers and the developers during the integration/testing phase of the product delivery lifecycle goes for weeks. Your typical team can spend 20% of their time setting up and polishing their test environments, but the CD framework sets it up automatically for them, specifically if you adopt continuous integration practice.

The baseline principle of continuous integration is that the code goes straight to the central repository after a series of test cases has been conducted. Developers incorporate the testing processes into their daily timeline and run unit tests to ensure that the code stored in the central repository is accurate. The version control system also verifies the code for possible integration problems. The perks here are:

• The code is continuously added and tested, meaning there are fewer chances for failures.
• Developers/testers don’t have to wait till the other team finishes their chunk of work.

According to CA Technologies data, this DevOps practice ensures 20% faster time to market for new services on average, especially if the team uses continuous deployment.

Better Product Quality. Bulletproof quality is wired into your product during the entire delivery process as the team has automated testing tools to discover possible problems within minutes (not days). Extra perk – the morale goes up as the testing team can focus on more advanced tasks such as UX or security testing, instead of catching the basic mishaps.

Additionally, teams using CD pipeline typically experience 50% fewer application failures and achieve higher recovery time. To be precise, they are twice more likely to recover in 10 minutes or less, and on average show a 28.3 minutes improvement over non-DevOps centric teams.

Competitive Advantage comes as a result of the two factors mentioned above. Better product quality plus faster delivery means that companies using continuous delivery can beat their competitors within the market race.

Reduced development costs. To justify the ROI of embracing continuous delivery for businesses, Zend has recently shared some curious data. CD teams tend to spend 50%-70% less time on problem resolution, meaning that as a client you will be billed for less productive hours.

Next, think about this – you are designing a product that will assume the usage of multiple hybrid environments (e.g. on-premises private cloud, such as Google Compute Engine; public clouds, such as Amazon Web Services (AWS). Or you can change multiple hybrid environments to cloud infrastructures (e.g. AWS, Google Cloud). If the release process is controlled manually, your ability to catch bugs and react to changes is somewhat limited.

Additionally, have you considered your total cost of ownership (TCO) when it comes to running the application in production? Data centers can be notorious for fluctuating usage patterns. For instance, it was discovered that during one month, over 76% of computing resources could have been wasted if the company didn’t use an elastic cloud environment.

Migrating your product to the cloud (public/private/hybrid) can save you a pretty penny in the long run as cloud migration can ensure up to 72% reduction in TCO. At the same time, you can scale your product on-demand whenever needed, considering that you will use a CD pipeline for deploying new features.

The Best Practices for Implementing an Efficient Continuous Deployment Process

So how to ensure the success of continuous delivery?

In a nutshell, there are two layering principles you should pay attention to – application architecture and organizational culture.

Your first step towards embracing CD is to start accounting for the two new attributes within your app architecture:

• Testability: Software should be designed in such fashion that most bugs (potentially) could be uncovered by developers using automated testing tools. You should simplify the environment and make sure that most regression/acceptance tests can be deployed automatically without much manual interference.

• Deployability assumes that deployments for different products/services can be performed independently and automatically. These systems can be reconfigured with minimal downtime, meaning that new builds can be deployed on-demand with high frequency.

In other words, your product should consist of easy-to-separate and loosely-coupled modules that could be tweaked and tested separately without causing a significant impact on other components.

But let’s face it – unless you are building an application from scratch your architecture may be way more rigid. The good news is that there’s no point in re-architecting a large-scale product right away either. Instead, you should experiment with the evolutionary architecture paradigm.

It’s one of the go-to continuous delivery best practices for monolith applications, which assumes that we keep the original “block” in place, yet ensure that new app architecture is created with the principles of testability and deployability in mind. As more and more functionality will be added, it will replace the original monolith block as illustrated below:

Image credit: continuousdelivery.com

Next, to analyze your CD pipeline maturity use the next 5-stage model.

Level 1 – Ground Zero

Your company’s delivery process looks the following way:

• Manual testing is conducted right after the development stage is completed to discover problems.
• The deployment process is manual and slow.
• Different teams (developers, testers, managers) seem to chase different goals and have conflicts.
• System integration is cumbersome and occurs only after the development module is complete.
• Change management is ad hoc, or even ignored at all.

Ok, so these are the common pain points and you want to get rid of those once and for all. Here’s how to implement continuous delivery in a step-by-step manner from here.

Level 2  – Time-Boxed Releases

Start with conducting some operational changes (as mentioned in the challenges section). You will need to work towards improving the collaboration between teams and add more clarity towards the entire development process.

One of the best practices here is to introduce time-boxed releases – when the team names the date for a new production-like environment and then manages to it.

Next, ensure that the next processes are in place:

• You have created a clear chain of responsibility. Everyone knows their tasks; timelines and actions required from them at any stage.

• Change management control is introduced. You now have the mechanisms to detect authorized changes.
• Key stakeholders actively participate in the decision process regarding the delivery.
• Some automated acceptance testing is introduced to reduce the load on testers and you are moving towards introducing continuous integration.
• The teams start creating testing environments for products early on.
• Teams work in sprints and manage to meet the planned release date.

The result of this stage should be achieving a reasonable release time box.  At this point, the timeline from the idea stage to product release to the production-like environment can be larger than the business demand, but we’ll deal with this in the next stage.

Level 3 – Improving Quality and Release Time

At this stage, you should work towards embracing regular releases within a defined time frame with the provisional build to the production-like environment in between those. Focus on meeting the next goals:

• You have enough automated test cases to catch and prevent the majority of problems early on, fast and automatically.
• The team has now embraced the trunk-based development with continuous integration of all changes.
• The integrated testing environment is well supplied with cases, and everything occurs mostly automatically.
• Each build goes through the automated unit and acceptance test designed for it.
• The testing team is no longer focusing on regression testing only. All the database changes are well documented and versioned.

The result of this stage is creating a well-oiled release procedure with a defined timeframe. The timeline from idea to release will still remain larger than your ideal anticipations, but we are just half-way through implementing CD. Stay put!

Level 4 – Release On Demand

Your next step is to speed up the time-to-market without compromising the product quality. At this stage, your teams should prioritize the next tasks:

• Your system automatically rejects bad changes in feature control, reducing the number of bugs.
• Automated test cases have been created to cover all the common cases, and are maintained both by developers and testers.
• Deployments to the beta version of your product happen automatically with little-to-no supervision by DevOps engineers and are ready for the deployment to production.
• All the work is delivered in short sprints indicated by the timeline (e.g. every two weeks).

The result of this stage is to have your software in an always releasable state and to create a release time box that is equal to or less than your ultimate business need.

Level 5 – Optimizing The Delivery Process

You have created a proper testing/deployment environment; you have reached an organizational consensus with your teams and key stakeholders. Now, it’s time to make your delivery process more agile and customer-centric.

Put the focus on optimizing the cycle time to learn more from your customers. This could be achieved the following way:

• Database changes are separated from application deployments.
• The architecture allows you to decouple deployment from release (dark launching).
• New requirements capture how the value of the new feature will be measured.
• Teams are now responsible for implementing/gathering those metrics from users.

If the above statements are true, your company now has more room for experimenting/innovating without hampering the entire product and ensure more solution-driven delivery. You are no longer launching just some features – you are testing the waters first, gather user feedback and then go all into fully developing an in-demand feature.

Romexsoft offers DevOps consulting services as a part of product development and helps companies of all sizes to embrace continuous delivery. If you are ready to improve your delivery process and speed up time-to-market, let’s get in touch!