DevOps and continuous delivery: releasing small software changes safely and often (Software and Cloud Foundations for Business Leaders) | READOS

Many businesses experience software releases as stressful events. Changes accumulate for weeks or months, a release date is set, testing is squeezed, the update is installed late at night or over a weekend, and something breaks. Users discover problems on Monday, fixes are rushed out and confidence falls. The natural reaction is to release less often, with more checks, which makes each release larger, riskier and more stressful still.

DevOps and continuous delivery reverse that logic. Instead of releasing large batches of change rarely, teams release small changes frequently, using automation to build, test and deploy every change the same way, and monitoring to detect problems quickly. Releases become routine rather than events. The ideas will be familiar to anyone who knows lean manufacturing: small batches, built-in quality, flow, fast feedback and continuous improvement, applied to software.

This article explains what DevOps means, how continuous integration, continuous delivery and continuous deployment differ, what a delivery pipeline does, techniques for releasing safely, monitoring and observability, the four measures widely used to judge software delivery, learning from incidents and how small businesses and their suppliers can apply these practices. It is part of a series on software and cloud foundations for business leaders.

What DevOps means

Traditionally, software organisations separated development, the people who build and change software, from operations, the people who run it in production. Developers were rewarded for delivering change; operations staff were rewarded for stability, and therefore tended to resist change. Handovers between them were slow, and when things went wrong, each side blamed the other.

DevOps, a term that emerged around 2009, describes a set of cultural and technical practices that bring the two together. Its core ideas include:

shared responsibility for both delivering change and running software reliably;
automation of building, testing, deploying and configuring systems;
small, frequent changes rather than large, infrequent ones;
fast feedback from tests, monitoring and users;
measurement of delivery performance and system health; and
continuous learning, especially from failures.

DevOps is not a job title, a tool or a team, although all three are often marketed under the name. It is a way of organising work so that change flows safely from idea to production.

Why small batches are safer

Large releases bundle many changes together. When something goes wrong, it is hard to tell which change caused it, harder to fix without affecting other changes and tempting to delay release further. Small releases contain few changes, so problems are easier to find, understand and reverse, and each release carries less risk.

Manufacturers learned the same lesson: large batches hide defects, lengthen lead times and tie up work in progress, as the cutting lead time with flow and pull article explains. In software, the “inventory” is code that has been written but not released, delivering no value and growing more likely to conflict with other changes.

Continuous integration

Continuous integration means developers merge their changes into the shared main code frequently, at least daily, and every merge is automatically built and tested. Problems are found within minutes of being introduced, while the change is fresh in the developer’s mind, rather than weeks later when many changes are combined.

Continuous integration depends on:

version control, with all code in a shared repository;
automated builds that turn code into runnable software;
automated tests that run on every change; and
a team habit of fixing a broken build immediately, before doing anything else.

Continuous delivery and continuous deployment

Continuous delivery extends continuous integration so that every change that passes the automated checks is ready to release to production at any time, through an automated, repeatable process. The decision to release may still be made by a person, but releasing is a routine, low-risk action rather than a project.

Continuous deployment goes one step further: every change that passes all automated checks is released to production automatically, without a manual decision. It suits teams with very strong automated testing and monitoring. Many businesses practise continuous delivery, keeping a human decision for release, especially for systems with regulatory, financial or safety implications.

The delivery pipeline

A delivery pipeline is the automated sequence of steps that takes a change from a developer’s commit to production. A typical pipeline:

Builds the software from the committed code.
Runs fast automated tests, such as unit tests, and checks code quality and formatting.
Checks security, scanning code and libraries for known vulnerabilities and accidental secrets.
Packages the software, for example as a container image, tagged with its version.
Deploys to a test environment and runs slower tests, such as integration and end-to-end tests.
Awaits approval, if the team requires a manual release decision.
Deploys to production, using a safe release technique.
Verifies the release, checking health and key measures, and rolls back automatically if problems appear.

Each stage acts as a quality gate: a change that fails at any stage stops, and the team is notified. Because every change goes through the same pipeline, releases are consistent, and the pipeline itself becomes a record of what was tested and released.

Tests that give real confidence

Continuous delivery is only as safe as its automated tests. Teams build a mix of fast, focused unit tests and fewer, slower tests of whole workflows. Tests must be reliable: tests that fail randomly, sometimes called flaky tests, teach people to ignore failures. They must also be capable of failing when something is genuinely wrong, a point explored in the tests that cannot fail article. Database changes, integrations and calculations involving money deserve particular attention.

Releasing safely

Several techniques reduce the risk of each production release:

Rolling deployments replace running copies of an application one at a time, so the service remains available throughout.
Blue-green deployments run the new version alongside the old one and switch traffic over when it is ready, allowing an instant switch back if problems appear.
Canary releases send a small share of users or traffic to the new version first, watch for errors and expand gradually.
Feature flags deploy new code switched off, then turn features on for selected users or all users when ready, and off again instantly if needed.
Automatic rollback returns to the previous version when health checks fail.

These techniques separate deployment, installing new code, from release, exposing new behaviour to users, which lets the business decide when customers see changes.

Database changes need special care, because data cannot be rolled back as easily as code. Teams often make database changes in backward-compatible steps, so old and new versions of the application can run against the same database during a release.

Monitoring and observability

Releasing frequently is safe only if problems are detected quickly. Monitoring tracks the health of systems: whether they are available, how fast they respond, how many errors occur and how much capacity they use. Observability goes further, providing enough detail through logs, metrics and traces of individual requests to understand why something is happening, not just that it is.

Good monitoring focuses on what users experience, such as whether customers can log in, search and place orders, and alerts a named person when something is wrong. Teams often define service level objectives, targets such as “99.9% of page requests succeed within one second”, and track performance against them.

Measuring delivery performance

Research by the DevOps Research and Assessment program, known as DORA, published in its annual State of DevOps reports and the 2018 book Accelerate, identified four measures of software delivery performance:

Deployment frequency: how often changes are released to production.
Lead time for changes: how long it takes from a change being committed to it running in production.
Change failure rate: the share of releases that cause a failure needing a fix or rollback.
Time to restore service: how quickly service is restored after a failure.

The first two measure speed; the last two measure stability. A key finding of the research was that these do not have to be traded against each other: teams that release more frequently and quickly also tend to have lower failure rates and recover faster, because small changes are easier to test, understand and reverse. The measures are most useful for a team tracking its own improvement over time, rather than for comparing teams or rewarding individuals.

Learning from incidents

Failures will still happen. DevOps cultures treat them as opportunities to improve the system rather than occasions for blame. After significant incidents, teams hold blameless post-incident reviews that establish what happened, how it was detected, how it was resolved and what conditions allowed it, and agree actions to prevent recurrence or reduce impact. The focus on conditions rather than culprits encourages people to report problems and near misses openly, as in safety management. The small failures worth explaining article explains why near misses and minor failures are worth studying.

Security in the pipeline

Building security checks into the pipeline, sometimes called DevSecOps, finds vulnerabilities early and consistently. Common checks include scanning libraries for known vulnerabilities, scanning code for common security mistakes, detecting secrets accidentally committed to code and checking container images and infrastructure definitions. Access to the pipeline itself must be protected, because anyone who can change it can change what is released to production.

Applying DevOps in a small business

Small businesses rarely have separate development and operations teams, but they often have the same gaps: a contractor who builds changes and a hosting provider who runs the servers, with nobody responsible for the whole. DevOps principles still apply:

ensure someone is responsible for both changing and running each important system;
ask suppliers to use version control, automated tests and automated deployment;
release smaller changes more often rather than saving them up;
require monitoring that alerts someone when customer-facing services fail;
agree a rollback approach before each release; and
review incidents together, without blame, and act on the findings.

Many hosting platforms and code hosting services include pipeline tools that make these practices accessible to small teams at modest cost.

A worked example

This is an illustrative example. The business and figures are hypothetical.

A subscription box business has a custom web application for managing subscriptions, payments and fulfilment. Its two developers release changes about once every six weeks, on a Sunday evening, after a weekend of manual testing. Roughly one release in three causes a problem that needs an urgent fix, and restoring service after a bad release has taken up to a day.

The developers set up a pipeline in their code hosting service. Every change is built and tested automatically, including tests for payment calculations and subscription renewals. Changes are deployed to a test environment automatically and to production after a quick review, using a blue-green approach that allows an instant switch back. Feature flags let the business turn on a new referral program for staff first, then for all customers. Monitoring checks sign-up, login and checkout every minute and alerts the developers by phone.

Within six months, the team releases small changes several times a week, usually during business hours. Problems still occur occasionally, but they are smaller, detected within minutes and reversed by switching back to the previous version, typically in under fifteen minutes. The business tracks the four delivery measures monthly and holds a short blameless review after each incident; two reviews lead to new automated tests for edge cases in subscription renewals.

How this applies to a small Australian business

Release small changes often rather than large changes rarely.
Automate builds, tests and deployments through a pipeline.
Make one party responsible for changing and running each system.
Use safe release techniques such as feature flags and quick rollback.
Monitor what customers experience and alert a named person.
Track the four delivery measures to see whether practices are improving.
Review incidents without blame and act on the findings.
Build security checks into the pipeline.

Common mistakes

Saving changes up for large, risky releases.
Automating deployment without reliable tests.
Buying tools and calling it DevOps without changing how work flows.
Using delivery measures to rank teams or individuals.
Monitoring servers but not customer journeys.
Blaming individuals for system failures.

Frequently asked questions

Is continuous deployment safe for business systems? It can be, with strong automated testing, monitoring and rollback. Many businesses prefer continuous delivery, keeping a human release decision, particularly for financial and regulated systems.

Does DevOps need a dedicated team? No. It describes how development and operations responsibilities are shared and how work flows. Separate “DevOps teams” can recreate the handovers DevOps is meant to remove.

How does a small business start? With version control, a basic automated pipeline, a few automated tests for the most important functions, monitoring of key customer journeys and a habit of smaller releases.

What if a supplier releases changes only a few times a year? Ask why, what automated testing and rollback they use, and whether smaller, more frequent releases would be possible. Infrequent releases are often a symptom of manual, risky processes.

Questions to ask

How often do we release changes, and how risky does each release feel?
What happens automatically between a change being made and it reaching customers?
How quickly could we reverse a bad release?
How would we know within minutes if customers could not check out or log in?
What did we change after our last incident?

Bringing it together

DevOps brings development and operations together around shared responsibility, automation, small batches, fast feedback and continuous learning. Continuous integration tests every change as it is made; continuous delivery keeps every change ready to release through an automated pipeline; safe release techniques, monitoring and quick rollback limit the impact of problems. The four delivery measures show that speed and stability improve together when changes are small, and blameless reviews turn failures into improvements. Small businesses can apply the same principles with modest tools and the right expectations of their suppliers.

Source: KEVOS editorial notes. Examples and figures in this article are illustrations.