The Four Key Metrics for Digital Efficiency

In 2018, American anthropologist Quero Cascio coined the acronym BANI to describe the world's current dynamics. It's an acronym that stands for brittle, anxious, nonlinear, and incomprehensible. Companies in this ecosystem must meet customer expectations and needs quickly, as well as follow changes in a non-linear fashion, which is provided by adequate developer velocity (training development teams to make them more agile, innovative, and productive in all areas of operations) and digital efficiency.

Many fail, but a framework capable of guiding organizations in overcoming challenges is gaining popularity in the DevOps universe: the 4 Key Metrics by Google.

Such metrics were first mentioned in the book Accelerate by Nicole Forsgren, Jez Humble, and Gene Kim. The work served as the foundation for the DevOps Research and Assessment (Dora) survey, which was sponsored by Google. This initiative publishes an annual report called State of DevOps that shows the best DevOps practices used in companies around the world and categorizes organizations as Low, Medium, High, or Elite performers. Every year, over 30,000 people take the survey.

According to the State of DevOps, Elite performers deploy code 208 times faster than Low performers and deliver 106 times faster. These businesses' failure recovery time is 2,604 times faster, and their failure rate is 7 times lower.

Given that the 4 Key Metrics play a fundamental role in this classification, it is interesting to work with this framework because it is didactic, manages to communicate with all levels of operations, and promotes various disciplines to be worked on in the evolution of a company's digital efficiency, highlighting the actions aimed at improving developer velocity that is most necessary in each case.

Learn more about the 4 Key Metrics below.

How to achieve developer velocity and digital efficiency

Silos within companies, decentralized objects, excessive perfectionism, and lack of continuous learning negatively impact developer velocity and are some of the obstacles faced by companies in the search for digital efficiency.

In order to cover the widest possible range of aspects capable of generating losses to the business, the 4 Key Metrics consider two fundamental perspectives, described below.

• Velocity— deployment frequency (frequency of deploying new code into production) and lead time for changes (time for the commit to leaving the developer's machine until it reaches production).
• Stability— change failure rate (of total deployments, those that caused incidents in production that impacted the user) and mean time to recover (time from the identification of an incident to the correction of this problem in production).

It is important to remember that the 4 Key Metrics do not sacrifice stability in favor of a simple delivery velocity. Let us now investigate them.

1. Deployment Frequency

Deployment Frequency is the metric that will define the frequency of deployments in production, that is, the frequency of insertion of new code in the production environment. It drives all the other key metrics.

Theoretically, the more deploys you make, the shorter the lead time, the lower the deployment rates, and the greater the failure recovery. In Accelerate and Dora Research, Deployment Frequency can be broken down into the following steps.

• Low performers — fewer than once per 6 months.
• Medium performers — deploys between 1 week and 1 month.
• High performers — deploys between 1 per day and 1 per week.
• Elite performers — deploy on-demand or multiple a day.

Deploying on demand means having enough maturity in the process. This is because, in real production scenarios, it is often not feasible to make multiple deployments a day due to limitations. In any case, defining mechanisms to act in this way if desired or if necessary is part of a business decision (thus eliminating technical restrictions).

It is also worth remembering that frequent deployments make recovery from failures much easier, as well as problem detection. Also, don't confuse Deployment Frequency with "Volume with Frequency."

It is preferable to continuously deploy the production timeline, breaking them down and making them concise and constant. The deployment of 10 projects in 1 day is of no use if there are 29 days without deployment and 10 projects at the end of the month.

Among the causes of low frequency can be:

• There are many modifications and frequent changes during the development cycle; lack of automated testing mechanisms within the continuous delivery/continuous integration (CD/CI) pipeline.
• Lack of automated testing mechanisms within the continuous delivery/continuous integration (CD/CI) pipeline.
• Organizational structure imbalance.
• Inefficiency in the deployment process can be caused by several blocks, critical paths to reach production, and a lot of bureaucracy.

If the goal is to optimize Deployment Frequency and, consequently, developer velocity. Here are some tips:

• Implement a CD/CI belt in a more integrated way, with automated tests.
• Work with smaller packages of code in production, thus having an improved tracking of failures in production.
• Reduce the number of technical debts of the application.
• Plan deliveries in releases, without piling up several codes.

2. Lead time for changes

The second metric in the velocity pillar of the 4 Key Metrics, Lead Time For Changes states that the difference between the developer's machine commit from the moment it is made to the moment it reaches production must be calculated. This reveals the development time required by the DevOps process to enable the activation of new features in production.

Along with other key metrics, it is divided into Low, Medium, High and Elite performers. This is closely linked to Deployment Frequency since the frequency of deployments is helped by an increasingly smaller Lead Time For Changes.

In addition to its importance, a low rate facilitates much better feedback with a shorter development cycle, providing an exact indication of where and when changes should be made.

In fact, within the key metrics, the Lead Time For Changes is better than Cycle Time and Lead Time as it protects projects from pre-development cycles, which are often complex and different according to the context. Both Cycle Time and Lead Time depend heavily on the customer's business focus and bureaucratic areas.

On the other hand, Lead Time For Changes does not have this dependency, since the measurement process is usually standard, with some minor deviations. Even so, it is important to emphasize that the Lead Time For Changes is an approximation, an average of the dates (between the commit date and the one in which that deployment was made) performed over the time to be considered.

Some of the possible ways to calculate this metric include some mechanisms that, when crossed, can promote interesting insights and drive strategies aimed at digital efficiency:

• Kambans
• Issue trackers (such as Jira and ServiceNow)
• CI/CD pipelines
• Git hooks
• Own code repository with Patterns, using tags and branches.

With reference to the causes for a high Lead Time For Changes, some of them can be:

• A high number of unplanned changes throughout the development process.
• Lack of clarity of requirements.
• Absence of a defined Ready criterion.
• Very expensive and manual tests.
• Complex and unnecessary bottlenecks and routes.
• Bureaucracy with multiple approval steps.

Fortunately, there are ways to reduce it, which are:

• Always working towards small and self-sufficient changes.
• Embedding automated tests within the CI/CD pipeline.
• Evaluate new code management options and new SCM models (trunk-based and feature toggle based, for example, even enabling a higher frequency of deploys).

3. Change failure rate

The third key metric is the change failure rate. It concerns the percentage of deploys that caused a production failure after applying changes (service outage, for example). Problems that occur in testing phases do not count, as well as failures that do not relate to new deploys.

Their result is obtained as follows: division of failed deploy by the total of deploys in a given period. In this case, the ideal is for the rate to be as close to zero as possible. Elite performers, for example, have a Change Failure Rate of 0% to 15%, according to the State of DevOps 2021; the others, from 16% to 30%.

In addition to providing valuable information about developer velocity, this key metric reveals the ability of professionals to learn from past mistakes. In fact, as in the previous situations, there are some circumstances that can harm the rate, affecting the progression of digital efficiency, such as:

• Manual deploys, susceptible to human error.
• Changes to non-replicable infrastructure.
• Manual execution of tests.
• Poor code quality, making it difficult to maintain and introduce new code, leading to increased complexity and unexpected failures.

However, some measures can optimize the results:

• Work on smaller, self-sufficient changes.
• Ensure that critical action and service-critical infrastructure configurations are visible and replicable.
• Apply automation at every stage of the CI/CD pipeline.
• Automate code reviews with specific tools that can help improve code and save time, such as Codacy.

4. Time to recover

Rounding out the list of the 4 Key Metrics is Time To Recover, another one on the stability pillar. It is related to the time it takes to recover from a problem that impacts the user in production, considering the detection of the failure to the correction and normalization of the service for the end user.

Classified among the levels Low, Medium, High, and Elite performers, is also called Mean Time To Recover (MTTR) and is even a general breakdown of the other various MTTs that exist in the process, encompassing them in a single metric (namely, Mean Time to Detect, Mean Time to Repair, Mean Time to Discover).

Since failures are common, having a low Time To Recover is a differential for a company, as it says a lot about the process and fault tolerance; a low recovery time of service represents high maturity in response to crises, therefore adequate developer velocity, and, of course, concern with digital efficiency.

To calculate it, just define a period to be evaluated and calculate the average time between the appearance of problems and the execution of correction deployments in the given timebox.

A high Time To Recover can have several causes, among them:

• Tests that are too expensive or done manually.
• Unbalance of people in the organizational structure of the team or company.
• Inefficient incident management processes, with many steps or no defined responsible.
• Non-use of issue tracking tools (or misuse).
• Absence of monitoring tools.
• Alerts that are not properly configured.

Finally, here are some tips to reduce it:

• Pay attention to the organizational structure, putting people with the right roles in the right places and times.
• Implement automated tests and integrated them into the CI/CD pipeline.
• Provide clear documents on the incident management process (with well-defined steps and their respective responsibilities).
• Find potential steps within the incident management process that can be automated, increasingly avoiding manual interference.
• Prepare the application for crisis scenarios, with specific error alerts.

4 Key Metrics and the quest for high performance

Together, Deployment Frequency, Lead Time For Changes, Change Failure Rate and Time To Recover, the 4 Key Metrics, provide a complete picture of the development teams' and companies' health, as well as information and means for them to reach the rank of Elite performers thanks to the developer velocity and digital efficiency they build in the process.

---