Do you remember from school the so-called visitations, which consisted of, for example, the school principal coming to the lesson as an observer? In many cases, these were modeled lessons, conducted methodically, with respect for both the students and the teacher in charge.
In many cases, it was different than usual - when this person, seen as a form of control, was not present.
Or perhaps you observe how, when approaching a speed camera, drivers reduce their speed even far below the speed limit in a given place, and just after passing it - accelerate rapidly?
Or perhaps in your daily work you are so familiar with the definitions of the KPIs assigned to you that you know how to get them met, even though their meaning will be distorted?
In organizations that introduce and monitor such metrics involuntarily, you often don’t see a rolling disaster unfolding in the company. And today a little about that.
Remember the Dieselgate scandal of 2015?
It’s a very good illustration of how the desire to avoid transformation (let’s not hide it, also for financial reasons) in one organization (in this case: costly technological modifications of key vehicle components, primarily the engine) led to multidimensional consequences for the entire market and, because of the environmental aspect, for the entire planet.
What was Dieselgate all about?
If you remember it was about meeting emission standards - that’s a lot, but the details are even more interesting.
Around 2010-2015, for new cars, regulators introduced the need to meet nitrogen oxide (NOₓ) emission standards to a level of about 0.07 g/mile (US: EPA Tier 2, Tier 3) or 0.08 g/km (EU: Euro 5, Euro 6).
Meeting these standards is measured under laboratory conditions, and every new vehicle model to be approved for sale and circulation in the US and EU, respectively, goes through this. The Volkswagen Group struggled to meet these standards without a significant loss of performance and an increase in production costs in some engines.
However, the KPI was to be met, so a mechanism (a so-called “defeat device,” according to VW’s statement and the EPA’s indictment) was developed to meet it. A number of mechanisms were developed that detected when a vehicle was under test.
How was the test detected and responded to?
The vehicle’s software detected a number of specific factors that could be observed under test conditions, including:
- Lack of steering wheel movement - in laboratory tests the car stands still.
- Fixed speed and load profile - tests are conducted according to specific driving cycles (e.g. NEDC, FTP-75).
- Test duration - the software was adjusted to the standard duration of the measurement procedure.
- Other parameters, such as engine temperature, atmospheric pressure, transmission mode.
What happened when the on-board computer went into “test mode”?
- Fuel dosage was changed - the engine was running in NOₓ reducing mode at the expense of efficiency.
- Exhaust gas recirculation (EGR) was increased - reducing the amount of oxygen in the mixture reduced NOₓ formation.
- The use of the SCR system (if present) was increased - urea (AdBlue) injection was more intensive during the test.
And this is still a strong simplification, because the ECU (Engine Control Unit), EGR (Exhaust Gas Recirculation), SCR (Selective Catalytic Reduction) or DPF (Diesel Particulate Filter) were involved in the “test mode.”
How was the affair detected, what were its effects and scale?
ICCT (or, realistically, researchers at West Virginia University under contract to ICCT) detected the anomalies based on anomalies in the data. In 2014, it was decided to conduct an on-road emissions test using the Portable Emissions Measurement System (PEMS) on two VW vehicles and one from BMW.
The BMW vehicle showed no significant differences between the laboratory and mobile tests, while the VW vehicles emitted 15 to 40 times more nitrogen oxides during actual driving than the stationary tests indicated.
VW pleaded guilty in September 2015. (that’s when, among other things, it was revealed that the software was used globally, not just in the US and EU) and suffered the consequences of its actions there, including:
- a drop in the value of the company’s shares by about 40% in a few days;
- more than $30 billion in fines, damages or repair costs.
The “defeat device” software was installed in some 11 million vehicles. The cost of its production - although not publicly disclosed - can be estimated rather in the millions of dollars anyway.
And all this to meet KPIs and… avoid transformation, including, above all, its costs.
Doesn’t this approach to KPIs already happen in corporations?
Let’s not be naive. Fueling sales by offering discounts or increasing expenditures on promotional activities is one of the simpler mechanisms. In companies where CAC is counted very strictly, one has to look for more sophisticated methods (although PowerPoint or Excel will always accept anything), but… how to detect that such a phenomenon is taking place at all, and makes it difficult to communicate that digital transformation has become a necessity.
Let’s try to fix it a little.
What are Destruction KPIs?
The name is not mine (I repeat, the name is not mine) - I took it from manufacturing industries, where these are indicators related to the destruction, elimination or disposal of specific resources (and if you don’t like it or it causes confusion - call it your own).
Sometimes it’s defective products, sometimes it’s the time it takes to dispose of them or the cost of destroying defective items relative to total production, sometimes it’s the number of obsolete/overdue products that are impossible to sell.
The key aspect that has inspired me to pin down Destruction KPIs as indicators for holding back or delaying digital transformation in large organizations is that they communicate and measure risk.
When to introduce Destruction KPIs?
I’d be lying if I didn’t say it’s always worth doing, although some such competencies are taken over by de facto operational controlling departments, which protect the organization, albeit mainly from a financial angle (or by combining business operations with finance).
In mature organizations, strategic controlling areas are also being created, but not about that today.
The need for Destruction KPIs in extreme cases can even be sensed. Perhaps you can see that in your organization some key metrics are characterized by myopia, where there is no longer room not only for innovation, but even for optimization, which can put a given KPI into an oscillation during a period of change.
In the current fascination with AI, especially generative AI, it is worth knowing that it is traditional ROI metrics that most often stifle experiments with genAI - mainly due to their uncertainty/unpredictability (cf. Cynefin Framework and degrees of uncertainty environment).
Fortunately, due to global pressures, more and more companies are coming to terms with the “loss budget” to allow testing how AI can help an organization.
Examples of Destruction KPIs for corporations
Resistance indicators, such as the Zombie Process Index
In operating systems (here: primarily those of the Unix family), this is what we call a program that has been terminated, but its shutdown has not yet been handled - and it is blocking resources and can destabilize system operation.
How can this be applied to large organizations? The Zombie Process Index can indicate the number of procedures inhibiting change or extending its time (e.g., through unnecessary bureaucracy - or necessary but unreasonably long deadlines for particular steps).
Each of these processes may, of course, have a different weight and impact on change, but such a global index has a different purpose: we are beginning to measure the cost of maintaining the status quo in this way.
Balancing on the tightrope
That is, KPIs for digital transformation vs. operational continuity. One that I often suggest to companies is the red/green zone matrix when mapping horizons with the Three Horizons Framework.
Identifying KPIs used as psychological weapons
According to Goodhart’s law, “When a measure becomes a target, it ceases to be a good measure.” I wrote about influencing the sales rate above, but where else can you look for such metrics?
Time-to-market has always been my prime suspect - both in operational and transformational processes (it’s important that even the introduction of digital transformation be constantly critiqued if you care about quality and making your organization more resilient for years or decades to come).
The greater the time-to-market pressure, the more you will find spaghetti code. If such practices are seen in transformation processes, raise the alarm, because we may be introducing more risk into the organization than there is currently.
Shadow Metrics - measuring the unmeasurable
- What is the organizational culture like?
- Does detecting an error mean looking for a cause, a solution or… blame?
- Is admitting a mistake rewarded or punished?
- Is there a culture of experimentation?
- What is the organization’s resilience to chaos?
- What is the turnover (high five, HR!) in the organization, in individual departments or… under individual managers/directors?
- How many hypotheses does IT test in a certain amount of time, e.g., per week?
- What do ADRs look like?
- Are they being created at all?
- Does IT develop software because it is respected for its competence and contribution to technology in the organization (simplifying: it is treated as a partner and center of compentency, i.e., also decisions), or… are they simply the only people in the company who can program?
Meta metrics and metrics tuning
Each metric is worth cyclically reviewing - up to and including an audit of the source data - to examine whether the intent behind the metric is still preserved, or whether the metric has been distorted or… has become unnecessary.
Deloitte has identified 46 transformation metrics - also review them in the context of how universal they are, and how much they may indicate the need for Destruction KPIs.
Just as it’s worth remembering the ethics of measurement (yes, yes, Dieselgate was about that, too), the more relevant the metric is to the transformation (or the organization in general), the more it’s worth introducing (in a clearly communicated way) its dynamics over time (preferably in real time - where possible), especially when the metric is dependent on external conditions (e.g., market variables).
Destruction KPIs we can multiply. It is very simple to suggest implementing, for example:
- Debt Ratio KPI - using transformation inputs to maintain operations
- Dynamic Thresholds - algorithmic adjustment of success thresholds over time
- Anti-KPIs - deliberate measurement of “failure” indicators (e.g., number of safely abandoned projects)
The key is for the organization to allow real implementation of the first of these, with the knowledge that it can demonstrate that the king is naked.
And working in such mature and conscious companies I wish you all.
And do you know what we will do next week? We will prepare a recruitment for the Ops team.