zmVault/the-failure-of-risk-management.md

---
id: the-failure-of-risk-management
aliases: []
tags:
  - type/media-commentary
---
# *The Failure of Risk Management*

The Failure of Risk Management
(Why It's Broken and How to Fix It)
by Douglas W. Hubbard

## Key Takeaways

### Definition of Risk

As it is most commonly understood,
risk _always_ implies a negative impact.

For boolean cases,
risk can be represented as a vector of **probability** and **loss**.

### Qualitative Metrics Must Be Avoided

Qualitative risk analysis
(i.e. risk matrices, scoring charts)
departs from legitimate statistical methodology
and has no robust evidence to suggest its efficacy.
There is good reason to believe that such methods
are deleterious to their intended purpose
in contradiction to the common response
that they are "better than nothing".

### Utility as a Measure of Value

Expected Value (Probability × Magnitude)
alone can not predict or inform risky decisions,
except for risk-neutral parties.
People and organizations are risk-averse
%% TODO: ... %%

For risk-neutral parties, expected value = CME

### Expert Opinion Must Be ~~Adjusted~~

Expert opinion is valuable despite its flaws.
%% TODO: ... %%

The book details the statistically observable tendency for people
to underestimate risk and to be overconfident in their beliefs.
It describes the process of "calibration"
by which people can be trained to compensate for this bias
and make predictions far more accurately.

See [[calibration-questions]] for more.

Experts tend to be good at creating heuristics,
but do not apply them consistently in practice.

> [!example]
> Chapter 7 describes a study where individual experts
> were shown to estimate risk differently for identical cases.

Chapter 13 introduces the [Brier Score](https://en.wikipedia.org/wiki/Brier_score)
as a method of evaluating the performance of an estimator,
evaluated as the mean squared error of their forecasts.

> [!example] p. 198
> Models based on expert opinion consistently outperform the same experts.

### Luck Looks Like Skill

> Chapter 7 p.154

Hubbard describes a study which concluded that,
given the number of German pilots and their overall victory/defeat figures,
there was a ~30% chance an individual would achieve The Red Baron's record
_by luck alone_.

He later refers to the popular tendency
to overvalue competence and undervalue luck
in the role of achieving improbable accomplishments
as the "Red Baron effect".

How many success stories are simply cases of winning a coin flipping tournament?

### Qualitative Labels are Problematic

> [!example] p. 170
> Experts do not agree on the bounds of terms expressing probability.
> "Likely" vs. "Very Likely"

> [!example] p. 182
> risk matrix type bucketing tends to inflate the significance of small risks.

### There's Always Enough Data

> [!quote] Voltaire
> Perfect is the enemy of good.

> [!quote] Jon Von Neumann
> The truth is much too complicated to allow anything but approximations.

Hubbard challenges the popular rebuttal
that any industry is so niche that
data sufficient for quantitative models
does not exist.

> [!quote] Fallacy of Close Analogy - p.236
> ...the belief that unless two things are identical in every way,
> nothing learned from one can be applied to the other.

### Value of Information

* Expected Value of Information (EVI)
* Expected Opportunity Loss (EOL)

$$
\text{EVI} = \text{EOL} - \text{EOL}|I
$$

EOL translates well to continuous probabilities.

### Single Point Estimates are Problematic

> [!example] p. 232
> Hubbard describes a case in the oil industry
> where decent estimating is simplified to the point of serious error
> (collapsing distributions to a single point for "accounting purposes")
> leading to the widespread underestimating of Earth's oil reserves.

The case closely mirrors construction estimating.

## Mentioned Topics and Abbreviations

* Analytic Hierarchy Process (AHP)
* Multi-Attribute Utility Theory (MAUT)
* Actuarial Science
* Options Theory (OT)
* Modern Portfolio Theory (MPT)
* Probabalistic Risk Analysis (PRA)
* Value at Risk (VaR)
* Loss-Exceedance Curve (LEC)
* Risk Tolerance
* Certain Monetary Equivalent (CME)
    * also called Certainty Equivalent

## Critiques

### _Exsupero Ursus_

Hubbard uses _exsupero ursus_ to describe the tendency of his detractors
to dismiss quantitative methods as inappropriate for their industry-specific risks.

> [!quote] Chapter 9 p.195
> Suppose a car buyer had a choice between buying two nearly identical automobiles,
> Car A and Car B.
> The only difference is that Car B is $1,000 more expensive,
> has fifty thousand more miles, and was once driven into a lake.
> But buyer chooses Car B because Car A doesn't fly.
> Neither does Car B, of course,
> but for some reason the buyer believes that Car A should fly
> and therefore chooses Car B.
> The buyer is committing the _exsupero ursus_ fallacy.

Based on this strawman it is clear Hubbard believes his detractors are _correct_
that qualitative methods can not capture the entire nuance of risk probability,
but that they are failing to acknowledge that their preferred alternatives
are not demonstrably more effective at doing so.

The nuance Hubbard dismisses without addressing
is the possibility of model _improvement_.
A most competent detractor would be aware of the apparent contradiction
but argue that their methods will eventually surpass quantitative methods
if they are further developed.
Such a position would additionally contextualize Hubbard's observations
that detractors become emotional in their defense.
To them, Hubbard's methods represent an attractive short-term gain
that would exclude a long-term payoff.

Hubbard's dismissal rubs me wrong
because it reads exactly as he describes the
"at least we're doing _something_" argument
throughout the book and just pages earlier.