zmVault/the-failure-of-risk-management.md

---
id: the-failure-of-risk-management
title: _The Failure of Risk Management_
tags:
  - status/complete
  - topic/risk
  - type/media-commentary
---
# _The Failure of Risk Management_

This is the commentary companion to [[hubbard_2020_failure]].

_The Failure of Risk Management_ (abbreviated here as tFoRM)

tFoRM is has two chief concerns:

* introducing actuarial methods to industries other than insurance.

* discrediting arguments against the use of actuarial methods
    in industries other than insurance.

## Mentioned Topics and Abbreviations

* Analytic Hierarchy Process (AHP)
* Multi-Attribute Utility Theory (MAUT)
* Actuarial Science
* Options Theory (OT)
* Modern Portfolio Theory (MPT)
* Probabilistic Risk Analysis (PRA)
* Value at Risk (VaR)
* Loss-Exceedance Curve (LEC)
* Risk Tolerance
* Certain Monetary Equivalent (CME)
    * also called Certainty Equivalent

One or more curves referenced in the text are
[Indifference curves](https://en.wikipedia.org/wiki/Indifference_curve)
but not referred to as such.

## Key Takeaways

### Definition of Risk

As it is most commonly understood,
risk _always_ implies a negative impact.
Analyses that suggest otherwise are unhelpful.

For boolean cases,
risk can be represented as a vector of **probability** and **loss**.

### Qualitative Metrics Must Be Avoided

Qualitative risk analysis
(i.e. risk matrices, scoring charts)
departs from legitimate statistical methodology
and has no robust evidence to suggest its efficacy.
In fact, there is good reason to believe that such methods
are _deleterious_ to their intended purpose,
in contradiction to the common refrain
that they are "better than nothing".

### Qualitative Labels are Problematic

> [!example] p. 170 (pp.)
> Experts do not agree on the bounds of terms expressing probability
> (e.g. "Likely" vs. "Very Likely").

> [!example] p. 182 (pp.)
> risk matrix type bucketing
> tends to inflate the significance of small risks.

### Expert Opinion Must Be ~~Adjusted~~

Expert opinion is valuable, but its weaknesses must be compensated for.

%% TODO: %%

Experts tend to be good at creating [[heuristics]],
but do not apply them consistently in practice.

> [!example]
> Chapter 7 describes a study where individual experts
> were shown to estimate risk differently for identical cases.

> [!example] p. 198
> Models based on expert opinion consistently outperform the same experts.

#### Estimator Calibration

The book details the statistically observable tendency for people
to underestimate risk and to be overconfident in their beliefs.
It describes the process of "calibration"
by which people can be trained to compensate for this bias
and make predictions far more accurately.

See [[estimator-calibration]] for more.

Chapter 13 introduces the [Brier Score](https://en.wikipedia.org/wiki/Brier_score)
as a method of evaluating the performance of an estimator,
equal to the mean squared error of their forecasts.

### Luck Looks Like Skill

Individually improbable events
become likely with increased sampling.

This the unstated other half of the **law of large numbers**

> How many success stories
> are simply cases of winning a coin flipping tournament?

#### Red Baron Effect

> [!quote] [[hubbard_2020_failure#Chapter 7 The Limits Of Expert Knowledge Why We Don't Know What We Think We Know About Uncertainty]]
> ...given the number of pilots and the win ratio,
> there was about a 30 percent chance that, by luck alone,
> one pilot would have gotten eighty kills,
> the number Manfred von Richthofen is credited for.

That is, there is a 30% chance The Red Baron was a [[lucky-fools|lucky fool]]

He later refers to the popular tendency
to overvalue competence and undervalue luck
in the role of achieving improbable accomplishments
as the "Red Baron effect".

### There's Always Enough Data

> [!quote] Voltaire
> Perfect is the enemy of good.

> [!quote] Jon Von Neumann
> The truth is much too complicated to allow anything but approximations.

Hubbard challenges the popular rebuttal
that any industry is so niche that
data sufficient for quantitative models
does not exist.

> [!quote] Fallacy of Close Analogy (p.236)
> ...the belief that unless two things are identical in every way,
> nothing learned from one can be applied to the other.

### Utility as a Measure of Value

Expected Value (Probability × Magnitude)
alone cannot predict or inform risky decisions,
except for risk-neutral parties,
and people and organizations are risk-averse.

Game 1: Which would you pick?
* Option 1: a 100% chance to receive $10,000
* Option 2: a 10% chance to receive $100,000

Most people, being risk-averse, will pick option 1.

Suppose the payout of option 1 were lowered
until you would pick option 2.

That value is your **Certain Monetary Equivalent (CME)**
for a 10% chance of $100,000.

For risk-neutral parties, expected value would equal CME

***

Another factor at play here
is that [[utility]] is not proportional to monetary value.

Consider these additional choices:

Game 2:
* Option 1: a 100% chance to receive $1,000
* Option 2: a 10% chance to receive $10,000

Game 3:
* Option 1: a 100% chance to receive $100,000
* Option 2: a 10% chance to receive $1,000,000

Assuming that the value of one dollar were linear,
all three games should have the same solution,
but in reality one's answers will differ.

#### Formulas

$$
\text{Utility} = 1 - e^{\frac{-X}{S}}
$$

where $X$ is Payoff, and $S$ is a scale unique to a decision maker.

$$
\text{CME} = -S \times \ln(1 - \text{Utility} \times \text{Pr})
$$

where $Pr$ is the probability of Payoff.

### Value of Information

* Expected Value of Information (EVI)
* Expected Opportunity Loss (EOL)

$$
\text{EVI} = \text{EOL} - \text{EOL}|I
$$

EOL translates well to continuous probabilities.

### Single Point Estimates are Problematic

> [!example] p. 232 (pp.)
> Hubbard describes a case in the oil industry
> where decent estimating is simplified to the point of serious error
> (collapsing distributions to a single point for "accounting purposes")
> leading to the widespread underestimating of Earth's oil reserves.

The case closely mirrors [[construction-estimating]].

## Critiques

### _Exsupero Ursus_

Hubbard uses _exsupero ursus_ to describe the tendency of his detractors
to dismiss quantitative methods as inappropriate for their industry-specific risks.

> [!quote] Chapter 9 p.195
> Suppose a car buyer had a choice between buying two nearly identical automobiles,
> Car A and Car B.
> The only difference is that Car B is $1,000 more expensive,
> has fifty thousand more miles, and was once driven into a lake.
> But buyer chooses Car B because Car A doesn't fly.
> Neither does Car B, of course,
> but for some reason the buyer believes that Car A should fly
> and therefore chooses Car B.
> The buyer is committing the _exsupero ursus_ fallacy.

This and the other false equivalence analogy
that is the namesake of the fallacy show that,
while Hubbard believes his detractors are _correct_
that qualitative methods can not capture the entire nuance of risk probability,
he believes they are failing to acknowledge that their preferred alternatives
are not demonstrably more effective at doing so.

There is an obvious reason why a decision-maker
might prefer a human expert over a heuristic algorithm,
even if the algorithm is demonstrated
to outperform the human in all relevant metrics:
_Adaptability_.

It's likely that this preference is demonstrably unreasonable in many or most cases,
but that it isn't acknowledged severely weakens Hubbard's argument.

A most competent detractor would be aware of the apparent contradiction
but argue that their methods will eventually surpass quantitative methods
if they are further developed.
Such a position would also contextualize Hubbard's observations
that detractors become emotional in their defense.
To them, Hubbard's methods represent an attractive short-term gain
that would exclude a long-term payoff.

Hubbard's dismissal rubs me wrong
because it reads exactly as he describes the
"at least we're doing _something_" argument
throughout the book and just pages earlier.

### "Quantitative Methods"

Hubbard makes a strong case against qualitative risk management,
but I found his attempts to introduce alternatives wanting.

## In the Context of Other Works

### _Fooled by Randomness_

[[hubbard_2020_failure]] is in many ways a response
to [[taleb_2001_fooled-by-randomness]].

I'm willing to give [[#_Exsupero Ursus_]] much more slack
after reading Taleb's introduction.
See [[fooled-by-randomness#Qualitative Probability]].

Unfortunately for Hubbard, FbR is about why math is dumb,
and so will always be more popular than tFoRM,
which is about why math is good, actually.

In retrospect it's interesting that the one point the books share
is that [[#Luck Looks Like Skill]],
which is Taleb's basis for why quantitative methods are flawed.

### _Loss Models_

[[hubbard_2020_failure]] is obviously inspired in some part
by [[klugman-et-al_2019_loss-models]] and its prior editions,
which is strange because neither the text nor Klugman
are mentioned in the text.

The opening line of _Loss Models_,

> [!quote] [[klugman-et-al_2019_loss-models#1.1.1 The Modeling Process]] (emphasis added)
> The model-based approach should be considered
> _in the context of the objectives of any given problem._

is just [[#_Exsupero Ursus_]].