zmVault/the-failure-of-risk-management.md

---
id: the-failure-of-risk-management
aliases: []
tags:
  - type/media-commentary
---
# *The Failure of Risk Management*

The Failure of Risk Management
(Why It's Broken and How to Fix It)
by Douglas W. Hubbard

## Key Takeaways

### Qualitative Metrics Must Be Avoided

Qualitative risk analysis
(i.e. risk matrices, scoring charts)
departs from legitimate statistical methodology
and has no robust evidence to suggest its efficacy.
There is good reason to believe that such methods
are deleterious to their intended purpose
in contradiction to the common response
that they are "better than nothing".

### Utility as a Measure of Value

Expected Value (Probability × Magnitude)
alone can not predict or inform risky decisions,
except for risk-neutral parties.
People and organizations are risk-averse
%% TODO: ... %%

For risk-neutral parties, expected value = CME

### Expert Opinion Must Be ~~Adjusted~~

Expert opinion is valuable despite its flaws.
%% TODO: ... %%

The book details the statistically observable tendency for people
to underestimate risk and to be overconfident in their beliefs.
It describes the process of "calibration"
by which people can be trained to compensate for this bias
and make predictions far more accurately.

Experts tend to be good at creating heuristics,
but do not apply them consistently in practice.

> [!example]
> Chapter 7 describes a study where individual experts
> were shown to estimate risk differently for identical cases.

Chapter 13 introduces the [Brier Score](https://en.wikipedia.org/wiki/Brier_score)
as a method of evaluating the performance of an estimator,
evaluated as the mean squared error of their forecasts.

#### The Difficulty of Calibration

##### Boolean Examples

> The melting point of tin is higher than the melting point of aluminum.

> In English, the word “quality” is more frequently used that the word “speed”.

reductive (used more frequently where?)

> Any male pig is referred to as a hog.

reductive (referred to by whom?)

> California’s giant sequoia trees are named for an early 19th century leader of the Cherokee Indians.

reductive

> The Model T was the first car produced by Henry Ford.

reductive (Henry Ford didn't produce cars)

> When rolling 2 dice, a roll of 7 is more likely than a 3.

facile

> No one has ever been reported to have been hit by any object that fell from space.

reductive (reported by whom?)

> Sir Christopher Wren was a British anthropologist.

> Pakistan does not border Russia.

unnecessary negative form, otherwise good.

> The Navy won the first Army-Navy football game.

should specify the official event name, otherwise good.

> The paperback version of the book “The Da Vinci Code”, as of July 2007, still ranks in the top 500 bestselling books on Amazon.

obtuse phrasing, dated topic, otherwise good

> Italian has more words than any other language.

reductive (what is a word? what dialect?)

> The month of August is named after a Greek god.

borderline facile, reductive

> The deepest ocean trench is deeper than the Grand Canyon.

facile

> Abraham Lincoln was the first president born in a log cabin.

deceptive phrasing

> As of July of 2007, more people search Google for  “Harry Potter” than “Hillary Clinton” (according to GoogleTrends).

obtuse phrasing, dated topic, otherwise good

> The population of Alabama is higher than the population of Arizona.

borderline facile, deceptive phrasing

> No category 5 hurricane hit the US in 2004.

> The UK is among the top 10 largest economies in the world (by GDP).

> The movie Forest Gump has grossed more to date than E.T. The Extra Terrestrial.

obtuse phrasing, dated topic, otherwise good

##### Interval Examples

> What percentage of bronze is typically made of copper?

reductive

> How many countries have at least one McDonald’s?

As of when?

> How many employees did eBay have in the first quarter of 2006
> What was the population of Miami (within the city limits, not the entire metropolitan area) in 1990?
> How many casualties did the French suffer in the Battle of Waterloo?
> What is the range in miles of a Minuteman Missile?
> What is the percentage of IT jobs in the US were unfilled in 1997?
> The Supremes’ (with Diana Ross) song “Stop! In the Name of Love” was how long? (minutes, seconds)
> How many undergraduates attended Cambridge in 1990?
> If you could jump 50 feet straight up into the air, how many seconds would you be airborne before you landed?
> How many gallons are in a bushel (they are both measures of volume)?
> How many sovereign rulers has England had in the last thousand years?
> If the air temperature was 5 degrees below zero (Fahrenheit) and the wind speed was 15 mph, what would the temperature adjusted for wind-chill be?
> Average cost of testing in software development is what percentage of total project costs?
> On average, if a software development project was projected to take 17 months, it actually takes how many months?
> How many meters tall is the Sears Tower?
> How many gold medals did Jesse Owens win at the 1936 Berlin Olympics?
> In 2005, the average combined MPG for all US cars and light trucks on the road was how much?
> The average house in the United States uses how many gallons of water per day?
> What was the average price in the United States of a house sold in 2001?

##### Writing Good Calibration Questions

A good calibration question should not feel like it could be a "trick" question.

Definitions/terminology are *always* contentious,
questions based on them always feel deceptive.

Interval "questions" should describe the quantity
rather than phrase it as a question.

##### Strategy for Answering Calibration Questions

Confidence should never be less than probability of picking randomly
(50% for true)

### Luck Looks Like Skill

> Chapter 7 p.154

Hubbard describes a study which concluded that,
given the number of German pilots and their overall victory/defeat figures,
there was a ~30% chance an individual would achieve The Red Baron's record
_by luck alone_.

He later refers to the popular tendency
to overvalue competence and undervalue luck
in the role of achieving improbable accomplishments
as the "Red Baron effect".

### There's Always Enough Data

Hubbard challenges the popular rebuttal
that any industry is so niche that
data sufficient for quantitative models
does not exist.

> [!quote] Fallacy of Close Analogy - p.236
> ...the belief that unless two things are identical in every way,
> nothing learned from one can be applied to the other.

## Mentioned Topics and Abbreviations

* Analytic Hierarchy Process (AHP)
* Multi-Attribute Utility Theory (MAUT)
* Actuarial Science
* Options Theory (OT)
* Modern Portfolio Theory (MPT)
* Probabalistic Risk Analysis (PRA)
* Value at Risk (VaR)
* Loss-Exceedance Curve (LEC)
* Risk Tolerance
* Certain Monetary Equivalent (CME)
    * also called Certainty Equivalent

## Critiques

### _Exsupero Ursus_

Hubbard uses _exsupero ursus_ to describe the tendency of his detractors
to dismiss quantitative methods as inappropriate for their industry-specific risks.

> [!quote] Chapter 9 p.195
> Suppose a car buyer had a choice between buying two nearly identical automobiles,
> Car A and Car B.
> The only difference is that Car B is $1,000 more expensive,
> has fifty thousand more miles, and was once driven into a lake.
> But buyer chooses Car B because Car A doesn't fly.
> Neither does Car B, of course,
> but for some reason the buyer believes that Car A should fly
> and therefore chooses Car B.
> The buyer is committing the _exsupero ursus_ fallacy.

Based on this strawman it is clear Hubbard believes his detractors are _correct_
that qualitative methods can not capture the entire nuance of risk probability,
but that they are failing to acknowledge that their preferred alternatives
are not demonstrably more effective at doing so.

The nuance Hubbard dismisses without addressing
is the possibility of model _improvement_.
A most competent detractor would be aware of the apparent contradiction
but argue that their methods will eventually surpass quantitative methods
if they are further developed.
Such a position would additionally contextualize Hubbard's observations
that detractors become emotional in their defense.
To them, Hubbard's methods represent an attractive short-term gain
that would exclude a long-term payoff.

Hubbard's dismissal rubs me wrong
because it reads exactly as he describes the
"at least we're doing _something_" argument
throughout the book and just pages earlier.