7.4 KiB
id, aliases, tags, title
| id | aliases | tags | title | |||||
|---|---|---|---|---|---|---|---|---|
| the-failure-of-risk-management |
|
_The Failure of Risk Management_ |
The Failure of Risk Management
The Failure of Risk Management (Why It's Broken and How to Fix It) by Douglas W. Hubbard
Mentioned Topics and Abbreviations
- Analytic Hierarchy Process (AHP)
- Multi-Attribute Utility Theory (MAUT)
- Actuarial Science
- Options Theory (OT)
- Modern Portfolio Theory (MPT)
- Probabalistic Risk Analysis (PRA)
- Value at Risk (VaR)
- Loss-Exceedance Curve (LEC)
- Risk Tolerance
- Certain Monetary Equivalent (CME)
- also called Certainty Equivalent
Key Takeaways
Definition of Risk
As it is most commonly understood, risk always implies a negative impact.
For boolean cases, risk can be represented as a vector of probability and loss.
Qualitative Metrics Must Be Avoided
Qualitative risk analysis (i.e. risk matrices, scoring charts) departs from legitimate statistical methodology and has no robust evidence to suggest its efficacy. In fact, there is good reason to believe that such methods are deleterious to their intended purpose, in contradiction to the common refrain that they are "better than nothing".
Utility as a Measure of Value
Expected Value (Probability × Magnitude) alone cannot predict or inform risky decisions, except for risk-neutral parties, and people and organizations are risk-averse.
Game 1: Which would you pick?
- Option 1: a 100% chance to receive $10,000
- Option 2: a 10% chance to receive $100,000
Most people, being risk-averse, will pick option 1.
Suppose the payout of option 1 were lowered until you would pick option 2.
That value is your Certain Monetary Equivalent (CME) for a 10% chance of $100,000.
For risk-neutral parties, expected value would equal CME
Another factor at play here is that utility is not proportional to monetary value.
Consider these additional choices:
Game 2:
- Option 1: a 100% chance to receive $1,000
- Option 2: a 10% chance to receive $10,000
Game 3:
- Option 1: a 100% chance to receive $100,000
- Option 2: a 10% chance to receive $1,000,000
Assuming that the value of one dollar were linear, all three games should have the same solution, but in reality one's answers will differ.
Formulas
\text{Utility} = 1 - e^{\frac{-X}{S}}
where X is Payoff, and S is a scale unique to a decision maker.
\text{CME} = -S \times \ln(1 - \text{Utility} \times \text{Pr})
where Pr is the probability of Payoff.
Expert Opinion Must Be Adjusted
Expert opinion is valuable, but its weaknesses must be compensated for.
Experts tend to be good at creating heuristics, but do not apply them consistently in practice.
[!example] Chapter 7 describes a study where individual experts were shown to estimate risk differently for identical cases.
[!example] p. 198 Models based on expert opinion consistently outperform the same experts.
Estimator Calibration
The book details the statistically observable tendency for people to underestimate risk and to be overconfident in their beliefs. It describes the process of "calibration" by which people can be trained to compensate for this bias and make predictions far more accurately.
See estimator-calibration for more.
Chapter 13 introduces the Brier Score as a method of evaluating the performance of an estimator, equal to the mean squared error of their forecasts.
Luck Looks Like Skill
[!cite] Chapter 7 p.154 (pp.) Hubbard describes a study which concluded that, given the number of German pilots and their overall victory/defeat figures, there was a ~30% chance an individual would achieve The Red Baron's record by luck alone.
He later refers to the popular tendency to overvalue competence and undervalue luck in the role of achieving improbable accomplishments as the "Red Baron effect".
This the unstated other half of the law of large numbers: improbable events become likely with increased sampling.
How many success stories are simply cases of winning a coin flipping tournament?
Qualitative Labels are Problematic
[!example] p. 170 (pp.) Experts do not agree on the bounds of terms expressing probability (e.g. "Likely" vs. "Very Likely").
[!example] p. 182 (pp.) risk matrix type bucketing tends to inflate the significance of small risks.
There's Always Enough Data
[!quote] Voltaire Perfect is the enemy of good.
[!quote] Jon Von Neumann The truth is much too complicated to allow anything but approximations.
Hubbard challenges the popular rebuttal that any industry is so niche that data sufficient for quantitative models does not exist.
[!quote] Fallacy of Close Analogy (p.236) ...the belief that unless two things are identical in every way, nothing learned from one can be applied to the other.
Value of Information
- Expected Value of Information (EVI)
- Expected Opportunity Loss (EOL)
\text{EVI} = \text{EOL} - \text{EOL}|I
EOL translates well to continuous probabilities.
Single Point Estimates are Problematic
[!example] p. 232 (pp.) Hubbard describes a case in the oil industry where decent estimating is simplified to the point of serious error (collapsing distributions to a single point for "accounting purposes") leading to the widespread underestimating of Earth's oil reserves.
The case closely mirrors construction estimating.
Critiques
Exsupero Ursus
Hubbard uses exsupero ursus to describe the tendency of his detractors to dismiss quantitative methods as inappropriate for their industry-specific risks.
[!quote] Chapter 9 p.195 Suppose a car buyer had a choice between buying two nearly identical automobiles, Car A and Car B. The only difference is that Car B is $1,000 more expensive, has fifty thousand more miles, and was once driven into a lake. But buyer chooses Car B because Car A doesn't fly. Neither does Car B, of course, but for some reason the buyer believes that Car A should fly and therefore chooses Car B. The buyer is committing the exsupero ursus fallacy.
This and the other false equivalence analogy that is the namesake of the fallacy show that, while Hubbard believes his detractors are correct that qualitative methods can not capture the entire nuance of risk probability, he believes they are failing to acknowledge that their preferred alternatives are not demonstrably more effective at doing so.
There is an obvious reason why a decision-maker might prefer a human expert over a heuristic algorithm, even if the algorithm is demonstrated to outperform the human in all relevant metrics: Adaptability.
It's likely that this preference is demonstrably unreasonable in many or most cases, but that it isn't acknowledged severely weakens Hubbard's argument.
A most competent detractor would be aware of the apparent contradiction but argue that their methods will eventually surpass quantitative methods if they are further developed. Such a position would also contextualize Hubbard's observations that detractors become emotional in their defense. To them, Hubbard's methods represent an attractive short-term gain that would exclude a long-term payoff.
Hubbard's dismissal rubs me wrong because it reads exactly as he describes the "at least we're doing something" argument throughout the book and just pages earlier.