vault backup: 2025-11-04 16:43:34

the-failure-of-risk-management.md

@@ -2,11 +2,11 @@
 id: the-failure-of-risk-management
 aliases: []
 tags:
+  - authorship/original
   - destiny/permanent
   - status/complete
   - topic/risk
   - type/media-commentary
-  - authorship/original
 title: _The Failure of Risk Management_
 ---
 # _The Failure of Risk Management_
@@ -45,30 +45,33 @@ Qualitative risk analysis
 (i.e. risk matrices, scoring charts)
 departs from legitimate statistical methodology
 and has no robust evidence to suggest its efficacy.
-There is good reason to believe that such methods
-are deleterious to their intended purpose
-in contradiction to the common response
+In fact, there is good reason to believe that such methods
+are _deleterious_ to their intended purpose,
+in contradiction to the common refrain
 that they are "better than nothing".
 
 ### Utility as a Measure of Value
 
 Expected Value (Probability × Magnitude)
-alone can not predict or inform risky decisions,
+alone cannot predict or inform risky decisions,
 except for risk-neutral parties,
-but people and organizations are risk-averse.
+and people and organizations are risk-averse.
 
 Game 1: Which would you pick?
 * Option 1: a 100% chance to receive $10,000
 * Option 2: a 10% chance to receive $100,000
 
 Most people, being risk-averse, will pick option 1.
-Suppose the prize of option 1 were lowered
+
+Suppose the payout of option 1 were lowered
 until you would pick option 2.
 
 That value is your **Certain Monetary Equivalent (CME)**
 for a 10% chance of $100,000.
 
-For risk-neutral parties, expected value = CME
+For risk-neutral parties, expected value would equal CME
+
+***
 
 Another factor at play here
 is that utility is not proportional to monetary value.
@@ -103,10 +106,22 @@ where $Pr$ is the probability of Payoff.
 
 ### Expert Opinion Must Be ~~Adjusted~~
 
-Expert opinion is valuable despite its flaws.
+Expert opinion is valuable, but its weaknesses must be compensated for.
 
 <!-- TODO: -->
 
+Experts tend to be good at creating heuristics,
+but do not apply them consistently in practice.
+
+> [!example]
+> Chapter 7 describes a study where individual experts
+> were shown to estimate risk differently for identical cases.
+
+> [!example] p. 198
+> Models based on expert opinion consistently outperform the same experts.
+
+#### Estimator Calibration
+
 The book details the statistically observable tendency for people
 to underestimate risk and to be overconfident in their beliefs.
 It describes the process of "calibration"
@@ -115,19 +130,9 @@ and make predictions far more accurately.
 
 See [[estimator-calibration]] for more.
 
-Experts tend to be good at creating heuristics,
-but do not apply them consistently in practice.
-
-> [!example]
-> Chapter 7 describes a study where individual experts
-> were shown to estimate risk differently for identical cases.
-
 Chapter 13 introduces the [Brier Score](https://en.wikipedia.org/wiki/Brier_score)
 as a method of evaluating the performance of an estimator,
-evaluated as the mean squared error of their forecasts.
-
-> [!example] p. 198
-> Models based on expert opinion consistently outperform the same experts.
+equal to the mean squared error of their forecasts.
 
 ### Luck Looks Like Skill
 
@@ -142,19 +147,21 @@ to overvalue competence and undervalue luck
 in the role of achieving improbable accomplishments
 as the "Red Baron effect".
 
-This the unstated other half of the **law of large numbers**,
-that improbable events become likely with increased sampling.
+This the unstated other half of the **law of large numbers**:
+improbable events become likely with increased sampling.
 
-How many success stories are simply cases of winning a coin flipping tournament?
+> How many success stories
+> are simply cases of winning a coin flipping tournament?
 
 ### Qualitative Labels are Problematic
 
 > [!example] p. 170 (pp.)
-> Experts do not agree on the bounds of terms expressing probability.
-> "Likely" vs. "Very Likely"
+> Experts do not agree on the bounds of terms expressing probability
+> (e.g. "Likely" vs. "Very Likely").
 
 > [!example] p. 182 (pp.)
-> risk matrix type bucketing tends to inflate the significance of small risks.
+> risk matrix type bucketing
+> tends to inflate the significance of small risks.
 
 ### There's Always Enough Data
 
@@ -169,7 +176,7 @@ that any industry is so niche that
 data sufficient for quantitative models
 does not exist.
 
-> [!quote] Fallacy of Close Analogy - p.236
+> [!quote] Fallacy of Close Analogy (p.236)
 > ...the belief that unless two things are identical in every way,
 > nothing learned from one can be applied to the other.
 
@@ -212,17 +219,26 @@ to dismiss quantitative methods as inappropriate for their industry-specific ris
 > and therefore chooses Car B.
 > The buyer is committing the _exsupero ursus_ fallacy.
 
-Based on this strawman it is clear Hubbard believes his detractors are _correct_
+This and the other false equivalence analogy
+that is the namesake of the fallacy show that,
+while Hubbard believes his detractors are _correct_
 that qualitative methods can not capture the entire nuance of risk probability,
-but that they are failing to acknowledge that their preferred alternatives
+he believes they are failing to acknowledge that their preferred alternatives
 are not demonstrably more effective at doing so.
 
-The nuance Hubbard dismisses without addressing
-is the possibility of model _improvement_.
+There is an obvious reason why a decision-maker
+might prefer a human expert over a heuristic algorithm,
+even if the algorithm is demonstrated
+to outperform the human in all relevant metrics:
+_Adaptability_.
+
+It's likely that this preference is demonstrably unreasonable in many or most cases,
+but that it isn't acknowledged severely weakens Hubbard's argument.
+
 A most competent detractor would be aware of the apparent contradiction
 but argue that their methods will eventually surpass quantitative methods
 if they are further developed.
-Such a position would additionally contextualize Hubbard's observations
+Such a position would also contextualize Hubbard's observations
 that detractors become emotional in their defense.
 To them, Hubbard's methods represent an attractive short-term gain
 that would exclude a long-term payoff.