vault backup: 2025-12-15 16:59:22
This commit is contained in:
@@ -62,6 +62,83 @@ are _deleterious_ to their intended purpose,
|
||||
in contradiction to the common refrain
|
||||
that they are "better than nothing".
|
||||
|
||||
### Qualitative Labels are Problematic
|
||||
|
||||
> [!example] p. 170 (pp.)
|
||||
> Experts do not agree on the bounds of terms expressing probability
|
||||
> (e.g. "Likely" vs. "Very Likely").
|
||||
|
||||
> [!example] p. 182 (pp.)
|
||||
> risk matrix type bucketing
|
||||
> tends to inflate the significance of small risks.
|
||||
|
||||
|
||||
### Expert Opinion Must Be ~~Adjusted~~
|
||||
|
||||
Expert opinion is valuable, but its weaknesses must be compensated for.
|
||||
|
||||
%% TODO: %%
|
||||
|
||||
Experts tend to be good at creating heuristics,
|
||||
but do not apply them consistently in practice.
|
||||
|
||||
> [!example]
|
||||
> Chapter 7 describes a study where individual experts
|
||||
> were shown to estimate risk differently for identical cases.
|
||||
|
||||
> [!example] p. 198
|
||||
> Models based on expert opinion consistently outperform the same experts.
|
||||
|
||||
#### Estimator Calibration
|
||||
|
||||
The book details the statistically observable tendency for people
|
||||
to underestimate risk and to be overconfident in their beliefs.
|
||||
It describes the process of "calibration"
|
||||
by which people can be trained to compensate for this bias
|
||||
and make predictions far more accurately.
|
||||
|
||||
See [[estimator-calibration]] for more.
|
||||
|
||||
Chapter 13 introduces the [Brier Score](https://en.wikipedia.org/wiki/Brier_score)
|
||||
as a method of evaluating the performance of an estimator,
|
||||
equal to the mean squared error of their forecasts.
|
||||
|
||||
### Luck Looks Like Skill
|
||||
|
||||
> [!cite] Chapter 7 p.154 (pp.)
|
||||
> Hubbard describes a study which concluded that,
|
||||
> given the number of German pilots and their overall victory/defeat figures,
|
||||
> there was a ~30% chance an individual would achieve The Red Baron's record
|
||||
> _by luck alone_.
|
||||
|
||||
He later refers to the popular tendency
|
||||
to overvalue competence and undervalue luck
|
||||
in the role of achieving improbable accomplishments
|
||||
as the "Red Baron effect".
|
||||
|
||||
This the unstated other half of the **law of large numbers**:
|
||||
improbable events become likely with increased sampling.
|
||||
|
||||
> How many success stories
|
||||
> are simply cases of winning a coin flipping tournament?
|
||||
|
||||
### There's Always Enough Data
|
||||
|
||||
> [!quote] Voltaire
|
||||
> Perfect is the enemy of good.
|
||||
|
||||
> [!quote] Jon Von Neumann
|
||||
> The truth is much too complicated to allow anything but approximations.
|
||||
|
||||
Hubbard challenges the popular rebuttal
|
||||
that any industry is so niche that
|
||||
data sufficient for quantitative models
|
||||
does not exist.
|
||||
|
||||
> [!quote] Fallacy of Close Analogy (p.236)
|
||||
> ...the belief that unless two things are identical in every way,
|
||||
> nothing learned from one can be applied to the other.
|
||||
|
||||
### Utility as a Measure of Value
|
||||
|
||||
Expected Value (Probability × Magnitude)
|
||||
@@ -116,82 +193,6 @@ $$
|
||||
|
||||
where $Pr$ is the probability of Payoff.
|
||||
|
||||
### Expert Opinion Must Be ~~Adjusted~~
|
||||
|
||||
Expert opinion is valuable, but its weaknesses must be compensated for.
|
||||
|
||||
%% TODO: %%
|
||||
|
||||
Experts tend to be good at creating heuristics,
|
||||
but do not apply them consistently in practice.
|
||||
|
||||
> [!example]
|
||||
> Chapter 7 describes a study where individual experts
|
||||
> were shown to estimate risk differently for identical cases.
|
||||
|
||||
> [!example] p. 198
|
||||
> Models based on expert opinion consistently outperform the same experts.
|
||||
|
||||
#### Estimator Calibration
|
||||
|
||||
The book details the statistically observable tendency for people
|
||||
to underestimate risk and to be overconfident in their beliefs.
|
||||
It describes the process of "calibration"
|
||||
by which people can be trained to compensate for this bias
|
||||
and make predictions far more accurately.
|
||||
|
||||
See [[estimator-calibration]] for more.
|
||||
|
||||
Chapter 13 introduces the [Brier Score](https://en.wikipedia.org/wiki/Brier_score)
|
||||
as a method of evaluating the performance of an estimator,
|
||||
equal to the mean squared error of their forecasts.
|
||||
|
||||
### Luck Looks Like Skill
|
||||
|
||||
> [!cite] Chapter 7 p.154 (pp.)
|
||||
> Hubbard describes a study which concluded that,
|
||||
> given the number of German pilots and their overall victory/defeat figures,
|
||||
> there was a ~30% chance an individual would achieve The Red Baron's record
|
||||
> _by luck alone_.
|
||||
|
||||
He later refers to the popular tendency
|
||||
to overvalue competence and undervalue luck
|
||||
in the role of achieving improbable accomplishments
|
||||
as the "Red Baron effect".
|
||||
|
||||
This the unstated other half of the **law of large numbers**:
|
||||
improbable events become likely with increased sampling.
|
||||
|
||||
> How many success stories
|
||||
> are simply cases of winning a coin flipping tournament?
|
||||
|
||||
### Qualitative Labels are Problematic
|
||||
|
||||
> [!example] p. 170 (pp.)
|
||||
> Experts do not agree on the bounds of terms expressing probability
|
||||
> (e.g. "Likely" vs. "Very Likely").
|
||||
|
||||
> [!example] p. 182 (pp.)
|
||||
> risk matrix type bucketing
|
||||
> tends to inflate the significance of small risks.
|
||||
|
||||
### There's Always Enough Data
|
||||
|
||||
> [!quote] Voltaire
|
||||
> Perfect is the enemy of good.
|
||||
|
||||
> [!quote] Jon Von Neumann
|
||||
> The truth is much too complicated to allow anything but approximations.
|
||||
|
||||
Hubbard challenges the popular rebuttal
|
||||
that any industry is so niche that
|
||||
data sufficient for quantitative models
|
||||
does not exist.
|
||||
|
||||
> [!quote] Fallacy of Close Analogy (p.236)
|
||||
> ...the belief that unless two things are identical in every way,
|
||||
> nothing learned from one can be applied to the other.
|
||||
|
||||
### Value of Information
|
||||
|
||||
* Expected Value of Information (EVI)
|
||||
@@ -211,7 +212,7 @@ EOL translates well to continuous probabilities.
|
||||
> (collapsing distributions to a single point for "accounting purposes")
|
||||
> leading to the widespread underestimating of Earth's oil reserves.
|
||||
|
||||
The case closely mirrors construction estimating.
|
||||
The case closely mirrors [[construction-estimating]].
|
||||
|
||||
## Critiques
|
||||
|
||||
@@ -265,7 +266,26 @@ throughout the book and just pages earlier.
|
||||
Hubbard makes a strong case against qualitative risk management,
|
||||
but I found his attempts to introduce alternatives wanting.
|
||||
|
||||
## _Loss Models_
|
||||
## In the Context of Other Works
|
||||
|
||||
### _Fooled by Randomness_
|
||||
|
||||
[[hubbard_2020_failure]] is in many ways a response
|
||||
to [[taleb_2001_fooled-by-randomness]].
|
||||
|
||||
I'm willing to give [[#_Exsupero Ursus_]] much more slack
|
||||
after reading Taleb's introduction.
|
||||
See [[fooled-by-randomness#Qualitative Probability]].
|
||||
|
||||
Unfortunately for Hubbard, FbR is about why math is dumb,
|
||||
and so will always be more popular than tFoRM,
|
||||
which is about why math is good, actually.
|
||||
|
||||
In retrospect it's interesting that the one point the books share
|
||||
is that [[#Luck Looks Like Skill]],
|
||||
which is Taleb's basis for why quantitative methods are flawed.
|
||||
|
||||
### _Loss Models_
|
||||
|
||||
[[hubbard_2020_failure]] is obviously inspired in some part
|
||||
by [[klugman-et-al_2019_loss-models]] and its prior editions,
|
||||
|
||||
Reference in New Issue
Block a user