vault backup: 2025-08-07 15:12:16
This commit is contained in:
@@ -12,6 +12,14 @@ by Douglas W. Hubbard
|
||||
|
||||
## Key Takeaways
|
||||
|
||||
### Definition of Risk
|
||||
|
||||
As it is most commonly understood,
|
||||
risk _always_ implies a negative impact.
|
||||
|
||||
For boolean cases,
|
||||
risk can be represented as a vector of **probability** and **loss**.
|
||||
|
||||
### Qualitative Metrics Must Be Avoided
|
||||
|
||||
Qualitative risk analysis
|
||||
@@ -44,6 +52,8 @@ It describes the process of "calibration"
|
||||
by which people can be trained to compensate for this bias
|
||||
and make predictions far more accurately.
|
||||
|
||||
See [[calibration-questions]] for more.
|
||||
|
||||
Experts tend to be good at creating heuristics,
|
||||
but do not apply them consistently in practice.
|
||||
|
||||
@@ -55,125 +65,8 @@ Chapter 13 introduces the [Brier Score](https://en.wikipedia.org/wiki/Brier_scor
|
||||
as a method of evaluating the performance of an estimator,
|
||||
evaluated as the mean squared error of their forecasts.
|
||||
|
||||
#### The Difficulty of Calibration
|
||||
|
||||
##### Boolean Examples
|
||||
|
||||
> The melting point of tin is higher than the melting point of aluminum.
|
||||
|
||||
> In English, the word “quality” is more frequently used that the word “speed”.
|
||||
|
||||
reductive (used more frequently where?)
|
||||
|
||||
> Any male pig is referred to as a hog.
|
||||
|
||||
reductive (referred to by whom?)
|
||||
|
||||
> California’s giant sequoia trees are named for an early 19th century leader of the Cherokee Indians.
|
||||
|
||||
reductive
|
||||
|
||||
> The Model T was the first car produced by Henry Ford.
|
||||
|
||||
reductive (Henry Ford didn't produce cars)
|
||||
|
||||
> When rolling 2 dice, a roll of 7 is more likely than a 3.
|
||||
|
||||
facile
|
||||
|
||||
> No one has ever been reported to have been hit by any object that fell from space.
|
||||
|
||||
reductive (reported by whom?)
|
||||
|
||||
> Sir Christopher Wren was a British anthropologist.
|
||||
|
||||
> Pakistan does not border Russia.
|
||||
|
||||
unnecessary negative form, otherwise good.
|
||||
|
||||
> The Navy won the first Army-Navy football game.
|
||||
|
||||
should specify the official event name, otherwise good.
|
||||
|
||||
> The paperback version of the book “The Da Vinci Code”, as of July 2007, still ranks in the top 500 bestselling books on Amazon.
|
||||
|
||||
obtuse phrasing, dated topic, otherwise good
|
||||
|
||||
> Italian has more words than any other language.
|
||||
|
||||
reductive (what is a word? what dialect?)
|
||||
|
||||
> The month of August is named after a Greek god.
|
||||
|
||||
borderline facile, reductive
|
||||
|
||||
> The deepest ocean trench is deeper than the Grand Canyon.
|
||||
|
||||
facile
|
||||
|
||||
> Abraham Lincoln was the first president born in a log cabin.
|
||||
|
||||
deceptive phrasing
|
||||
|
||||
> As of July of 2007, more people search Google for “Harry Potter” than “Hillary Clinton” (according to GoogleTrends).
|
||||
|
||||
obtuse phrasing, dated topic, otherwise good
|
||||
|
||||
> The population of Alabama is higher than the population of Arizona.
|
||||
|
||||
borderline facile, deceptive phrasing
|
||||
|
||||
> No category 5 hurricane hit the US in 2004.
|
||||
|
||||
> The UK is among the top 10 largest economies in the world (by GDP).
|
||||
|
||||
> The movie Forest Gump has grossed more to date than E.T. The Extra Terrestrial.
|
||||
|
||||
obtuse phrasing, dated topic, otherwise good
|
||||
|
||||
##### Interval Examples
|
||||
|
||||
> What percentage of bronze is typically made of copper?
|
||||
|
||||
reductive
|
||||
|
||||
> How many countries have at least one McDonald’s?
|
||||
|
||||
As of when?
|
||||
|
||||
> How many employees did eBay have in the first quarter of 2006
|
||||
> What was the population of Miami (within the city limits, not the entire metropolitan area) in 1990?
|
||||
> How many casualties did the French suffer in the Battle of Waterloo?
|
||||
> What is the range in miles of a Minuteman Missile?
|
||||
> What is the percentage of IT jobs in the US were unfilled in 1997?
|
||||
> The Supremes’ (with Diana Ross) song “Stop! In the Name of Love” was how long? (minutes, seconds)
|
||||
> How many undergraduates attended Cambridge in 1990?
|
||||
> If you could jump 50 feet straight up into the air, how many seconds would you be airborne before you landed?
|
||||
> How many gallons are in a bushel (they are both measures of volume)?
|
||||
> How many sovereign rulers has England had in the last thousand years?
|
||||
> If the air temperature was 5 degrees below zero (Fahrenheit) and the wind speed was 15 mph, what would the temperature adjusted for wind-chill be?
|
||||
> Average cost of testing in software development is what percentage of total project costs?
|
||||
> On average, if a software development project was projected to take 17 months, it actually takes how many months?
|
||||
> How many meters tall is the Sears Tower?
|
||||
> How many gold medals did Jesse Owens win at the 1936 Berlin Olympics?
|
||||
> In 2005, the average combined MPG for all US cars and light trucks on the road was how much?
|
||||
> The average house in the United States uses how many gallons of water per day?
|
||||
> What was the average price in the United States of a house sold in 2001?
|
||||
|
||||
##### Writing Good Calibration Questions
|
||||
|
||||
A good calibration question should not feel like it could be a "trick" question.
|
||||
|
||||
Definitions/terminology are *always* contentious,
|
||||
questions based on them always feel deceptive.
|
||||
|
||||
Interval "questions" should describe the quantity
|
||||
rather than phrase it as a question.
|
||||
|
||||
##### Strategy for Answering Calibration Questions
|
||||
|
||||
Confidence should never be less than probability of picking randomly
|
||||
(50% for true)
|
||||
> [!example] p. 198
|
||||
> Models based on expert opinion consistently outperform the same experts.
|
||||
|
||||
### Luck Looks Like Skill
|
||||
|
||||
@@ -189,8 +82,25 @@ to overvalue competence and undervalue luck
|
||||
in the role of achieving improbable accomplishments
|
||||
as the "Red Baron effect".
|
||||
|
||||
How many success stories are simply cases of winning a coin flipping tournament?
|
||||
|
||||
### Qualitative Labels are Problematic
|
||||
|
||||
> [!example] p. 170
|
||||
> Experts do not agree on the bounds of terms expressing probability.
|
||||
> "Likely" vs. "Very Likely"
|
||||
|
||||
> [!example] p. 182
|
||||
> risk matrix type bucketing tends to inflate the significance of small risks.
|
||||
|
||||
### There's Always Enough Data
|
||||
|
||||
> [!quote] Voltaire
|
||||
> Perfect is the enemy of good.
|
||||
|
||||
> [!quote] Jon Von Neumann
|
||||
> The truth is much too complicated to allow anything but approximations.
|
||||
|
||||
Hubbard challenges the popular rebuttal
|
||||
that any industry is so niche that
|
||||
data sufficient for quantitative models
|
||||
@@ -200,6 +110,27 @@ does not exist.
|
||||
> ...the belief that unless two things are identical in every way,
|
||||
> nothing learned from one can be applied to the other.
|
||||
|
||||
### Value of Information
|
||||
|
||||
* Expected Value of Information (EVI)
|
||||
* Expected Opportunity Loss (EOL)
|
||||
|
||||
$$
|
||||
\text{EVI} = \text{EOL} - \text{EOL}|I
|
||||
$$
|
||||
|
||||
EOL translates well to continuous probabilities.
|
||||
|
||||
### Single Point Estimates are Problematic
|
||||
|
||||
> [!example] p. 232
|
||||
> Hubbard describes a case in the oil industry
|
||||
> where decent estimating is simplified to the point of serious error
|
||||
> (collapsing distributions to a single point for "accounting purposes")
|
||||
> leading to the widespread underestimating of Earth's oil reserves.
|
||||
|
||||
The case closely mirrors construction estimating.
|
||||
|
||||
## Mentioned Topics and Abbreviations
|
||||
|
||||
* Analytic Hierarchy Process (AHP)
|
||||
|
||||
Reference in New Issue
Block a user