vault backup: 2026-01-20 17:02:44

hubbard_2020_failure.md

@@ -419,6 +419,92 @@ outperforms the alternative model, such as intuition.
 
 #### The Measurement Inversion
 
+Consider a decision analysis model for a new product.
+You have uncertainties about the cost and duration of development, materials costs once production starts, demand in different markets, and so on.
+This would be just like a typical cost-benefit analysis with a cash flow but instead of exact numbers, we use probability distributions to represent our uncertainty.
+We can even include the probability of a development project failure (no viable product was developed and the project was cancelled) or even more disastrous scenarios such as a major product recall.
+Any of these variables could be measured further with some cost and effort.
+So, which one would you measure first and how much would you be willing to spend?
+For years, I've been computing the value of additional information on every uncertain variable in a model.
+
+Suppose we ran ten thousand scenarios in a simulation and determined that 1,500 of these scenarios resulted in a net loss.
+If we decide to go ahead with this product development, and we get one of these undesirable scenarios, the amount of money we would lose is the *opportunity loss (OL)*---the cost of making the wrong choice.
+If we didn't lose money, then the OL was zero.
+We can also have an OL if we decide not to approve the product but then find out we *could* have made money.
+In the case of rejecting the product, the OL is the difference between the lease and the money we made on the widgets if we would have made money---zero if the equipment did not make money (in which case we were right to reject the idea).
+
+The *expected opportunity loss (EOL)* is each possible opportunity loss times the chance of that loss---in other words, the chance of being wrong times the cost of being wrong.
+In our Monte Carlo simulation, we simply average the OL for all of the scenarios.
+For now, let's say that given the current level of uncertainty about this product, you still think the lease is a good idea.
+So we average all 1500 scenarios the OL was positive (we lost money) and 8500 scenarios where OL was zero (me made the right choice).
+Suppose we find that the EOL is about $600,000.
+
+The EOL is equivalent to another term called the *expected value of perfect information (EVPI)*.
+The EVPI is the most you would reasonably be willing to pay if you could eliminate all uncertainty about this decision.
+Although it is almost impossible to ever get perfect information and eliminate all uncertainty, this value is useful as an absolute upper bound.
+If we can reduce the $600,000 EOL by half with a market survey that would cost $18,000, then the survey is probably a good deal.
+If you want to see a spreadsheet calculation of this type of problem, go to this book's website at [www.howtomeasureanything.com/riskmanagement](http://www.howtomeasureanything.com/riskmanagement).
+
+This becomes more enlightening when we compute the value of information for each variable in a model, especially when the models get very large.
+This way we not only get an idea for how much to spend on measurement but also which specific variables we need to measure and how much we might be willing to spend on them.
+I have done this calculation for more than 150 quantitative decision models in which most had about fifty to one hundred variables (for a total of about 10,000 variables, conservatively).
+From this, I've seen patterns that still persist every time I add more analysis to my library.
+The two main findings are:
+
+* Relatively few variables require further measurement---
+    but there are almost always *some*.
+
+* The uncertain variables with the highest EVPI
+    (highest value for further measurement)
+    tend to be those that the organization almost never measures,
+    *and* the variables they *have* been measuring have, on average, the lowest EVPI.
+
+I call this second finding the *measurement inversion*,
+and I've seen it in IT portfolios, military logistics, environmental policy,
+venture capital, market forecasts, and every other place I've looked.
+
+> [!info] The Measurement Inversion
+> The persistent tendency to focus on the least valuable measurements
+> at the expense of those more likely to improve decisions.
+
+It seems that almost everybody, everywhere,
+is systematically measuring all the wrong things.
+It is so pervasive and impactful
+that I have to wonder how much this affects the gross domestic product.
+Organizations appear to measure what they know how to measure
+without wondering whether they should learn new measurement methods for very-high-value uncertainties.
+
+How does tendency toward a measurement inversion
+affect risk assessment and, in turn, risk management?
+Highly uncertain and impactful risks
+may tend to get much less analysis than the easier-to-list, mundane events.
+The possibility of existential risks due to a major product recall,
+corporate scandal, major project failure, or factory disaster
+get less attention than the listing of much more routine and less impactful events.
+Conventional risk matrices are often populated with risks
+that are estimated to be so likely that they should happen several times a year.
+I've even seen risks estimated to be 80 percent, 90 percent,
+or even 100 percent probable in the next twelve months.
+At that level, that is more of a reliable cost of doing business.
+Of course, cost control is also important but it's not the same as risk management.
+If it is something you routinely *budget* for, it might not be the kind of risk
+upper management needs to see in a risk assessment.
+
+Also, as an analyst myself as well as a manager of many analysts,
+I can tell you that analysts are not immune to wanting to use a modeling method
+because it uses the latest buzzwords.
+Perhaps an analyst just recently learned about random forests, Bayesian networks, or deep learning.
+If she finds it interesting and wants to use it,
+she can find a way to make it part of the solution.
+The measurement inversion shows that our intuition fails us
+regarding where we need to spend more time reducing uncertainty in probabilistic models.
+Unless we estimate the value of information,
+we may go down the deep rabbit hole of adding more and more detail to a model
+and trying to gather data on less relevant issues.
+
+Periodically, we just need to back up and ask if we are really capturing the main risks
+and if we are adding detail where it informs decisions most.
+
 #### Is Monte Carlo Too Complicated?
 
 #### Notes