zmVault/hubbard_2020_failure.md

id, aliases, title, tags, authors, publisher, type, year, dg-publish

id	aliases	title	tags	authors	publisher	type	year	dg-publish
		The Failure of Risk Management: Why It's Broken and How to Fix It, Second Edition	authorship/other exclude-from-word-count topic/risk type/media/book	Douglas W. Hubbard	John Wiley & Sons, Inc.	book	2020	false

The Failure of Risk Management: Why It's Broken and How to Fix It, Second Edition

%% This note, with the exception of comments like this one (reserved for notes on transcription) consists only of content from the text. For commentary see the companion the-failure-of-risk-management. %%

Part One: An Introduction To The Crisis

Chapter 1: Healthy Skepticism For Risk Management

A "Common Mode Failure"

Key Definitions: Risk Management And Some Related Terms

What Failure Means

Scope And Objectives Of This Book

Notes

Chapter 2: A Summary Of The Current State Of Risk Management

A Short And Entirely-Too-Superficial History Of Risk

Current State Of Risk Management In The Organization

Current Risks And How They Are Assessed

Notes

Chapter 3: How Do We Know What Works?

Anecdote: The Risk Of Outsourcing Drug Manufacturing

Why It's Hard To Know What Works

An Assessment Of Self-Assessments

Potential Objective Evaluations Of Risk Management

What We May Find

Notes

Chapter 4: Getting Started: A Simple Straw Man Quantitative Model

A Simple One-For-One Substitution

The Expert As The Instrument

A Quick Overview Of "Uncertainty Math"

Establishing Risk Tolerance

Supporting The Decision: A Return On Mitigation

Making The Straw Man Better

Note

Part Two: Why It's Broken

Chapter 5: The "Four Horsemen" Of Risk Management: Some (Mostly) Sincere Attempts To Prevent An Apocalypse

Actuaries

War Quants: How World War II Changed Risk Analysis Forever

Economists

Management Consulting: How A Power Tie And A Good Pitch Changed Risk Management

Comparing The Horsemen

Major Risk Management Problems To Be Addressed

Notes

Chapter 6: An Ivory Tower Of Babel: Fixing The Confusion About Risk

The Frank Knight Definition

Knight's Influence In Finance And Project Management

A Construction Engineering Definition

Risk As Expected Loss

Defining Risk Tolerance

Defining Probability

Enriching The Lexicon

Notes

Chapter 7: The Limits Of Expert Knowledge: Why We Don't Know What We Think We Know About Uncertainty

The Right Stuff: How A Group Of Psychologists Might Save Risk Analysis

Mental Math: Why We Shouldn't Trust The Numbers In Our Heads

"Catastrophic" Overconfidence

The Mind Of "Aces": Possible Causes And Consequences Of Overconfidence

Inconsistencies And Artifacts: What Shouldn't Matter Does

Answers To Calibration Tests

Notes

Chapter 8: Worse Than Useless: The Most Popular Risk Assessment Method And Why It Doesn't Work

A Few Examples Of Scores And Matrices

Does That Come In "Medium"?: Why Ambiguity Does Not Offset Uncertainty

Unintended Effects Of Scales: What You Don't Know Can Hurt You

Different But Similar-Sounding Methods And Similar But Different-Sounding Methods

Notes

Chapter 9: Bears, Swans And Other Obstacles To Improved Risk Management

Algorithm Aversion And A Key Fallacy

Algorithms Versus Experts: Generalizing The Findings

A Note About Black Swans

The exsupero ursus fallacy is reinforced by authors of very popular books who seem to depend heavily on some version of the fallacy. One such author is former Wall Street trader and mathematician Nassim Taleb. He wrote The Black Swan and other books critical of common practice in risk management, especially in (but not limited to) the financial world, as well as the nonquantitative hubris of Wall Street.

A heretic of financial convention, he argues that Nobel Prize-winning modern portfolio theory and options theory (briefly mentioned in chapter 5) are fundamentally flawed and are in fact no better than astrology. In fact, Taleb considers this prize is itself an intellectual fraud. After all, as he rightly points out, it was not established in the will of Alfred Nobel, but by the Royal Bank of Sweden seventy-five years after Nobel's death. He even claims that once, in a public forum, he riled up one such prizewinner to the point of red-faced, fist-pounding anger.

Taleb bases a lot of his thesis on the fact that the impact of chance is unappreciated by mostly everyone. He sees the most significant events in history as being completely unforeseeable. He calls these events black swans in reference to an old European expression that went something like "That's about as likely as finding a black swan." The expression was based on the fact that no European had ever seen a swan that was black---until Europeans traveled to Australia. Until the first black swans were sighted, black swans were a metaphor for impossibility. Taleb puts September 11, 2001, stock market crashes, major scientific discoveries, and the rise of Google in his set of black swans. Each event, he argues, was not only unforeseen but utterly unforeseeable based on our previous experience. People will routinely confuse luck with competence and they will presume that the lack of seeing an unusual event to date is somehow proof that the event cannot occur.

Managers, traders, and the media seem to be especially susceptible to these errors. Out of a large number of managers, some managers will have made several good choices in a row by chance alone. This is what I called the Red Baron effect in a previous chapter. Such managers will see their past success as indicators of competence and, unfortunately, will act with high confidence on equally erroneous thinking in the future. Taleb recognizes the problems of overconfidence researched by Kahneman and others. Indeed, Taleb says Kahneman is the only Economics Nobel Prize winner he respects.

I think part of Taleb's skepticism is refreshing and on point. I agree with many of Taleb's observations on the misplaced faith in some models and will discuss this further in the next chapter. I might even include Taleb as one source of inspiration for identifying new categories of fallacies (and giving it a Latin name in order to sound official). Taleb coined a fallacy he refers to as the ludic fallacy, derived from the Latin word for "games of chance." Taleb defines the ludic fallacy as the assumption that the real world necessarily follows the same rules as well-defined games of chance.

Now, here is where Taleb errs. He doesn't just argue that risk management is flawed. He argues that risk management itself is impossible and that all we can do is make ourselves antifragile. I think he is just using a very different definition of risk management--- which even he uses inconsistently. No matter what he calls it, he is promoting a particular set of (vaguely defined) methods that have the objective of reducing risk. This reduction in risk will require resources. Using the definition I propose in [chapter 6], determining how to use resources to reduce risk is part of risk management. He actually contradicts himself on this point when he promotes redundancy as a method of becoming antifragile and refers to it as the "central risk management property of natural systems." So, yes, we are both talking about risk management. He focuses on particular approaches to it, but it is risk management just the same.

Confusion and inconsistency about whether managing fragility is, in practice, part of managing risks is not the only problem in his thesis. Taleb commits every form of the exsupero ursus fallacy throughout most of what he writes.

Specifically, (1) he presumes the lack of perfection of one model automatically necessitates use of the other regardless of relative performance, (2) he commits the anecdotal fallacy when looking for evidence of relative performance, and (3) he presumes that a given model was even being used when he identifies them as the culprit in major risk events.

In an interview for Fortune Taleb claimed, "No model is better than a faulty model." Again, having no model is never an option. One way or another, a model is being used. Taleb's model is simply his common sense, which is, as Albert Einstein defines it, "merely the deposit of prejudice laid down in the human mind before the age of eighteen." As with every other model, common sense has its own special errors.

We've seen the research that shows overwhelming evidence of the flaws of unaided intuition compared to even simple statistical models, and Taleb offers no empirical data to the contrary. Taleb does briefly mention the work of Meehl but dismisses it. Without making any mention of the huge numbers of conclusive results by Meehl and his colleagues, Taleb claims the entire body of research is invalid by claiming "that these researchers did not have a clear idea of where the burden of empirical evidence lies" and goes on to suggest that they lacked "rigorous empiricism." He offers no details about how more than one hundred peer-reviewed, published studies by several researchers veers from the required rigorous empiricism.

Kahneman, who actually is a psychologist like Meehl, would apparently disagree with Taleb on Meehl's methods. Taleb considers Kahneman a significant influence on his work, but who does Kahneman consider to be a significant influence on his work? Meehl. I wouldn't presume to speak for Kahneman but I wonder if he might point out to Taleb how the burden of proof was accepted and met overwhelmingly by Meehl, whereas Taleb's evidence merely amounts to, at best, selected anecdotes of shortcomings or entirely imagined straw man arguments. Taleb even sometimes cites the work of Phil Tetlock to support some other point he makes but never references Tetlock's enormous twenty-year study where he concluded that it was "impossible" to find a domain where humans clearly outperformed algorithms.

Instead of relying on large controlled studies, Taleb commits the error of arguing that single events effectively disprove a probabilistic model. He uses the apparent unforeseeability of specific events as evidence of a flaw in risk analysis. The implication is that if quantitative analysis worked, then we could make exact predictions of specific and extraordinary events such as 9/11 or the rise of Google. When arguing against the use of various statistical models in economics he states that "the simple argument that Black Swans and tail events run the socioeconomic world---and these events cannot be predicted--- is sufficient to invalidate their statistics."[^09-12] Yes, the rare events---black swans--- are individually impossible to predict precisely. But unless he can show that his alternative model (apparently his intuition) would also have predicted such events exactly, then he commits exsupero ursus when he says imperfection alone is sufficient to prefer intuition over statistics.

In addition to Kahneman, it is worth pointing out others whose work Taleb cites to make a point but who, if you actually looked at what they are doing, would contradict Taleb. For example, Taleb says he admires the mathematician Edward Thorp, who developed a mathematically sound basis for card counting in blackjack in the 1960s. Now, if the objective of card counting was to predict every hand, even the most extraordinarily rare combinations as Taleb would seem to require, then Ed Thorp's method certainly fails. But Ed Thorp's method works---that's why the casinos quit letting him play--- because his system resulted in better bets on average after a large number of hands. Taleb is also a fan of the mathematician Benoit Mandelbrot, who used the mathematics of fractals to model financial markets. Similar to Thorp and Taleb, Mandelbrot was equally unable to predict specific extraordinary events exactly, but his models are preferred by some because they seem to generate more realistic patterns that look like they could be from real data.

If anecdotal evidence were sufficient to compare model performance, one could simply point out that Taleb's investment firm, Empirica Capital LLC, closed in 2004 after several years of mediocre returns.[^09-13] He had one very good year in 2000 (a 60 percent return) because while everyone else was betting on dot-com, he bet on dot-bomb. But the returns the following years were far enough below the market average that the good times couldn't outweigh the bad for his fund.

Similar to the news pundits rejecting Nate Silver's findings or the sportscasters rejecting the methods used by the Oakland A's, Taleb merely shows that it is possible to find an error in a model if one looks hard enough. Again, the question is not whether to model (intuition is a model, too) or whether one model is imperfect (both models are imperfect) but which measurably outperforms the other and does so in many trials not just single anecdotes.

Finally, Taleb makes the error of presuming what methods were actually being used when he blames them for an event. He argues, for example, that the downfall of long-term capital management (LTCM) disproves options theory. Recall that options theory won the Nobel Prize for Robert Merton and Myron Scholes, both of whom were on the board of directors for LTCM. The theory was presumably the basis of the trading strategy of the firm. But an analysis of the failure of LTCM shows that a big reason for its downfall was the excessive use of leverage in trades---an issue that isn't even part of options theory. That appeared to be based on intuition.

Taleb also states that the crash of 1987 disproved modern portfolio theory (MPT), which would seem to presume that at least some significant proportion of fund managers used the method. I find fund managers to be tight-lipped about their specific methods, but one fund manager did tell me how "learning the theory is important as a foundation but 'real-world' decisions have to be based on practical experience, too." In fact, I found no fund managers who didn't rely partly, if not mostly, on intuition. Finally, if we are looking for explanations of the mortgage crisis, neither MPT nor options theory had anything to do with the practice of giving out mortgages to large numbers of people lacking the ability to pay them. That was more of a function of a system that incentivized banks to give risky loans without actually accepting the risk.

Finally, Taleb seems to make a variety of other points that, similar to the previous points, seem so inconsistent he ends up undermining the point he makes. For example, explaining the outcomes in terms of the narrative fallacy committed by others is sometimes itself a narrative fallacy. Arguing that "experts" don't know so much is not supported by quoting other experts. He argues that rare events defy quantitative models, but then gives specific examples of computing rare events with quantitative models (he shows the odds of getting the same result in a coin flip many times in a row and argues the benefits of Mandelbrot's mathematical models in the analysis of market fluctuations).

Taleb criticizes the use of historical data in forecasts but apparently sees no irony in his argument. He looks at several examples in which history was a poor predictor. In other words, he is assessing the validity of using historical examples by using historical examples. What Taleb and others prove with such examples is merely that what I will call a naive historical analysis can be very misleading. Taleb demonstrates his point by using the example of a turkey. The turkey had a great life right up until Thanksgiving. So, for that turkey, history was a poor indicator. So how is Taleb able to see this problem? He simply looks at the larger history of turkeys.

All he is doing is using what we may call a history of histories, or meta-historical analysis, to show how wrong naive historical analysis can be. The error in historical analysis in a stock price, for example, is to look only at the history of that stock and only for recent history. If we look at all historical analysis for a very long period of time, we find how often naive historical analysis can be wrong.

Taleb's own "experience," as extensive as it might be (at least in finance), is also just a historical analysis---just a very informal type with lots of errors in both recall and analysis, as shown in [chapter 7]. No thinking person can ever honestly claim to have formed any idea totally independent of previous observations. It just doesn't happen.

Even Taleb's ludic fallacy seems to be a fallacy itself. Sam Savage calls it the "ludic fallacy-fallacy." As Savage describes it, we cannot rationally address real-world problems of uncertainty "without first understanding the simple arithmetic of dice, cards, and spinners." Of course, Taleb is right when he says we shouldn't assume that we have defined any problem perfectly. That certainly would be an error, and if that were Taleb's point, that would be valid. But, again, whether a particular model is perfect is not the right question. The most relevant question is whether a probabilistic model---even a simple one--- outperforms the alternative model, such as intuition.

Major Mathematical Misconceptions

We're Special: The Belief That Risk Analysis Might Work, But Not Here

Notes

Chapter 10: Where Even The Quants Go Wrong: Common And Fundamental Errors In Quantitative Models

A Survey Of Analysts Using Monte Carlos

The Risk Paradox

Financial Models And The Shape Of Disaster: Why Normal Isn't So Normal

Following Your Inner Cow: The Problem With Correlations

The Measurement Inversion

Consider a decision analysis model for a new product. You have uncertainties about the cost and duration of development, materials costs once production starts, demand in different markets, and so on. This would be just like a typical cost-benefit analysis with a cash flow but instead of exact numbers, we use probability distributions to represent our uncertainty. We can even include the probability of a development project failure (no viable product was developed and the project was cancelled) or even more disastrous scenarios such as a major product recall. Any of these variables could be measured further with some cost and effort. So, which one would you measure first and how much would you be willing to spend? For years, I've been computing the value of additional information on every uncertain variable in a model.

Suppose we ran ten thousand scenarios in a simulation and determined that 1,500 of these scenarios resulted in a net loss. If we decide to go ahead with this product development, and we get one of these undesirable scenarios, the amount of money we would lose is the opportunity loss (OL)---the cost of making the wrong choice. If we didn't lose money, then the OL was zero. We can also have an OL if we decide not to approve the product but then find out we could have made money. In the case of rejecting the product, the OL is the difference between the lease and the money we made on the widgets if we would have made money---zero if the equipment did not make money (in which case we were right to reject the idea).

The expected opportunity loss (EOL) is each possible opportunity loss times the chance of that loss---in other words, the chance of being wrong times the cost of being wrong. In our Monte Carlo simulation, we simply average the OL for all of the scenarios. For now, let's say that given the current level of uncertainty about this product, you still think the lease is a good idea. So we average all 1500 scenarios the OL was positive (we lost money) and 8500 scenarios where OL was zero (me made the right choice). Suppose we find that the EOL is about $600,000.

The EOL is equivalent to another term called the expected value of perfect information (EVPI). The EVPI is the most you would reasonably be willing to pay if you could eliminate all uncertainty about this decision. Although it is almost impossible to ever get perfect information and eliminate all uncertainty, this value is useful as an absolute upper bound. If we can reduce the $600,000 EOL by half with a market survey that would cost $18,000, then the survey is probably a good deal. If you want to see a spreadsheet calculation of this type of problem, go to this book's website at www.howtomeasureanything.com/riskmanagement.

This becomes more enlightening when we compute the value of information for each variable in a model, especially when the models get very large. This way we not only get an idea for how much to spend on measurement but also which specific variables we need to measure and how much we might be willing to spend on them. I have done this calculation for more than 150 quantitative decision models in which most had about fifty to one hundred variables (for a total of about 10,000 variables, conservatively). From this, I've seen patterns that still persist every time I add more analysis to my library. The two main findings are:

• Relatively few variables require further measurement--- but there are almost always some.

• The uncertain variables with the highest EVPI (highest value for further measurement) tend to be those that the organization almost never measures, and the variables they have been measuring have, on average, the lowest EVPI.

I call this second finding the measurement inversion, and I've seen it in IT portfolios, military logistics, environmental policy, venture capital, market forecasts, and every other place I've looked.

[!info] The Measurement Inversion The persistent tendency to focus on the least valuable measurements at the expense of those more likely to improve decisions.

It seems that almost everybody, everywhere, is systematically measuring all the wrong things. It is so pervasive and impactful that I have to wonder how much this affects the gross domestic product. Organizations appear to measure what they know how to measure without wondering whether they should learn new measurement methods for very-high-value uncertainties.

How does tendency toward a measurement inversion affect risk assessment and, in turn, risk management? Highly uncertain and impactful risks may tend to get much less analysis than the easier-to-list, mundane events. The possibility of existential risks due to a major product recall, corporate scandal, major project failure, or factory disaster get less attention than the listing of much more routine and less impactful events. Conventional risk matrices are often populated with risks that are estimated to be so likely that they should happen several times a year. I've even seen risks estimated to be 80 percent, 90 percent, or even 100 percent probable in the next twelve months. At that level, that is more of a reliable cost of doing business. Of course, cost control is also important but it's not the same as risk management. If it is something you routinely budget for, it might not be the kind of risk upper management needs to see in a risk assessment.

Also, as an analyst myself as well as a manager of many analysts, I can tell you that analysts are not immune to wanting to use a modeling method because it uses the latest buzzwords. Perhaps an analyst just recently learned about random forests, Bayesian networks, or deep learning. If she finds it interesting and wants to use it, she can find a way to make it part of the solution. The measurement inversion shows that our intuition fails us regarding where we need to spend more time reducing uncertainty in probabilistic models. Unless we estimate the value of information, we may go down the deep rabbit hole of adding more and more detail to a model and trying to gather data on less relevant issues.

Periodically, we just need to back up and ask if we are really capturing the main risks and if we are adding detail where it informs decisions most.

Is Monte Carlo Too Complicated?

Notes

Part Three: How To Fix It

Chapter 11: Starting With What Works

Speak The Language

Quantifying the Appetite for Risk

Break It Down, Then Do the Math

Getting Your Probabilities Calibrated

Using Data For Initial Benchmarks

It's Been Measured Before

You Have More Data Than You Think

You Need Less Data Than You Think

A Reference Class Error: Revisiting the Turkey

Checking The Substitution

Simple Risk Management

Notes

Chapter 12: Improving The Model

Empirical Inputs

Adding Detail To The Model

Advanced Methods For Improving Expert's Subjective Estimates

Other Monte Carlo Tools

Self-Examinations For Modelers

Notes

Chapter 13: The Risk Community: Intra- And Extra-Organizational Issues Of Risk Management

Getting Organized

Managing The Model

Incentives For A Calibrated Culture

Extraorganizational Issues: Solutions Beyond Your Office Building

Growing the Profession

Of all the professions in risk management, that of the actuary is the only one that is actually a legally recognized profession. Becoming an actuary requires a demonstration of proficiency through several standardized tests. It also means adopting a code of professional ethics enforced by some licensing body. When actuaries sign their name to the Statement of Actuarial Opinion of an insurance company, they put their license on the line. As with doctors and lawyers, if they lose their license, they cannot just get another job next door. The industry of modelers of uncertainties outside of insurance could benefit greatly from this level of professional standards.

Standards organizations, government affiliated and otherwise, have always been a key part of what makes a profession a profession. But standards organizations such as PMI, NIST, and others are all guilty of explicitly promoting the ineffectual methods previously debunked. The scoring methods developed by these institutions should be disposed of altogether. These organizations should stay out of the business of designing risk analysis methods until they begin to involve people with quantitative decision analysis backgrounds in their standards-development process. Professionals should take charge of the direction their profession evolves by insisting the standards move in this direction.

Practical Observations From Trustmark

Final Thoughts On Quantitative Models And Better Decisions

Notes

Appendix: Additional Calibration Tests And Answers

Calibration Test for Ranges: A

1. How many feet tall is the Hoover Dam?
2. How many inches long is a $20 bill?
3. What percentage of aluminum is recycled in the United States?
4. When was Elvis Presley born?
5. What percentage of the atmosphere is oxygen by weight?
6. What is the latitude of New Orleans? [Hint: Latitude is 0 degrees at the equator and 90 degrees at the North Pole.]
7. In 1913, the United States military owned how many airplanes?
8. The first European printing press was invented in what year?
9. What percentage of all electricity consumed in US households was used by kitchen appliances in 2001?
10. How many miles tall is Mount Everest?
11. How long is Iraq's border with Iran in kilometers?
12. How many miles long is the Nile?
13. In what year was Harvard founded?
14. What is the wingspan (in feet) of a Boeing 747 jumbo jet?
15. How many soldiers were in a Roman legion?
16. What is the average temperature of the abyssal zone (where the oceans are more than 6,500 feet deep) in degrees F?
17. How many feet long is the Space Shuttle Orbiter (excluding the external tank)?
18. In what year did Jules Verne publish 20,000 Leagues Under the Sea?
19. How wide is the goal in field hockey (in feet)?
20. The Roman Coliseum held how many spectators?

Answers to Calibration Test for Ranges: A

1. 726 feet
2. 63/16ths (6.1417) inches
3. 45 percent
4. 1935
5. 21 percent
6. 29.95
7. 23
8. 1450
9. 26.7 percent
10. 5.5 miles
11. 1,458 kilometers
12. 4,160 miles
13. 1636
14. 196 feet
15. 6,000
16. 39 F
17. 122 feet
18. 1870
19. 12 feet
20. 50,000

Calibration Test for Ranges: B

1. The first probe to land on Mars, Viking 1, landed there in what year?
2. How old was the youngest person to fly into space?
3. How many meters tall is the Sears Tower?
4. What was the maximum altitude of the Breitling Orbiter 3, the first balloon to circumnavigate the globe, in miles?
5. On average, what percentage of the total software development project effort is spent in design?
6. How many people were permanently evacuated after the Chernobyl nuclear power plant accident?
7. How many feet long were the largest airships?
8. How many miles is the flying distance from San Francisco to Honolulu?
9. The fastest bird, the falcon, can fly at a speed of how many miles per hour in a dive?
10. In what year was the double helix structure of DNA discovered?
11. How many yards wide is a football field?
12. What was the percentage growth in internet hosts from 1996 to 1997?
13. How many calories are in 8 ounces of orange juice?
14. How fast would you have to travel at sea level to break the sound barrier (in mph)?
15. How many years was Nelson Mandela in prison?
16. What is the average daily calorie intake in developed countries?
17. In 1994, how many nations were members of the United Nations?
18. The Audubon Society was formed in the United States in what year?
19. How many feet high is the world's highest waterfall (Angel Falls, Venezuela)?
20. How deep beneath the sea was the Titanic found (in miles)?

Answers to Calibration Test for Ranges: B

1. 1976
2. 26
3. 443 meters
4. 6.9 miles
5. 20 percent
6. 350,000
7. 803 feet
8. 2,394 miles
9. 200 mph
10. 1953
11. 53.3 yards
12. 70 percent
13. 120
14. 760 mph
15. 27
16. 3,300 calories
17. 184
18. 1905
19. 3,212 feet
20. 3.36 miles

Calibration Test for Binary: A

1. The Lincoln Highway was the first paved road in the United States, and it ran from Chicago to San Francisco.
2. The Silk Road joined the two ancient kingdoms of China and Afghanistan.
3. More American homes have microwaves than telephones.
4. Doric is an architectural term for a shape of roof.
5. The World Tourism Organization predicts that Europe will still be the most popular tourist destination in 2020.
6. Germany was the second country to develop atomic weapons.
7. A hockey puck will fit in a golf hole.
8. The Sioux were one of the Plains Indian tribes.
9. To a physicist, plasma is a type of rock.
10. The Hundred Years' War was actually more than a century long.
11. Most of the fresh water on Earth is in the polar ice caps.
12. The Academy Awards ("Oscars") began over a century ago.
13. There are fewer than two hundred billionaires in the world.
14. In Excel, ^ means "take to the power of."
15. The average annual salary of airline captains is over $150,000.
16. By 1997, Bill Gates was worth more than $10 billion.
17. Cannons were used in European warfare by the eleventh century.
18. Anchorage is the capital of Alaska.
19. Washington, Jefferson, Lincoln, and Grant are the four presidents whose heads are sculpted into Mount Rushmore.
20. John Wiley & Sons is not the largest book publisher.

Answers for Calibration Test Binary: A

1. FALSE
2. FALSE
3. FALSE
4. FALSE
5. TRUE
6. FALSE
7. TRUE
8. TRUE
9. FALSE
10. TRUE
11. TRUE
12. FALSE
13. FALSE
14. TRUE
15. FALSE
16. TRUE
17. FALSE
18. FALSE
19. FALSE
20. TRUE

Calibration Test for Binary: B

1. Jupiter's "Great Red Spot" is larger than Earth.
2. The Brooklyn Dodgers' name was short for "trolley car dodgers."
3. Hypersonic is faster than subsonic.
4. A polygon is three-dimensional and a polyhedron is two-dimensional.
5. A 1-watt electric motor produces 1 horsepower.
6. Chicago is more populous than Boston.
7. In 2005, WalMart sales dropped below $100 billion.
8. Post-it Notes were invented by 3M.
9. Alfred Nobel, whose fortune endows the Nobel Peace Prize, made his fortune in oil and explosives.
10. A BTU is a measure of heat.
11. The winner of the first Indianapolis 500 clocked an average speed of under 100 mph.
12. Microsoft has more employees than IBM.
13. Romania borders Hungary.
14. Idaho is larger (in area) than Iraq.
15. Casablanca is on the African continent.
16. The first manmade plastic was invented in the nineteenth century.
17. A chamois is an alpine animal.
18. The base of a pyramid is in the shape of a square.
19. Stonehenge is located on the main British island.
20. Computer processors double in power every three months or less.

Answers for Calibration Test Binary: B

1. TRUE
2. TRUE
3. TRUE
4. FALSE
5. FALSE
6. TRUE
7. FALSE
8. TRUE
9. TRUE
10. TRUE
11. TRUE
12. FALSE
13. TRUE
14. FALSE
15. TRUE
16. TRUE
17. TRUE
18. TRUE
19. TRUE
20. FALSE