--- title: How to Measure Anything in Project Management tags: - authorship/other - type/media/book authors: - Douglas W. Hubbard - Dr. Alexander Budzier - Andreas Bang Leed --- # How to Measure Anything in Project Management ## Foreword ## Preface ## Acknowledgments ## About the Authors ## Chapter 1 --- A World-scale Risk and a World-scale Opportunity ### The Size of Projects ### The Size of Project Problems ### Efforts to Fix Projects: The Emergence of Project Management ### A Path Forward: The Meta-Project ### Notes ## Chapter 2 --- A Measurement Primer for Project Management ### The Concept of Measurement #### A Definition of Measurement #### Definition #### Measurement and Probabilities for Practical Decision-making #### Are Scales Really Measurements? ### The Object of Measurement #### What Do You See When You See More of It? #### Why Do You Care? ### The Methods of Measurement #### Statistical Significance: What's the Significance? #### Small Samples Tell You More Than You Think #### The Rule of Five ### Other Sources of Measurement Aversion #### The Cost Objection #### Measurements Change What Is Being Measured #### Statistics Can Prove Anything #### Ethical Objections to Measurement ### Notes ## Chapter 3 --- How We Know What Works ### Skepticism for Project Managers #### The Analysis Placebo #### The Problem of Feedback and Learning ### How to Test Methods #### Controlled Experiments and Component Testing #### Evaluating Sources ### The Performance of Quantitative Methods #### Experts Versus Algorithms #### The Exsupero Ursus Fallacy: Algorithm Aversion #### Potential Reasons for Exsupero Ursus ### Improving the Human Expert #### Calibrating the Expert #### The Expert Consistency Component #### Collaboration on Estimates #### The Decomposition Component ### A Summary of Research on Other Project Planning and Management Methods #### Reference Class Forecasting #### Various Project Management Methods #### The Performance of Monte Carlo Simulations ### Notes ## Chapter 4 --- The Project Decision Model: The Reason for Measurements ### Two Types of Project Measurements #### Proto-purpose Discovery Measurements #### Decision-driven Measurements #### Unproductive Incentives vs. Measurements ### Decisions Before: Thinking Slow #### Exploration vs. Exploitation #### Tracking the Outside World #### Choosing How to Run the Project ### How Models Indicate What to Measure #### The Expected Value of Information: A Simple Introduction #### The Measurement Inversion: Measuring the Wrong Things #### The Value of Imperfect Measurements ### An Aspirational Model #### The Rise of Digital Twins #### Digital Twins in Project Management #### A Practical Path Forward ### Notes ## Chapter 5 --- Project Uncertainty and Risk: A Primer ### Basic Concepts and Definitions #### Uncertainty as a Probability Distribution #### Risk: A Special Case of Uncertainty #### Definitions for Uncertainty, Risk, and Their Measurements ### The Problem with Current Methods #### Why Risk "Scores" Don't Work #### How the Risk Matrix Makes Scores Worse ### A Quantitative Risk Model: Starting Very Simple #### The One-for-One Substitution #### Monte Carlo Mechanics: A Brief Introduction ### Supporting Decisions #### A Return on Mitigation #### How Much Risk Do You Tolerate? #### Risk Versus Return: The Powerful Theory of Utility ### Simple Tools for Measuring Uncertainty and Risk #### A First Estimate of a Discrete Probability #### A First Estimate of a Continuous Probability ### Final Clarifications #### Case Examples for What Probability Means #### Uncertainty Versus Risk Versus Opportunity #### Epistemic Versus Aleatory Uncertainty #### Even More Ordinal Scales #### Risk as Governance or Compliance #### The Problem of "Black Swans" #### Some Items That Aren't Really Risks ### More Improvements to Come ### Notes ## Chapter 6 --- Calibrated Subjective Probability Estimates ### Introduction to Subjective Probability #### Two Extremes of Subjective Confidence ### Calibration Exercise #### The Calibration Exercises #### Evaluating Performance and Typical Results ### Improving Calibration #### The Equivalent Bet #### More Techniques #### More Advanced Calibration Topics to Come ### The Effects of Calibration ### Conceptual Obstacles to Calibration #### Conflating Uncertainty with Knowing Nothing #### Hypotheses That Contradict the Data #### Objections Based on the Philosophical Debate in Statistics ### Notes ## Chapter 7 --- Cost and Schedule Measurements ### The Big Plan Versus Iteration: Meta-measurements of Common Estimation Methods ### Top-down Estimations: Reference Class Forecasting ### Bottom-up Forecasting with Monte Carlo #### A Deterministic View of Tasks #### Probability Distributions for Project Tasks #### Correlations #### Multiple Prerequisites and Stochastic Critical Paths #### Parade of Trades ### Comparing Top-Down and Bottom-Up: Case Examples #### The Swedish Nuclear Waste Program #### High-speed Rail ### How to Improve the Models #### The Granularity of the Monte Carlo Model #### Distributions and Biases #### Correlations #### Improving the RCF with Monte Carlo ### Notes ## Chapter 8 --- Betting on Benefits ### Meta-Measurements of Benefits #### How Much Should Benefits Be to Justify a Project? #### Why This May Be Optimistic #### Why Measuring Benefits Is Rare ### Fermi Decompositions for Benefits #### Introduction to Fermi #### Some Example Decompositions ### Monetizing Benefits #### Forecasts of Monetary Impacts #### Preferences #### Quantifying Preferences #### The Use of Scores and Multiple Objectives #### An Example of Challenging Benefit Measurement: Biodiversity ### Measuring What Matters in Projects #### A (Slightly) More Realistic Information Value Calculation #### The High Information Values for Projects #### Getting Started on Measuring What Matters ### Considering Risk and Return #### A Risk Neutral Decision-maker for Projects #### Adding Utility Theory to Projects #### Some Alternatives Within Utility Math #### Are Executives Too Risk Averse for Projects? #### The (Apparent) Utility Paradox ### A Framework and Its Consequences #### Findings from Quantitative Analysis of Past Projects #### How and When, Not Just Whether #### Benefits Are Not Just for Project Approval Decisions ### Notes ## Chapter 9 --- Measuring Progress ### The Progress Problem #### Simple Progress, Simple Interventions ### Earned Value Management #### EVM Basics #### The XRL Example #### Recovery vs. Performance #### Forecasting with EVM ### Progress in Information Projects #### Waterfall #### Agile and Measurement in Other Software Development Methods #### Summarizing Software Metric Difficulties ### Four Stories and Lessons #### Interfaces in a Global Bank Transformation #### An Energy Project Front End #### Construction Constraints #### Testing as Software Checkpoints #### Lessons ### The Remaining Project Simulation #### Conditional Reference Class Forecasting (CRCF) #### The Bottom-Up Simulation for the Remaining Project #### Further Considerations for the RPA ### Notes ## Chapter 10 --- More Measurement Methods Made Easy ### Intuition for the Habitually Scientific #### A Jelly Bean Example #### A Little Probability Theory #### Consequences of Probability Theory #### Myths Exposed by Probability Theory #### Significant Points About Statistical Significance #### Basic Sampling Methods #### The "Mathless" Table for Medians #### Estimating a Population Proportion ### Project Cancellation Rates as a Function of Duration #### Measuring Population Size ### Measuring Some Things by Knowing Other Things #### Controlled Experiments #### Regression #### More Advanced Methods of Regression and Classification ### Estimating the Whole Distribution ### Summarizing Methods #### Brainstorming a Measurement Approach #### Data Gathering Methods ### A Review of Methods in This Chapter ### Notes on Surveys ### Notes ## Chapter 11 --- The Meta-Project: Implementing Better Project Measurements ### Start with the End in Mind: The Continuous Improvement Process #### Measure What Matters #### (Real) Skepticism and Meta-measurements #### Measuring and Forecasting the Outside World #### AI: The Most Important Project Ecosystem Measurement? #### More Thinking, Fewer Projects, Bigger Wins ### Start Your Meta-Project #### Examples of Meta-Projects Deliverables: Continuous Improvement #### Develop an Initial Team #### Assess the Current State of the Project Portfolio #### Considerations for the Meta-Project Plan #### The Pilot Project #### Scaling to the Final Deliverable ### Organizational Challenges #### Resistance to Change #### Addressing Organizational Objections to Measurement #### The Politics of Measurement ### Notes ## Chapter 12 --- A Call to Action for the Industry ### Call to Action for Project Software Vendors #### Put Decisions at the Center #### Deal in Uncertainties #### Build the User-buyer-builder Federation #### Be the Vendor That Measures Its Performance #### Be Forward-Looking ### Call to Action for the Standard-Setting Bodies ### Call to Action for Consultants, Researchers, and Advisory Firms ### Big Future Projects #### A Mars Mission #### Stopping Hurricanes #### The Meta-Project ### Notes ## Appendix 1 --- Analysis of Survey Responses on Project Management Practices ### Introduction and Data Overview #### Success Metrics: Cost and Schedule Overrun Ratios ### Overview of Project Management Practices Reported in the Survey #### Project Management Methodologies #### Cost and Schedule Estimation Methods #### Uncertainty and Risk Assessment Tools #### Certifications ### Results #### Project Management Methodologies #### Cost and Schedule Estimation Methods #### Uncertainty and Risk Assessment Tools #### Certifications #### Interpreting the (Mostly) Statistically Insignificant Results ## Appendix 2 --- Reference Class Data on Project Cost, Schedule, and Benefit Overruns ### Relevance of the Data and Reference Class Forecasting ### Using Historical Data to Improve Estimates -- An Example ### Notes ## Appendix 3 --- Selected Distributions ### Uniform ### Beta ### Beta PERT ### Triangular ### Binary ### Normal ### Lognormal ### Power Law ### Truncated Power Law ### Quantile-parameterized ### Gamma Poisson ### Stochastic Information Packet ## Appendix 4 --- Chapter 6 Calibration Question Answers ### Appendix Answers to Confidence Interval Questions ### Answers to True/False Questions ## Appendix 5 --- Measuring Biodiversity ### The Benefits of Biodiversity ### Variables to Measure for Biodiversity ### Notes