I’ve been reading a lot lately about data-informed decision making…more than is likely healthy for me. And so much of what I read begins and ends with the assumption that more data is always better. The texts jump right in to how you collect, analyze, interpret, and report data, as if it’s always a net benefit to do so. Few actually pause to explore when and why you should measure at all.
cc flickr Sami
Which is why I admire and appreciate Douglas W. Hubbard’s book, How to Measure Anything. Hubbard certainly explains the how of many approaches to data gathering and analysis (in rather relentless detail). But he also spends a good portion of the book exploring the when and why. From his perspective, more information can certainly have value to a decision, but it also has a cost. And unless that value exceeds that cost, you should be spending your time, money, and attention somewhere else.
The two factors that determine the value of additional information, according to Hubbard, are:
- the CHANCE of being wrong, and
- the COST of being wrong.
The chance of being wrong relates to your level of uncertainty…how likely it is that you can’t frame a productive answer already. The cost of being wrong relates to consequence…how badly it will hurt if you make a poor choice, or forgo an alternate choice that would have been much better.
For many decisions we make every day, the chance of being wrong is low, because the decision is frequent and consistent and we have a good understanding of possible outcomes – your usual order at your usual coffee shop, for example.
Also for many decisions, the cost of being wrong is low. You might experience temporary disappointment, or lose a few dollars or minutes. Not pleasant, but not catastrophic either.
For these decisions, the value of more information isn’t worth the cost. Just choose, move on, and note the outcome so you make a better choice next time.
But for other decisions, when uncertainty and potential negative consequence are high, additional information can have dramatic value over its cost. These are moments when your team and your board should dig deep, think hard, and gather more evidence.
Don’t get me wrong, I am a huge fan of decision-making based on observed evidence, rather than habit or assumption. But I’m also a fan of spending limited time, money, and energy in ways that actually matter to the work. A large part of that balance is knowing when it’s time to measure, and when it’s time to move.
NOTE: Hubbard’s book is NOT specifically about the arts, or nonprofits, or any particular industry at all. And it’s a tough slog in many parts (lots of math and statistics). But the framing chapters are among the most clear and practical I’ve read on the subject.
Great point, Andrew. I, too, have grown weary of “big data” research projects in search of a question. I think a lot of people talk in a very rote way about “data-based decision making’ without really thinking about how decisions get made or the appropriate role of “data in decision making. I’ve found that some of the best outcomes of research have nothing to do with short-term decision making, but lie in building a culture of inquiry and a capacity to ask better questions.
Yay How to Measure Anything! We’ve been using this book and the ideas discussed therein extensively in staff training around decision making at Fractured Atlas for the post couple of years.