"Data-driven" has become a way of saying "I would like to avoid being accountable for a judgement." There is a real distinction underneath the cliché, and it is worth recovering.
What data is good at
Data is excellent at settling questions where the options are enumerable and the feedback loop is short. Which of these two subject lines performs better. Whether this checkout step can be removed. What the marginal return on the ninth thousand pounds of spend looks like. In that territory, arguing from taste is simply worse than measuring.
What it is structurally bad at
It cannot evaluate an option you did not build. Every test is a comparison between things that already exist, which means a purely data-driven organisation converges steadily toward local maxima and never discovers the thing three hills over. It is also poor at anything with a payback period longer than the measurement window, which includes most brand investment, most platform work, and most hiring.
- Short loop, enumerable options, reversible: let the data decide.
- Long loop, generative options, expensive to reverse: use data as evidence in a judgement, and write the judgement down.
Write down the prediction
The single practice that improves decision quality most is also the cheapest: before running the test, write down what you expect to happen and why. Teams that do this discover within two quarters which of their intuitions are reliable and which are noise, which is worth more than any individual test result.
- Experimentation
- Decision-making
- Analytics
