We are back after taking a short break while Doug was in Hawaii, though it seems like half of his body is still there. And while we’re all living vicariously through his stories and wishing we, too, were on an island, today we are actually discussing a great topic that Doug threw out. Jess and Doug are tackling what almost everyone is getting wrong about data and metrics.
Audio:
Video:
Additional Resources:
- [Blog] The Difference Between Valuable & Vanity Metrics
- [Blog] What it Means to be Data-Driven, How It's Different from Metrics & How to Apply it
- [Blackline Podcast} Episode 45: Have Data-Driven Mindsets Jumped The Shark: Using Data Effectively
Show Notes:
Disclaimer: Doug loves data and metrics. The ability to gather data and the opportunities we have is tremendous.
That disclaimer being said, Doug thinks we’re jumping the shark. We’re data obsessed; data addicted. We’re getting to the point where in more cases than not, data is actually doing more harm than good. One of Doug’s favorite phrases works well with this - data is far more like a lamppost is to a drunk…used more for support than for illumination.
Doug isn’t a fan of when people question what the data says because the answer to that is what do you want it to say? Data is getting a reputation as being the truth and that’s a dangerous reputation to have in this world of algorithms. To believe for a second that there’s no opinion and subjectivity that goes into data is a major mistake.
This hit Doug hard a few weeks ago as he was listening to Michael Lewis’s podcast, Against the Rules. He was interviewing Bill James who wrote a book for years called The Bill James Baseball Abstract. James is not a fan of the movement that he started and his underlying point was that the value of the data was always about asking better questions. It allowed you to get closer to the truth. One thing he believes is that we’ll never get to the truth, you can only get closer. One complaint about this abstract was that there was no rating, and towards the end of publication they started adding a rating. James hated ratings because the point was that you’re bringing judgment.
Doug brings this up to say the problem is not the numbers, it’s how people use them. Numbers start out as a tool for thinking and they wind up replacing thought. The idea that data tells the whole story is not true. Metrics are good when they impact our thinking. Data and metrics are good when they impact and drive our hypotheses. Data and metrics are bad when we use them to answer. When we think of them as the answer to a question, they are so bad that there’s a law for this - Goodhart’s Law. Goodhart’s Law states that the moment you take a strong measurement and you turn it into an objective, that is when it stops being a good measurement.
The example Doug gives of this is marketing is click rate. This email was successful and it had a 10% click rate. Our emails need to have a high click rate, so we turn the click rate into an objective. In reality the email wasn’t successful because it had a 10% click rate. It had a 10% click rate because it was successful. Now, that doesn’t mean that click rate doesn’t matter. The question you have to ask is, “How does this add to our thinking?”
What people are getting wrong is they are not using data and metrics as a tool for thinking, they’re using it as the thought itself and not taking into account what it means.
How do you know what to pay attention to?
Doug’s bigger issue isn’t the metrics people are using, it’s how they are using them. In terms of what to pay attention to and what not to pay attention to, there’s nothing wrong if you’re totally unsure. Start with what everyone else is paying attention to.
This, again, gets into whether you are using those metrics to answer questions or not. The problem with results is to use that as a guide would be like trying to drive a car looking in your rearview mirror. Results are lagging indicators that don’t tell the whole story. They don’t get into the why. You should pay attention to the efforts that lead to outcomes because we’re trying to influence the outcomes. How are you using data to ask better questions? Data is useless if there’s no hypothesis.
Why is that? Because you can’t ask good questions. The reason we obsess about data is because we seek predictability which means we get what we expect. The question isn’t “what’s the data?” The question is “what’s the data relative to what you thought?”
What people do is they start looking at everything through a rearview mirror because humans are really good at creating rationalization. So we look at the data and start explaining the why. That might be where you start because you have to start somewhere, but you shouldn’t stop there. That’s where you can create your next hypothesis. Going back to the click rate example, let’s get click rate from 10% to 15%. You could have the hypothesis that if you can increase the click rate by 50%, it’ll increase whatever by whatever percent. So what could you do to increase from 10% to 15%? What most people will find is if you do that, the objective becomes getting to 15% and will prevent the 15% to become the objective.
Above all else you want to avoid resulting. You have to focus more on what’s causing the results and using the data to help understand that versus looking in the rearview mirror and paying attention solely to the results.
Another thing to understand about numbers is that they become the score. What you measure gets done. It’s not just about what you measure, but also how you measure. Here’s the thing – what matters is really hard to measure. When we put a measure on it, we tend to look for measurements and that’s why we get the bias towards results because they’re easier to identify. The moment you announce the metric, you’ve radically changed behavior.
The Four Flavors of Goodhart’s Law (we’ll focus on the first 3)
You can find the article Doug references here.
1. RegressiveImagine you have to hire candidates for a job. What you really want to measure is their future job performance, but you can’t measure this directly. You learn that IQ is correlated to job performance, so you decide to administer an IQ test instead. What could go wrong? You’ll find at first that you’ll see an uptick because you’re being attentive to something, but then all of a sudden you can imagine what goes on from there.
2. ExtremeThis occurs when you pick a measurement because it’s correlated to your goal in normal situations, but then adopting this measure makes you optimize for that measure.
3. AdversarialThis is where Goodhart’s Law really gets in trouble. The example they use is when the French colonial government created a bounty program. There was a huge rate issue at the time, so they paid a reward for each rat killed. The cost of having everyone bring in dead rats was not manageable, so they changed it to only needing to bring in a severed tail. So instead of killing the rats, people just cut off the tails and to make things worse they started to breed rats just to cut off the tail. The net result was more rats. (GROSS!)
4. Causal (not one that was covered in this episode)
We can’t forget that data is merely a representation of the situation and the moment you start paying attention to the data is when everything changes. It’s a tool for thinking, not thought itself. If you’re not adjusting your KPI four times a year, you’re probably not doing it right. More importantly, if you’re not talking about the context behind the measure, you’re not doing it right. If there’s no hypothesis, you’re not doing it right.
Do you think people aren’t paying enough attention to trends and they’re just looking at the numbers as they are right now?
No, Doug thinks people are looking at trends, but you don't have a valuable trend if you don’t have a hypothesis behind the trend that you’re looking at.
Why are forecasts always wrong?
How you measure is just as important as what you measure.
Next Steps:
-
Follow Jess, Doug & Imagine on socials for updates on the show or other insights:
-
Subscribe to the show on Spotify & Apple Podcasts
-
Listen to Episode 27: A Frank Discussion of The Role of The CRM Admin Today