When I was first learning about marketing (I never got very far with it), one of the things I remember reading was how important studies are. If you can share a study that supports your point, it becomes immediately more compelling. The same is true of writing books. When I wrote my habit book, several people told me that I should dig through studies, find some that supported my points, and then include them.
I don’t find studies compelling at all, and generally disregard them when it comes to making decisions. There are three main reasons why I do this.
First, it is a lot easier to prove correlation than causation. For example, a study could probably show that people who buy Rolls Royces live longer than those that don’t. The argument could be put forth that riding in such a fine automobile is so good for the soul that the owner gets to live longer, but it is probably just the case that if you have the money for a Rolls Royce, you also have the money for good health care. The “one glass of wine a day” argument could also fall into the category. Could it be that regular wine drinkers who don’t overindulge are just people with reasonable restraint and better financial means than average? Probably.
People who write studies are actually usually pretty careful to note correlation vs. causation, but media outlets show no such restraint. That’s why you see all sorts of magazine articles that say things like, “Could eating broccoli once a week make you live for an extra year?”.
Because it is so much easier to find correlation than causation, you can be assured that most results will be the product of correlation.
Second, studies don’t claim to be 100% confident. The standard measure is 95%, which essentially means that there is a 95% chance that the findings are true (the actual definition of this is a little more nuanced). If all studies that were completed were published, this would be great. In that case if 100 groups of scientists studied something and around 95% of them found one result, we could be very certain that it was true.
What actually happens, though, is that unremarkable data is not published. So if one hundred groups test to see if acai berries increase lifespan, 95 find that it doesn’t, 5 find that it does, what will happen is that the 95 won’t bother publishing (as publishing takes a tremendous amount of work), and the 5 will publish that it does. As a result you have 5 groups saying that they are 95% sure that acai berries increase lifespan. That sounds very compelling!
When every study shows the same thing and there are a lot of them (like the effects of tobacco, for example), you can assume that they are true. But when one study shows something, it’s shocking, and there’s no clear path for causation, it’s almost certainly not really true.
Last, I lower my confidence in studies because every individual is different. That doesn’t mean that I should smoke because maybe I’m not affected by tobacco, but it does mean that if the average person needs 7 hours of sleep but my own research shows that 8 works best for me, I should probably stick to 8. We each have individual strengths and weaknesses that often affect us more than general averages do.
A great example would be that college graduates make more money than non college graduates. Besides being potentially affected by causation vs correlation, I know that given my own skillset and preferences, my lifetime earnings will probably be much higher as a dropout. That may not be true for most (though I suspect lifetime earnings – tuition cost – interest that could have been earned on that cost is not better for a great many many people), but it’s probably true for me.
When the path for causation is obvious or well-explained, there is scientific consensus, and the factors at play are applicable to you, a study is probably relevant to you. If it’s not, it may be best to disregard it.
###
Photo is of some cool plants I saw at the botanical gardens in Hilo, Hawaii. Great place to visit if you’re in Hilo.
I still have some spots for Superhuman 3 this year. Come! It will be awesome.
Leave a Reply