When I was first learning about marketing (I never got very far with it), one of the things I remember reading was how important studies are. If you can share a study that supports your point, it becomes immediately more compelling. The same is true of writing books. When I wrote my habit book, several people told me that I should dig through studies, find some that supported my points, and then include them.
I don't find studies compelling at all, and generally disregard them when it comes to making decisions. There are three main reasons why I do this.
First, it is a lot easier to prove correlation than causation. For example, a study could probably show that people who buy Rolls Royces live longer than those that don't. The argument could be put forth that riding in such a fine automobile is so good for the soul that the owner gets to live longer, but it is probably just the case that if you have the money for a Rolls Royce, you also have the money for good health care. The "one glass of wine a day" argument could also fall into the category. Could it be that regular wine drinkers who don't overindulge are just people with reasonable restraint and better financial means than average? Probably.
People who write studies are actually usually pretty careful to note correlation vs. causation, but media outlets show no such restraint. That's why you see all sorts of magazine articles that say things like, "Could eating broccoli once a week make you live for an extra year?".
Because it is so much easier to find correlation than causation, you can be assured that most results will be the product of correlation.
Second, studies don't claim to be 100% confident. The standard measure is 95%, which essentially means that there is a 95% chance that the findings are true (the actual definition of this is a little more nuanced). If all studies that were completed were published, this would be great. In that case if 100 groups of scientists studied something and around 95% of them found one result, we could be very certain that it was true.
What actually happens, though, is that unremarkable data is not published. So if one hundred groups test to see if acai berries increase lifespan, 95 find that it doesn't, 5 find that it does, what will happen is that the 95 won't bother publishing (as publishing takes a tremendous amount of work), and the 5 will publish that it does. As a result you have 5 groups saying that they are 95% sure that acai berries increase lifespan. That sounds very compelling!
When every study shows the same thing and there are a lot of them (like the effects of tobacco, for example), you can assume that they are true. But when one study shows something, it's shocking, and there's no clear path for causation, it's almost certainly not really true.
Last, I lower my confidence in studies because every individual is different. That doesn't mean that I should smoke because maybe I'm not affected by tobacco, but it does mean that if the average person needs 7 hours of sleep but my own research shows that 8 works best for me, I should probably stick to 8. We each have individual strengths and weaknesses that often affect us more than general averages do.
A great example would be that college graduates make more money than non college graduates. Besides being potentially affected by causation vs correlation, I know that given my own skillset and preferences, my lifetime earnings will probably be much higher as a dropout. That may not be true for most (though I suspect lifetime earnings - tuition cost - interest that could have been earned on that cost is not better for a great many many people), but it's probably true for me.
When the path for causation is obvious or well-explained, there is scientific consensus, and the factors at play are applicable to you, a study is probably relevant to you. If it's not, it may be best to disregard it.
Photo is of some cool plants I saw at the botanical gardens in Hilo, Hawaii. Great place to visit if you're in Hilo.
I still have some spots for Superhuman 3 this year. Come! It will be awesome.
As a retired engineer, with experience in at least a half-dozen industries across consumer electronics to medical devices, I am hard-pressed to imagine a statement that could insult or defame marketing. Back in the days, mostly the 80's, when Americans made stuff and cared about trying to both be competitive in price and quality the only departments who always managed to escape quality measurement were mismanagement and marketing. Sometimes sales also skated free of actual evaluation. The link between product sales, customer perception, and corporate ranking is about as clear as mud and marketing keeps it ;that way so that no one ever tries to determine if the marketing department actually provides value. Smoke-blowers like Steve Jobs have muddied up those waters to the point of no-return, which also means the US will continue to degrade into a service economy that makes nothing and creates noting.
If you ever want to read examples of studies that are pure hogwash, I recommend medical product studies: both devices and drugs. After tossing out all of the patients who either didn't show any improvement or, worse, who died from exposure to the product, and cherry picking the few who actually could have seen improvement, even when the placebo group saw even better outcomes, an incredible number of so-called scientific studies mostly prove that a sucker is born considerably more often than ever 60 seconds.
HI, Excuse me while I go back in time. I read your Business Insider article 2011 about how to meet famous people. The thing that threw me was you deceleration on how simple it is to get somebody's email. I want to get to know a famous woman who starred in a successful sitcom. She has social media with millions of followers. She does her own posting but she never replies to the comments and of course she has no email that can be discovered. Otherwise she would be deluged. I am thinking of sending a letter to a business she co owns which will likely not be read. but there is no email.
I've been working in marketing for 7 years. I've read many many books on the topic. This is a mischaracterization. I've seen very little talking about the value of citing studies.
Yes, it's done in health/beauty products where there is an existing body of "medical or clinical wisdom". And I agree a lot of that is bogus.
But good marketing, like pickup, is about eliciting behavior compliance by increasingly gaining trust from the audience.
There's no one way to do that. I think you've been unfair to marketing in this post.
For businesses like CruiseSheets, the marketing has NOTHING to do with studies. Its trustworthiness, discount beyond anything anywhere else, and the convenience of having that all in one, easy to use place.
Tynan, your #1 and #2 describe the phenomena of trash media and publication bias very well but #3 is actually a repeat of #1: What all sleep studies say is „every mammal needs sleep and for humans it seems to be a median of 7 hours“ (I made that number up) - only trash media shorten that into irrecognizability. I avoid #1 totally by only accessing quality journalism and the false positive rate for #2 seems to be around 50% (as found in a big replication study in the area of psychology), not the 100% in your example. Therefore I derive a lot of food for thought from my consumption of study digests at a cost of just 10 minutes a day. With good value to me, as proven time and again.
Another source of food for thought are opinions by out-of-the-box-thinkers - and that includes your posts. One thing to consider there is their propensity to live in their very own echo chambers, sometimes showing contact allergies against other methods and schools of thought than their own. I can prize that in, can you?
"What actually happens, though, is that unremarkable data is not published."
That is NOT TRUE at all. Studies are published in top-tier journals ALL THE TIME which don't show the effects that are expected, don't have a significant groundbreaking outcome, and don't really show anything remarkable at all. Sometimes you test things and there's no difference between the control and the test variable. These studies are still published! Having "no significant difference" as the result of a study does not mean the study is insignificant! To say that studies aren't published unless they show a remarkable outcome is bullshit.
Hey Tynan, loyal reader for a long time, and love your stuff.
I'm an academic researcher (accounting) and can understand your viewpoint. In my eyes, the most important thing when considering research is the quality of it, which unfortunately requires a PhD do be done effectively.
Also, the legitimacy of different fields varies, I believe accounting research is conducted quite well, while other fields like exercise science can be really tough to trust.
Whenever I see a headline that begins with "New Study Proves...", I skip over it and move on to the next one. I love science, and I even like studies, but I have a big problem with the way studies are framed today, especially in the media. There are two major things wrong with these so-called scientific studies, which, combined, give us misleading and often outright incorrect headlines which many of us use to inform our decisions.
The first principle that is crucial to understand is that correlation is not the same thing as causation. For example, people who send their children to private schools are more likely to be convicted of stock market fraud than those who don't.
New Study Shows That Sending Children to Private School Could Lead to Criminal Behavior in Parents!
Well, no. People who have the money to send their kids to private school are more likely to be in a position to conduct stock frauds. There is a link between the two things, but it is not a causal link. In other words, sending your children to private school is not going to turn you into a criminal.
If you're afflicted with this idea that you need something to be perfect - whether that be a skill, deliverable, project, whatever - you probably know how crippling it can be. You'll spend 2x as much time getting something from 90% to 100% as you did getting it from 0% to 90%.
A while back I started thinking about how little I learned in college/high school, and then realizing that I was dead wrong. I definitely learned a lot of concepts, most of which are applied in totally different ways than my teachers and professors intended.
Take calculus for example. Don't really care about it in my day to day life...doesn't help me much. But the concept of limits has been awesome for understanding how to view perfection and growth in general.
If you're unfamiliar, a limit is a value that a function "approaches" - never to actually reach the value...which is why it's called the limit.
If you view your progression at anything as a limit approaching perfection, I guarantee you'll have a much better time in life. For example, if you're trying to ship out a project for a client, it's often best to ship it out at 95% rather than 100%. You might be thinking that this shortchanges the client's investment in you, but it's not true. Shipping out at 95% allows you to ship something else out at 95% instead of spending that time on the last 5% of the initial deliverable.