A lot of numbers are thrown at us everyday by people who want to convince us to agree with them. A good study adds credibility to the point being made. But not all studies are credible.
In The Price of Freedom is Personal Responsibility, I shared tips and tools to help verify the information you come across. This article shows how to evaluate the quality of research being referenced.
A Google search for “new study shows” displays over 650,000,000 results. That’s a lot of new studies! How do you determine which of these studies are credible?
You might question a study if the results don’t fit with your experience of reality. For example, those with personal experience eating when under stress would question today’s top Google search result – New Study Shows Stress Eating is a Myth. When you question the results, it’s worth taking a closer look. If you look closely in this article, you’ll see that only 59 people participated in the study – not nearly enough to draw such a huge conclusion.
But what about today’s number two search result? – Shocking New Study Shows Chemo Kills Half of Cancer Patients. It’s hard to simply dismiss this based on personal experience. However, if you read the article closely, you’ll see that although this looks like a credible study, these results were an indictment of one particular hospital, not chemo treatment. And if you click on the link to the research study itself, you’ll see that the conclusion of the study is that certain groups of people (type of cancer, age, general health, etc.) are at greater risk with chemo treatment. Nothing shocking or new about that. This article, published in Natural News, is an example of how the results of a credible study are manipulated to promote their own agenda, alternative medicine.
Unfortunately it has become a common practice for people to use questionable research or to manipulate research results to support their agenda.
Don’t Be Manipulated by Numbers
If you don’t know how to evaluate the quality of research, you can be easily manipulated by numbers or fall prey to confirmation bias (the tendency to see new information as confirmation of what you already believe).
The good news is, it’s not as hard as it might seem to learn. You don’t need to be a statistician to evaluate the quality of research. There are some simple things anyone can do that are not difficult at all.
4 Questions to Ask When Reading Articles That Reference Research
When numbers are being quoted, ask these questions to highlight potential concerns.
1. Who conducted the research?
Research produced or funded by groups with a strong political or commercial agenda is less likely to be trustworthy, since these groups have a vested interest in the study’s findings supporting their viewpoint.
2. Who is reporting the results?
Media coverage often seeks to be attention grabbing and succinct. As a result, they sometimes oversimplify the research or grab only a small piece and not the whole picture.
When those reporting the results have a political or commercial agenda, they may not fully or accurately summarize the original research. For example, the report of the study about chemo for cancer patients.
3. Are they jumping to conclusions?
Correlation does not imply causation. Just because two things are related does not mean one caused the other. For example, numerous studies showed that women taking hormone replacement therapy (HRT) also had a lower incidence of heart disease. For many years people assumed that HRT protected against heart disease. But later studies showed this is not true. They are simply correlated because the women who took HRT also took better care of their health in other areas as well.
4. Is this a one-time study?
A single study, no matter how good, needs to be viewed in the context of other research on the topic. What does other research on this topic show? Results obtained at one time might not hold true at another time.
6 Steps to Easily and Quickly Evaluate the Quality of Research
If the article reporting research raises questions for you, click on the link and take a look at the study itself. Beware of articles reporting research when no link to the study is provided.
Another advantage to looking directly at research results is that you often find interesting and useful information not included in articles referencing the study.
1. Consider the trustworthiness of researchers.
Are you familiar with the organization that conducted the research? Research conducted through a respected, known organization like Gallup or Harvard is more likely to be trustworthy.
2. Look at who was in the group.
Reported research is based on the assumption that the findings for a sample of people can be generalized to the larger population. How well does the sample represent the larger population? What groups are reported and how well do the respondents represent that group? What was the sample size? As demonstrated with the survey I discussed earlier, 59 people is too small to generalize from.
3. How was the data gathered?
What was the procedure for selecting the sample? When results are obtained by publishing an online survey, consider what motivated the individuals to take the survey. They might not be representative of the larger population. Take a look at the questions that were asked. If a test was given, was it a standardized test?
4. How was the data measured?
Was the data analyzed carefully or is this simply a study reporting numbers? Do they report measures of reliability and validity? Reliability means the study will produce the same results if you repeat it. Validity means it measures what it claims to measure.
5. How was the data analyzed?
Are the reported differences significant? A difference between 25% and 40% might not mean anything at all. A “significant” result is one that is unlikely to occur randomly or due to chance. Differences must be statistically “significant” to be considered meaningful.
6. Are the conclusions justified?
Is the sample size large enough to generalize from?
Are the conclusions based on sufficient data?
Are the reported differences meaningful statistically?
If the study is attributing causation, is more than correlation demonstrated?
Do the results make sense based on what you read?
Jesse, this article needs to be required reading for everyone. With all the half-truths and no=truths floating around this election, it is very easy to fall into group think. I wrote USA TODAY earlier this year, asking how they could survey 100 people and make a determination as to who was ahead in the polls. Huh?? It does not equate.
Another thing to examine is who is acting as “spokesperson” for the product or research. When you discover the financial ramifications for that person, it throws everything in question.
Brilliant, Jesse
Indeed, it is important to consider the source of both the research and also of the person or organization using the research. And even then, as you point out, a source that appears neutral like USA Today might publish questionable results simply in order to increase sales. Thanks for your insights, Eileen.
Excellent post Jesse! Back in June after completing spring quarter, I was going to write on a similar topic after what I learned in one of my courses. When I first went to college, it was back in the late 80’s and early 90’s. I didn’t own a computer until the mid-nineties, so having spent the last 1.5 years redoing sciences again (because they expire after 10 years), I have been able to fill the gap on some of these things! Very important to do since there is so we are inundated with so much info on the internet, you are exactly right! It can be overwhelming unless we have some guidelines that help us sift out the credible info from unreliable sources.
The course I took that really focused on finding and utilizing credible resources was a Nutrition class. One of the topics we covered was ‘Bad Science’. One of the questions asked for that assignment was:
‘How is it that news coverage of science research may create “bad science” out of legitimate science?’
My response (based on a specific section in our textbook):
‘According to Highlight 1 in Understanding Nutrition (Whitney, 2016), there are a variety of factors that contribute to misinformation from the news. Some of the reasons given had to do with pressure of a good story and timing reporters face that could prevent them from doing adequate research or ensuring there is enough data to support claims to improve accuracy. Another reason had to do with the way reporters write a compelling headline to attract readers. In order to grab a readers attention, specific language could be used that may misrepresent facts and ultimately, mislead the readers.’
Another important distinction made is differentiating between credible websites and general interest websites. For example, my blog and most of the blogs we follow here on Twitter would fall into the latter. Many of us have created a website on a certain topic or topics yet it does not really hold up to scientific scrutiny because most of us don’t back our statements up with research, etc. That’s not to say that ‘we’ and our message aren’t important. However, when it comes to some things, it would be better to cite a more credible source backed by research rather than MY ‘opinion’ on a certain topic.
Nutrition is one topic that DOES lead many astray because the average lay person may be confused on some of the titles various ‘nutritionists’ hold. There are those that have spent years in school training to be a certified nutritionist. Then there are those that call themselves ‘nutritionists’ because they spent a few months attending an online ‘school’ that really isn’t backed by legitimate research.
I wish I would have known some of these things I learned in this class about 8 years ago when I fell prey to an online school for $8000. It’s important to do our due diligence in research and on the surface, the website seemed credible. Looking back, I now know that it is not.
I can’t say that I could never fall for a less than credible website, person, or product again, however, I am equipped with a few more key areas to watch for when shopping.
I agree with all of your points. It takes extra time to do these things, yet it’s really important that we try to find the most credible information so we aren’t led astray, and ultimately, so we don’t lead others astray as well.
~Samantha Hall
Interesting how the lessons from the field of nutrition are similar to those of leadership. Thanks for sharing your learnings, Samantha!