What Makes a Good Research Study?
Find out what separates a solid research study from a so-so one.
One day you read online that drinking coffee reduces the chances of having age-related memory decline. You start drinking coffee. The next month you read that drinking coffee increases your chances of having age-related memory decline. What gives? In this article, you will learn how to “decipher” research studies to figure out what a research study is really saying – and what it doesn’t say. You’ll also discover how to tell if the reporting on a particular study was accurate or not. Ask yourself the following six questions when looking at a research study. Keep in mind these are just six of the many factors that make up a “clean” study.
1. Did the study use a placebo, and were the staff blinded to treatment?
The brain is very susceptible to placebos. There is evidence that even when you tell study subjects (participants) that they are getting a placebo, they improve (Carvalho, et al., 2016). In pharmaceutical studies, the US Food and Drug Administration (FDA) requires pharmaceutical companies to do double-blind placebo-controlled studies. This means that the study subjects, the physicians dispensing the drug, and the clinicians rating the subjects’ behavior don’t know what subjects are getting – drug or placebo. This eliminates a lot of bias, and it helps show whether the drug actually works.
2. Was there a bogus/sham treatment?
A bogus/sham treatment is one in which subjects are given a treatment that looks very much like the real treatment, except for one major difference. The bogus/sham treatment doesn’t actually provide the therapeutic part of the treatment. For example, some acupuncture studies use a sham/bogus treatment, such as a 2017 study by Ugurlu, et al. regarding acupuncture treatment for fibromyalgia.
Bogus/sham treatments, when compared to active treatments, help researchers discover whether the active treatment is what works, or the fact that people think they are getting the active treatment.
3. How many people were there in the study (N)?
Logic says the more people you have in a study, or the study’s “N”, the better chance you have of your study representing of the general population (the “generalizability” of a study). Let’s say you’re studying the effects of apple juice on ADHD symptoms, and you have a total N of ten people. By chance, seven of those ten people have severe ADHD, two have moderate ADHD, and one has mild ADHD. You now could throw off the results of your study because you have so many people with severe ADHD in the study. When you have more subjects, or a larger N, in a study, there is more of a chance that you would have people that have mild, moderate, and severe ADHD.
4. Were the study groups randomized?
A good study randomizes their subjects into the active treatment and placebo groups. This means that the subjects are in those particular groups by chance. This provides extra “backup” that the effects from a treatment were actually from that treatment, not from study staff bias.
5. Who conducted the research, and who is paying for it?
If the people that created a treatment are also testing a treatment, this is a concern. When you have a horse in the race, so to speak, it is more difficult to be unbiased. Another concern is if an entity with a vested interest in a particular study outcome is paying for that study. For example, if there is a study on the effectiveness of widgets, and the sole source of funding is Widgets are Wonderful, Inc., and the researchers are employees of Widgets are Wonderful, that study better have some seriously good methodology to help eliminate bias. Even better, an independent research group is funded by an organization without ties to the study outcome.
6. Was the article published in a refereed (peer-reviewed/scholarly) journal?
In a refereed journal, a manuscript is reviewed by other experts in the field before it is published as an article. The authors of the manuscript are not disclosed to the reviewers, in order to reduce possible bias. When we review manuscripts for a journal, there are three main categories: reject, meaning the article goes no further; accept, with revisions, meaning the authors must edit their article before resubmitting it for publication; and accept as written, which is rare, but once in a while there is a manuscript with such good research methodology and writing that no additional editing is needed.
When a journal is not refereed, the standard of inclusion into that journal is not as high. This means the quality of the research may not be up to the same standards. Look up the journal online to find out if it is a peer-reviewed journal.
If you don’t have university access, you can at least access the abstracts of journal articles at Google Scholar. The abstract lets you know the study’s methodology, the number of study subjects, the outcomes, and the author’s conclusions.
You may also see the term “open-access” used to describe a journal. An open-access journal is one that users can freely access, without a subscription or fees. Some open-access journals are peer-reviewed, some are not.
Copyright 2018 Sarkis Media
Originally posted at Psychology Today at: What Makes a Good Research Study?
