So you read about a study that might be relevant or important to you, and want to know if it is anything you should put credence in. How should you go about this?

As an example, let’s take a study that that I just found at Medical News Today

New device can detect cancer in just a drop of blood

This should be important, since we know that early detection is key to successful treatment of cancer, and some cancers are very hard to detect in the early stages.

But is this web site a good place to find information? A little checking finds that Media Bias Fact Check gave them a high rating for Factual Reporting, and lists them as Pro-Science. It also has a caution that they sometimes review alternative medicine topics on a similar basis. So there is reason to think that this site is accurately portraying the information they have been given. Also, at the top, right under the author byline, is a note that says it was fact-checked, and gives the name of the person who fact-checked it. This is not definitive, of course, but we have enough information to think this is promising

Replication

The first thing I would want to know is whether the study has been replicated. And this poses a problem in general because the data you need is generally found in academic databases.

https://www.quora.com/How-can-I-find-a-replication-of-a-study

If you are on good terms with a local university they might let you get some access, or maybe if you signed up for a course you would get access. But it isn’t just as simple as doing a Google search, at least not yet. As I happen to have a niece who is a university librarian, I would offer the suggestion that becoming friends with one of those experts in locating information cannot do you any harm. They are the professionals at finding information.

If you can get access, a good starting point is the Web of Science, a database of citations. While it is online, it is subscription-based, so unless you are wealthy enough to afford a subscription you won’t get to it. Hence my suggestion you become good friends with a university librarian. In any case, the way academic studies work is based on citations, which means a reference to previous work. If someone is replicating a particular study, they would normally give a citation to that study.

One obvious problem is that by definition if the result comes from a brand new study, there has not even been time yet for anyone to replicate it.

In the case of this study, it is a new result, so no replication yet.  But a note of interest is that the data for their study is available to other researchers:

“The authors declare that all data supporting the findings of this study are available within the paper and its Supplementary Information files. The raw and analysed datasets generated during the study are available for research purposes from the corresponding author on reasonable request.”

Where was this published?

This study was published in Nature Biomedical Engineering. Nature is a British publication that is regarded as one of the premiere science journals. This is the kind of place you want to publish in if you can to burnish your credentials. So this is excellent. Now, we need to understand that even Nature has to occasionally retract articles  because of problems later discovered, but as a data point in our evaluation, this is a strong indication of quality. And we can look at the Abstract and other information, but not the entire paper which was just published (February 25, 2019) as I write this. But there is quite enough information to evaluate whether this is promising. They are quite happy to sell you the article or a subscription to the journal, but I did not choose to do that.

Author Credentials

If a study has come from an author with good credentials that raises its significance. But what does that mean exactly? First, the author(s) should have a good reputation and credentials in the field of the study. For example, if it as about treating diabetes I would want to see some pretty solid credentials in endocrinology before I put any confidence in it. A lot of people get PhDs, and in a lot of fields. This is a little easier to research since you can research the author(s) on Google. In the case of this study, there are eight authors, all at the University of Kansas in the U.S., and they are in departments like Bioanalytical Chemistry, Chemical Engineering, Pathology and Laboratory Medicine, and Cancer Center. These are exactly what you would want to see as credentials for authors on something like this. Furthermore, all of the funding sources are acknowledged, and include organizations like the National Institutes of Health, which is a major funder of medical research in the U.S.

How was the study done?

The technology involved is a device that uses microfluidic chips, which allow very small amounts of fluid, such a single drop of blood, to be analyzed. The chip is set up to detect a specific chemical in the blood that indicates cancer. To quote from the Abstract:

“We used the device for the detection—in 2 μl plasma samples from 20 ovarian cancer patients and 10 age-matched controls—of exosome subpopulations expressing CD24, epithelial cell adhesion molecule and folate receptor alpha proteins, and suggest exosomal folate receptor alpha as a potential biomarker for early detection and progression monitoring of ovarian cancer.”

For our purposes here, the key part is where they state that they had 20 ovarian cancer patients and 10 age-matched controls. That tells us that what we have here is a case-control study, so we know that this is around the middle of our hierarchy of evidence at this point. This type of study is not one that has randomization or any “blindness” in the study design. So not the strongest possible evidence, but probably appropriate for what they are doing at this stage. The patients were selected for ovarian cancer because that is known to be very hard to detect in early stages, so this matches what the study was looking for. Since there are only 20 patients in the study group, the sample size is not huge, but this is an initial study. With encouraging results we would look for either (or both) a bigger study by these people, or a replication by other researchers. If those continue to find positive results, that would certainly strengthen the case for this technology.

Summary

This is not an exhaustive analysis, but it does illustrate just how much information you can get from a little web searching, and how to use the data you find. The point, as I have said previously, is that I believe in evidence-based medicine, which means medicine informed by good quality scientific research. I do not have the qualifications to be a doctor, but I can use my intelligence to make sure that my doctor is basing his practice on sound science (he is). But in reading the press and social media, I suspect that somewhere in the neighborhood of 80% of the articles I read are not well-informed. And if people put in one-half the effort in getting health information that they put in to evaluate the best computer equipment, they would be a lot better off.

Listen to the audio version of this post on Hacker Public Radio!