Update (1/3/18) I’ve been overwhelmed with requests for the shorter guide, and the email address below no longer works. So I’ve uploaded a copy of the guide for anyone to download and share here: How to read and understand a scientific article. Please feel free to use it however you wish (although I’d appreciate being credited as the author). I apologize to everyone who emailed me and didn’t get a response! If you would like to let me know who you are and what you’re using it for in the comments below, I’d love to hear!
Update (8/30/14): I’ve written a shorter version of this guide for teachers to hand out to their classes. If you’d like a PDF, shoot me an email: jenniferraff (at) utexas (dot) edu.
Last week’s post (The truth about vaccinations: Your physician knows more than the University of Google) sparked a very lively discussion, with comments from several people trying to persuade me (and the other readers) that their paper disproved everything that I’d been saying. While I encourage you to go read the comments and contribute your own, here I want to focus on the much larger issue that this debate raised: what constitutes scientific authority?
It’s not just a fun academic problem. Getting the science wrong has very real consequences. For example, when a community doesn’t vaccinate children because they’re afraid of “toxins” and think that prayer (or diet, exercise, and “clean living”) is enough to prevent infection, outbreaks happen.
“Be skeptical. But when you get proof, accept proof.” –Michael Specter
What constitutes enough proof? Obviously everyone has a different answer to that question. But to form a truly educated opinion on a scientific subject, you need to become familiar with current research in that field. And to do that, you have to read the “primary research literature” (often just called “the literature”). You might have tried to read scientific papers before and been frustrated by the dense, stilted writing and the unfamiliar jargon. I remember feeling this way! Reading and understanding research papers is a skill which every single doctor and scientist has had to learn during graduate school. You can learn it too, but like any skill it takes patience and practice.
I want to help people become more scientifically literate, so I wrote this guide for how a layperson can approach reading and understanding a scientific research paper. It’s appropriate for someone who has no background whatsoever in science or medicine, and based on the assumption that he or she is doing this for the purpose of getting a basic understanding of a paper and deciding whether or not it’s a reputable study.
The type of scientific paper I’m discussing here is referred to as a primary research article. It’s a peer-reviewed report of new research on a specific question (or questions). Another useful type of publication is a review article. Review articles are also peer-reviewed, and don’t present new information, but summarize multiple primary research articles, to give a sense of the consensus, debates, and unanswered questions within a field. (I’m not going to say much more about them here, but be cautious about which review articles you read. Remember that they are only a snapshot of the research at the time they are published. A review article on, say, genome-wide association studies from 2001 is not going to be very informative in 2013. So much research has been done in the intervening years that the field has changed considerably).
Before you begin: some general advice
Reading a scientific paper is a completely different process than reading an article about science in a blog or newspaper. Not only do you read the sections in a different order than they’re presented, but you also have to take notes, read it multiple times, and probably go look up other papers for some of the details. Reading a single paper may take you a very long time at first. Be patient with yourself. The process will go much faster as you gain experience.
Most primary research papers will be divided into the following sections: Abstract, Introduction, Methods, Results, and Conclusions/Interpretations/Discussion. The order will depend on which journal it’s published in. Some journals have additional files (called Supplementary Online Information) which contain important details of the research, but are published online instead of in the article itself (make sure you don’t skip these files).
Before you begin reading, take note of the authors and their institutional affiliations. Some institutions (e.g. University of Texas) are well-respected; others (e.g. the Discovery Institute) may appear to be legitimate research institutions but are actually agenda-driven. Tip: google “Discovery Institute” to see why you don’t want to use it as a scientific authority on evolutionary theory.
Also take note of the journal in which it’s published. Reputable (biomedical) journals will be indexed by Pubmed. [EDIT: Several people have reminded me that non-biomedical journals won’t be on Pubmed, and they’re absolutely correct! (thanks for catching that, I apologize for being sloppy here). Check out Web of Science for a more complete index of science journals. And please feel free to share other resources in the comments!] Beware of questionable journals.
As you read, write down every single word that you don’t understand. You’re going to have to look them all up (yes, every one. I know it’s a total pain. But you won’t understand the paper if you don’t understand the vocabulary. Scientific words have extremely precise meanings).
Step-by-step instructions for reading a primary research article
1. Begin by reading the introduction, not the abstract.
The abstract is that dense first paragraph at the very beginning of a paper. In fact, that’s often the only part of a paper that many non-scientists read when they’re trying to build a scientific argument. (This is a terrible practice—don’t do it.). When I’m choosing papers to read, I decide what’s relevant to my interests based on a combination of the title and abstract. But when I’ve got a collection of papers assembled for deep reading, I always read the abstract last. I do this because abstracts contain a succinct summary of the entire paper, and I’m concerned about inadvertently becoming biased by the authors’ interpretation of the results.
2. Identify the BIG QUESTION.
Not “What is this paper about”, but “What problem is this entire field trying to solve?”
This helps you focus on why this research is being done. Look closely for evidence of agenda-motivated research.
3. Summarize the background in five sentences or less.
Here are some questions to guide you:
What work has been done before in this field to answer the BIG QUESTION? What are the limitations of that work? What, according to the authors, needs to be done next?
The five sentences part is a little arbitrary, but it forces you to be concise and really think about the context of this research. You need to be able to explain why this research has been done in order to understand it.
4. Identify the SPECIFIC QUESTION(S)
What exactly are the authors trying to answer with their research? There may be multiple questions, or just one. Write them down. If it’s the kind of research that tests one or more null hypotheses, identify it/them.
Not sure what a null hypothesis is? Go read this, then go back to my last post and read one of the papers that I linked to (like this one) and try to identify the null hypotheses in it. Keep in mind that not every paper will test a null hypothesis.
5. Identify the approach
What are the authors going to do to answer the SPECIFIC QUESTION(S)?
6. Now read the methods section. Draw a diagram for each experiment, showing exactly what the authors did.
I mean literally draw it. Include as much detail as you need to fully understand the work. As an example, here is what I drew to sort out the methods for a paper I read today (Battaglia et al. 2013: “The first peopling of South America: New evidence from Y-chromosome haplogroup Q”). This is much less detail than you’d probably need, because it’s a paper in my specialty and I use these methods all the time. But if you were reading this, and didn’t happen to know what “process data with reduced-median method using Network” means, you’d need to look that up.
You don’t need to understand the methods in enough detail to replicate the experiment—that’s something reviewers have to do—but you’re not ready to move on to the results until you can explain the basics of the methods to someone else.
7. Read the results section. Write one or more paragraphs to summarize the results for each experiment, each figure, and each table. Don’t yet try to decide what the results mean, just write down what they are.
You’ll find that, particularly in good papers, the majority of the results are summarized in the figures and tables. Pay careful attention to them! You may also need to go to the Supplementary Online Information file to find some of the results.
It is at this point where difficulties can arise if statistical tests are employed in the paper and you don’t have enough of a background to understand them. I can’t teach you stats in this post, but here, here, and here are some basic resources to help you. I STRONGLY advise you to become familiar with them.
THINGS TO PAY ATTENTION TO IN THE RESULTS SECTION:
-Any time the words “significant” or “non-significant” are used. These have precise statistical meanings. Read more about this here.
-If there are graphs, do they have error bars on them? For certain types of studies, a lack of confidence intervals is a major red flag.
-The sample size. Has the study been conducted on 10, or 10,000 people? (For some research purposes, a sample size of 10 is sufficient, but for most studies larger is better).
8. Do the results answer the SPECIFIC QUESTION(S)? What do you think they mean?
Don’t move on until you have thought about this. It’s okay to change your mind in light of the authors’ interpretation—in fact you probably will if you’re still a beginner at this kind of analysis—but it’s a really good habit to start forming your own interpretations before you read those of others.
9. Read the conclusion/discussion/Interpretation section.
What do the authors think the results mean? Do you agree with them? Can you come up with any alternative way of interpreting them? Do the authors identify any weaknesses in their own study? Do you see any that the authors missed? (Don’t assume they’re infallible!) What do they propose to do as a next step? Do you agree with that?
10. Now, go back to the beginning and read the abstract.
Does it match what the authors said in the paper? Does it fit with your interpretation of the paper?
11. FINAL STEP: (Don’t neglect doing this) What do other researchers say about this paper?
Who are the (acknowledged or self-proclaimed) experts in this particular field? Do they have criticisms of the study that you haven’t thought of, or do they generally support it?
Here’s a place where I do recommend you use google! But do it last, so you are better prepared to think critically about what other people say.
(12. This step may be optional for you, depending on why you’re reading a particular paper. But for me, it’s critical! I go through the “Literature cited” section to see what other papers the authors cited. This allows me to better identify the important papers in a particular field, see if the authors cited my own papers (KIDDING!….mostly), and find sources of useful ideas or techniques.)
Now brace for more conflict– next week we’re going to use this method to go through a paper on a controversial subject! Which one would you like to do? Shall we critique one of the papers I posted last week?
UPDATE: If you would like to see an example, you can find one here
———————————————————————————————————
I gratefully acknowledge Professors José Bonner and Bill Saxton for teaching me how to critically read and analyze scientific papers using this method. I’m honored to have the chance to pass along what they taught me.
Do you have anything to add to this guide? A completely different approach that you think is better? Additional questions? Links to other resources? Please share in the comments!
Violent metaphors 
Reblogged this on To Talk Like This and Act Like That and commented:
This is gold!!!
Puis-je emprunter deux-trois paragraphes pour un site personnel ?
I’ve been reading many scientific articles for a graduate course I’m taking and one thing I would disagree on is that scientific terms have an extremely precise definition. There are articles out there devoted to defining and debating scientific terms.
I do agree that knowing the vocabulary is extremely important and you should look up any terms you do not know, but be aware that certain terms, such as community or climax in earlier papers (1900’s), have a definition that depends on the author.
I would also recommend reading some of these earlier papers as they give great insight as to how we arrived at certain theories or practices of science today.
This seems to me to be an excellent attempt to explain how to read a paper. A lot of scientists could benefit from it too.
I have only one small quibble. The reference given to explain the null hypothesis seems to me to be too inaccurate to be helpful, even for classical significance testing. And, more to the point, it doesn’t discuss the idea of the false discovery rate. Neglect of this is probably responsible for the lack of reproducibility that has plagued some branches of science. See http://www.dcscience.net/?p=6518 for an account of these ideas.
Some good points mentioned. Definitely note what has been omitted and always look for experimrntal controls. Always be sceptical when reading a paper and try to find alternative explanations…
I read this post last year when you first made it (it was going around some of the skeptic groups on facebook), and liked it so much I bookmarked it. I have referred to it several times since, as I am only an untrained amateur, but I really feel like it’s helped me get better. Now that I have my own blog, I hope you don’t mind me reblogging it at abreathofreason.com
Reblogged this on A Breath of Reason and commented:
This article is an important tool for anyone who isn’t formally trained in analyzing and interpreting a scientific paper. I have referred to it often, and I hope you will too.
Reblogged this on Deep thought and commented:
Read & Understand a scientific paper by Jennifer Raff
Thank you! I’m going to need this for my topic paper.
Thank you! I really like this post and I going to practice with this method to analyze and be more critical.
Reblogged this on Writing and other stuff and commented:
Wow. Thanks for sharing this.
Hi,
Will it be OK if I translate this post into Chinese language and post it on my blog? (This original post will of course be clearly attributed.)
Yes of course! Would you send me a link to the finished version? jenniferraff (at) utexas (dot) edu
Sure!
I’m always flummoxed by concerns about conflicts of interest among study authors AND study design as a layman. Researchers often comment that drug companies in particular will cherry pick participants, pull participants early who are showing side effects, and other chicanery to help skew findings. And my perception is that research studies are often sponsored by companies or special interests with an agenda. Example: There recently was a Canadian study carrying on about the lethality of eggs when independent researchers revealed the authors had very deep monetary connections with an anti egg, anti dairy group. If the scientific community allows this sort of thing to occur, I feel I’m pretty much powerless to sort it out. I don’t know that your reading approach, as good as it is, would help me. And even with a bonafide study, can the results even be duplicated? The media, and most laymen, carry on about ONE single study. Drug studies typically are what, three months long? No wonder we’ve seen patients develop cancer or heart failure with certain diabetes drugs with such a short evaluation period. Be careful what a physician suggests about a new drug…he or she might have been given cooked data by a drug sales person. I’m not interested in being a guinea pig for big pharma.
Frank Weir,
I think that’s a fair question. Many initial studies turn out to be incomplete or flat out wrong. So what’s one too do?
1) Rely on hundreds, thousands, or tens of thousands of studies, many done by people and agencies that are not beholden to Pharma. Look at what the people with ties to Pharma do for their own kids.
2) Look at what doctors do for their own kids. Not just one, but the vast, overwhelming majority of doctors, biological scientists, pharmaceutical CEOs, etc.
Gewisn: Now how in heavens name can one find out such intimate details about what people with ties to Big Pharma or MDs do for their own kids? What if you know no one that fits that description, not even socially (most people)? And where are you going to find funded researchers who are “not beholden to Pharma”?
Raises hand. I’m a funded researcher (by the NSF), no ties to big pharma. I’m fully up to date on all my vaccines, including yellow fever and rabies. Every physician that I know socially (including my own brother-in-law) is fully vaccinated, and their children were vaccinated according to the recommended schedule. Why? Because we understand how vaccines work. It’s as simple as that.
You’re right. I’m sure it’s impossible. Why should anyone even try?
Oh, wait, maybe because millions of kids suffering and deaths stand in the balance.
What to do? What to do?
Could you ask the people if they immunize their kids? Would you ask your doctor if he/she vaccinated their own kids?
Could you look at studies done in countries that are not so beholden to Big Pharma?
“And where are you going to find funded researchers who are “not beholden to Pharma”?”
Health insurance companies.
If you had ever dealt with one, you will soon realize they want to pay pharmaceutical companies as little as possible. This has caused problems when they refused to pay for anything except generics for some, and seem to always be keen on reminding us to get preventative care like blood tests, etc.
The CDC Vaccine Safety Datalink has participation from several health maintenance organizations:
http://www.cdc.gov/vaccinesafety/Activities/VSD.html
This recent study was done by health insurance:
JAMA. 2015 Apr 21;313(15):1534-40. doi: 10.1001/jama.2015.3077.
Autism occurrence by MMR vaccine status among US children with older siblings with and without autism.
By the way, I noticed something on how health insurance policies changed after they had to pay for lots of hospital admissions. In 1992 the health insurance company refused to pay for the “well child” checkup after age three, so they did not cover my oldest son’s four year old booster shots. But then the numbers came in on the 1990 measles epidemic and increases in pertussis hospitalizations.
Not long after that they were paying for the “well child” checkups and vaccines way past age four for my younger kids. It seemed they learned vaccines were cheaper in the long run. Though it was not all of them. They only started to pay for pediatric influenza vaccines a bit over five or years ago (there was a period where I took kids to the pharmacy for flu vaccine, because it was cheaper than doctor’s office, now the health insurance covers it at both places).
Also see:
Pediatrics. 2014 Apr;133(4):577-85.
Economic Evaluation of the Routine Childhood Immunization Program in the United States, 2009.
J Infect Dis. 2004 May 1;189 Suppl 1:S131-45.
An economic analysis of the current universal 2-dose measles-mumps-rubella vaccination program in the United States.
West J Med. 1996 Jul-Aug;165(1-2):20-5.
Pediatric hospital admissions for measles. Lessons from the 1990 epidemic.
West J Med. 1993 Oct;159(4):455-64.
Measles epidemic from failure to immunize.
Reblogged this on EIRE60 and commented:
This should be mandatory before going on the Net.
Thank you! It was really helpful 🙂
Reblogged this on Jumping Gene and commented:
Pretty important. Perhaps, schools need to delve into a set of how-to’s before assigning paper reading homeworks.
thank !! “How to read and understand a scientific paper”.
Hi Jennifer, thank you for this information. I hope all of the parents that are vehemently opposed to vaccines pause for a moment, consider the facts, have faith in the scientifically-rigorous independent research that is being conducted and use vaccines on themselves and their children.
There are excellent and free resources here by the Critical Appraisal Skills Programme (CASP) (a non profit organisation) http://www.casp-uk.net/#!casp-tools-checklists/c18f8
Reblogged this on gwoods09's Blog.
Hi,
thank you for the very useful informations. I would like to ask you if I can do a translation of the article in Italian language, and then publish it on my website (quoting the source, of course).
Yes you may! Please send me a link to the finished version? jenniferraff (at) utexas (dot) edu. Thanks!
Yes, of course! Thank you.
may peace be you .very informative work for beginners
Great post! I wish I could have found it several months earlier. It would have save me from lots of frustration trying to understand how to read the papers. I will definitely try this method.
I have found a blog http://goo.gl/gKAU3Z advising not to read discussion and interpretation of the original authors (unless you want to get familiar with competing theories). I find that sometimes I do understand results differently than the original authors. After I read the discussion, I sometimes feel that I have unfortunately misunderstood the results. I guess the confidence in your personal interpretation of what results are saying comes with the experience and increasing knowledge in your field. I guess the most dangerous point for a young researcher is to learn to remain objective and make sure you don’t just accept certain interpretation, because you don’t fully understand the results. Your post gives a nice guide to help with that.
Another big challenge for a young researcher is a transition from the notes on other peoples’ work to justify your own fundings. Have you ever thought on writing a post about writing a scientific paper?