Update (1/3/18) I’ve been overwhelmed with requests for the shorter guide, and the email address below no longer works. So I’ve uploaded a copy of the guide for anyone to download and share here: How to read and understand a scientific article. Please feel free to use it however you wish (although I’d appreciate being credited as the author). I apologize to everyone who emailed me and didn’t get a response! If you would like to let me know who you are and what you’re using it for in the comments below, I’d love to hear!
Update (8/30/14): I’ve written a shorter version of this guide for teachers to hand out to their classes. If you’d like a PDF, shoot me an email: jenniferraff (at) utexas (dot) edu.
Last week’s post (The truth about vaccinations: Your physician knows more than the University of Google) sparked a very lively discussion, with comments from several people trying to persuade me (and the other readers) that their paper disproved everything that I’d been saying. While I encourage you to go read the comments and contribute your own, here I want to focus on the much larger issue that this debate raised: what constitutes scientific authority?
It’s not just a fun academic problem. Getting the science wrong has very real consequences. For example, when a community doesn’t vaccinate children because they’re afraid of “toxins” and think that prayer (or diet, exercise, and “clean living”) is enough to prevent infection, outbreaks happen.
“Be skeptical. But when you get proof, accept proof.” –Michael Specter
What constitutes enough proof? Obviously everyone has a different answer to that question. But to form a truly educated opinion on a scientific subject, you need to become familiar with current research in that field. And to do that, you have to read the “primary research literature” (often just called “the literature”). You might have tried to read scientific papers before and been frustrated by the dense, stilted writing and the unfamiliar jargon. I remember feeling this way! Reading and understanding research papers is a skill which every single doctor and scientist has had to learn during graduate school. You can learn it too, but like any skill it takes patience and practice.
I want to help people become more scientifically literate, so I wrote this guide for how a layperson can approach reading and understanding a scientific research paper. It’s appropriate for someone who has no background whatsoever in science or medicine, and based on the assumption that he or she is doing this for the purpose of getting a basic understanding of a paper and deciding whether or not it’s a reputable study.
The type of scientific paper I’m discussing here is referred to as a primary research article. It’s a peer-reviewed report of new research on a specific question (or questions). Another useful type of publication is a review article. Review articles are also peer-reviewed, and don’t present new information, but summarize multiple primary research articles, to give a sense of the consensus, debates, and unanswered questions within a field. (I’m not going to say much more about them here, but be cautious about which review articles you read. Remember that they are only a snapshot of the research at the time they are published. A review article on, say, genome-wide association studies from 2001 is not going to be very informative in 2013. So much research has been done in the intervening years that the field has changed considerably).
Before you begin: some general advice
Reading a scientific paper is a completely different process than reading an article about science in a blog or newspaper. Not only do you read the sections in a different order than they’re presented, but you also have to take notes, read it multiple times, and probably go look up other papers for some of the details. Reading a single paper may take you a very long time at first. Be patient with yourself. The process will go much faster as you gain experience.
Most primary research papers will be divided into the following sections: Abstract, Introduction, Methods, Results, and Conclusions/Interpretations/Discussion. The order will depend on which journal it’s published in. Some journals have additional files (called Supplementary Online Information) which contain important details of the research, but are published online instead of in the article itself (make sure you don’t skip these files).
Before you begin reading, take note of the authors and their institutional affiliations. Some institutions (e.g. University of Texas) are well-respected; others (e.g. the Discovery Institute) may appear to be legitimate research institutions but are actually agenda-driven. Tip: google “Discovery Institute” to see why you don’t want to use it as a scientific authority on evolutionary theory.
Also take note of the journal in which it’s published. Reputable (biomedical) journals will be indexed by Pubmed. [EDIT: Several people have reminded me that non-biomedical journals won’t be on Pubmed, and they’re absolutely correct! (thanks for catching that, I apologize for being sloppy here). Check out Web of Science for a more complete index of science journals. And please feel free to share other resources in the comments!] Beware of questionable journals.
As you read, write down every single word that you don’t understand. You’re going to have to look them all up (yes, every one. I know it’s a total pain. But you won’t understand the paper if you don’t understand the vocabulary. Scientific words have extremely precise meanings).
Step-by-step instructions for reading a primary research article
1. Begin by reading the introduction, not the abstract.
The abstract is that dense first paragraph at the very beginning of a paper. In fact, that’s often the only part of a paper that many non-scientists read when they’re trying to build a scientific argument. (This is a terrible practice—don’t do it.). When I’m choosing papers to read, I decide what’s relevant to my interests based on a combination of the title and abstract. But when I’ve got a collection of papers assembled for deep reading, I always read the abstract last. I do this because abstracts contain a succinct summary of the entire paper, and I’m concerned about inadvertently becoming biased by the authors’ interpretation of the results.
2. Identify the BIG QUESTION.
Not “What is this paper about”, but “What problem is this entire field trying to solve?”
This helps you focus on why this research is being done. Look closely for evidence of agenda-motivated research.
3. Summarize the background in five sentences or less.
Here are some questions to guide you:
What work has been done before in this field to answer the BIG QUESTION? What are the limitations of that work? What, according to the authors, needs to be done next?
The five sentences part is a little arbitrary, but it forces you to be concise and really think about the context of this research. You need to be able to explain why this research has been done in order to understand it.
4. Identify the SPECIFIC QUESTION(S)
What exactly are the authors trying to answer with their research? There may be multiple questions, or just one. Write them down. If it’s the kind of research that tests one or more null hypotheses, identify it/them.
Not sure what a null hypothesis is? Go read this, then go back to my last post and read one of the papers that I linked to (like this one) and try to identify the null hypotheses in it. Keep in mind that not every paper will test a null hypothesis.
5. Identify the approach
What are the authors going to do to answer the SPECIFIC QUESTION(S)?
6. Now read the methods section. Draw a diagram for each experiment, showing exactly what the authors did.
I mean literally draw it. Include as much detail as you need to fully understand the work. As an example, here is what I drew to sort out the methods for a paper I read today (Battaglia et al. 2013: “The first peopling of South America: New evidence from Y-chromosome haplogroup Q”). This is much less detail than you’d probably need, because it’s a paper in my specialty and I use these methods all the time. But if you were reading this, and didn’t happen to know what “process data with reduced-median method using Network” means, you’d need to look that up.
You don’t need to understand the methods in enough detail to replicate the experiment—that’s something reviewers have to do—but you’re not ready to move on to the results until you can explain the basics of the methods to someone else.
7. Read the results section. Write one or more paragraphs to summarize the results for each experiment, each figure, and each table. Don’t yet try to decide what the results mean, just write down what they are.
You’ll find that, particularly in good papers, the majority of the results are summarized in the figures and tables. Pay careful attention to them! You may also need to go to the Supplementary Online Information file to find some of the results.
It is at this point where difficulties can arise if statistical tests are employed in the paper and you don’t have enough of a background to understand them. I can’t teach you stats in this post, but here, here, and here are some basic resources to help you. I STRONGLY advise you to become familiar with them.
THINGS TO PAY ATTENTION TO IN THE RESULTS SECTION:
-Any time the words “significant” or “non-significant” are used. These have precise statistical meanings. Read more about this here.
-If there are graphs, do they have error bars on them? For certain types of studies, a lack of confidence intervals is a major red flag.
-The sample size. Has the study been conducted on 10, or 10,000 people? (For some research purposes, a sample size of 10 is sufficient, but for most studies larger is better).
8. Do the results answer the SPECIFIC QUESTION(S)? What do you think they mean?
Don’t move on until you have thought about this. It’s okay to change your mind in light of the authors’ interpretation—in fact you probably will if you’re still a beginner at this kind of analysis—but it’s a really good habit to start forming your own interpretations before you read those of others.
9. Read the conclusion/discussion/Interpretation section.
What do the authors think the results mean? Do you agree with them? Can you come up with any alternative way of interpreting them? Do the authors identify any weaknesses in their own study? Do you see any that the authors missed? (Don’t assume they’re infallible!) What do they propose to do as a next step? Do you agree with that?
10. Now, go back to the beginning and read the abstract.
Does it match what the authors said in the paper? Does it fit with your interpretation of the paper?
11. FINAL STEP: (Don’t neglect doing this) What do other researchers say about this paper?
Who are the (acknowledged or self-proclaimed) experts in this particular field? Do they have criticisms of the study that you haven’t thought of, or do they generally support it?
Here’s a place where I do recommend you use google! But do it last, so you are better prepared to think critically about what other people say.
(12. This step may be optional for you, depending on why you’re reading a particular paper. But for me, it’s critical! I go through the “Literature cited” section to see what other papers the authors cited. This allows me to better identify the important papers in a particular field, see if the authors cited my own papers (KIDDING!….mostly), and find sources of useful ideas or techniques.)
Now brace for more conflict– next week we’re going to use this method to go through a paper on a controversial subject! Which one would you like to do? Shall we critique one of the papers I posted last week?
UPDATE: If you would like to see an example, you can find one here
———————————————————————————————————
I gratefully acknowledge Professors José Bonner and Bill Saxton for teaching me how to critically read and analyze scientific papers using this method. I’m honored to have the chance to pass along what they taught me.
Do you have anything to add to this guide? A completely different approach that you think is better? Additional questions? Links to other resources? Please share in the comments!
Violent metaphors 
One other great resource to get research articles besides pubmed is arXiv.org. They have particularly articles in the natural sciences, all open free of charge (the author uploads their own generated copy even for articles which eventually appear in embargoed journals). You can see if the article is accepted, published, or simply submitted in the meta description which is useful to tell if it has been peer-reviewed and which journal it is submitted to.
Another resource of course is Sci-hub to get nearly any primary research article you want instantly. Fantastic for particularly for citizens or scientists not at large research Universities with wide journal access, but perhaps less legal at this time.
I sent you an email looking for the PDF version of this information to relay to my students. If your email address at Utexas is no longer active, please provide another one that I can reach you. Thanks in advance.
Already sent to you.
I was interested in a .pdf of this article, as well. I teach an undergraduate research methods course and I would like to make this part of their supplemental reading. My email is cgrabowicz@ursinus.edu
Your article is quite helpful. Please send me the pdf file. My email: keduprof@163.com Thank you!
I would also like the PDF file for my students.
my email address is la5w7@mail.umkc.edu
Aloha! Great article. Could I also have permission/pdf to share with my nursing students? Thank you!
I would also really appreciate a PDF of this article to share with my students!! samorris@uab.edu. Thank you!!
You can all make a pdf copy of this by clicking “ctrl + p” which takes you the print page; or right click and select print. Then in the box asking you where to send it, click the arrow and select save as pdf.
Hello,
Thank you very much for posting this. I have been looking for something like this for ages. Great blog too! Keep up the good work.
Hello thanks for writing this, as a biology undergrad student I hope that it helps me better understand how to read science papers! Because at the moment I find it difficult. My question is: How do I go about doing number 11? Like where can I find critiques of papers? Especially recent ones!
There isn’t really a clearinghouse of critiques of papers. Instead, you find this either in review articles or in other primary articles that cite the article you’re reading. Often the commentary is subtle, so it takes practice to learn to read between the lines.
Thanks a lot it really helped me 🙂
Thought you’d like to know that I assign this to my Intro to Research class and also to our IMSD undergrads and grad students.
thank you so much!
Thanks you a lot! It helps me to read my papers in computer science. But I am a little bit confused how to distinguish SPECIFIC QUESTION from BIG QUESTION. Sometimes they look the same to me.
Hello, Jennifer,
With your permission, I would love a pdf version of this article to share with my undergraduate researchers in the McNair Scholars Program at the University of Central Missouri.
Thanks, Kari Azevedo
Hi Jennifer, I think your pieces are great. Since you say “If you like it, share it” I took the liberty of translating thiis useful piece into Dutch and sharing it through LinkedIn, here: https://www.linkedin.com/pulse/hoe-lees-en-begrijp-je-een-wetenschappelijk-artikel-voor-alex-verkade. Please let me know whether you are satisfied with my mention of your authorship. Cheers! Alex
Wow, this is just a silly off topic comment:
I saw your name, and then started to yearn for yummy yummy Cafe Noir cookies. Our family really enjoyed our visit to http://zaansmuseum.nl/?lang=en which included a functioning Verkade cookie manufacturing display (my husband’s father was from near Amsterdam, and this is one place hubby liked to go to as a child). It is a museum of windmills and Dutch history, especially industrial history.
Though, on further review is not quite off topic. I followed your links and discovered you are very active in science communication. Most excellent work, and I hope the children of my husband’s cousin get to learn and participate in what you have worked on (they should be in secondary school now).
Thanks Chris! There is a good probability that your husband’s cousin’s kids have seen at least some of our projects in school. And yes, Verkade is a last name famously associated with cookies and chocolate in the Netherlands. Unfortunately, no giant chocolate family inheritances for me. I’m not related to the cookie branch :).
Can you please send me a pdf of the article or even the shorter article I saw on Facebook? I teach biomedical science.
Dear Jennifer, i have read your article title How to read and understand scientific paper and would be most grateful if you could send to me the pdf version. Am a lecture in Ghana. Thank you in advance
Thank you for this post! I will distribute it to my intro to research methodology in Communication and Information Sciences class – they had to read their first academic journal article this week and found it pretty tough, so thanks in advance from them, too 🙂
Hi Jennifer,
This is a great piece. I would really appreciate a copy of the PDF to distribute to a first year course I am teaching at Australian Catholic University. I think this will really help to give students some tools to deal with and better understand the scientific literature.
Thanks, Nyree
Hi Jennifer: A colleague sent me this link to the guide for reading a scientific paper. I remember reading through it when it first came out and really appreciate what you’ve done. However, a couple of heads-up.
1) Your link to “questionable journals” takes us to Jeffrey Beall’s lists of predatory journals and publishers. Jeffrey is a friend of mine and I’ve had him speak to my Science Research Seminar class the last two years. CU Denver made Jeffrey take down the site because it was becoming a legal nightmare for the University; they felt they could no longer support him. When the FTC started coming after OMICS, OMICS in turn came after Jeffrey. It’s really unfortunate that we no longer have access to this great resource.
2) Your link to information about the null hypothesis has some really misleading information in it. From the get-go: “[T]he purpose of research is to answer a question or test a prediction, generally stated in the form of hypotheses.”
Hypotheses are not predictions. It’s a common misunderstanding and is taught incorrectly throughout a student’s science education. I published on this in the American Biology Teacher’s September 2015 issue. I’ve also blogged about it (https://mrdrscienceteacher.wordpress.com/2014/11/02/teaching-the-hypothesis/).
For example, the very first example of a hypothesis presented at the site is: “People who take a driver safety course will have a lower accident rate than those who do not take the course.”
This is a prediction based on the hypothesis that driver safety courses produce more cautious drivers.
The author goes on to say: “The research hypothesis (or hypotheses — there may be more than one) is our working hypothesis — our prediction, or what we expect to happen.”
No.
The author confuses experimental hypotheses with statistical null hypotheses. Indeed, the concept of the null comes from statistics, not the other way around.
Here is how I see it:
Hey, there are fruit flies on my bananas. Is it that fruit flies just like bananas, or that they are attracted to them because they are ripe?
I bet that the fruit flies are attracted to the bananas because they are ripe.
Hypothesis: Fruit flies are attracted to bananas when they are ripe.
Now at this point, yes, you could come up with a direct opposite statement and call it the null hypothesis, but I think that’s a waste of time. I also NEVER see this done in scientific publications, but maybe I haven’t looked closely enough. If a tested hypothesis is provided (and not written as a prediction, it is always written as a description of a possible pattern (generalizing hypothesis) or as an explanation (explanatory hypothesis). There is never an null statement paired with it that I have ever seen.
My argument is that the null statistical hypothesis is a mathematical expression borne from inferential statistics (e.g. O = E, u1 = u2, r = 0). The null statistical hypothesis states a current condition (not a prediction).
For Chi-square: The observed distribution and expected distributions ARE equal and any differences we see between them are just chance differences.
For the t-Test: The two populations from which the samples were taken HAVE equal means and the difference we observe between the sample means is merely a result of sampling error.
For Regression: There IS no relationship between the X and Y variables and the appearance of a relationship is accidental.
I disagree that a null hypothesis can be a prediction of what should happen as the result of an experiment if the experimental hypothesis is false. That just mucks it all up and adds to the confusion that at least some kinds of hypotheses can be written as predictions.
We never accept the statistical null hypothesis unless it is actually true and, for example, the observed distribution does in fact equal the expected distribution. If this is the case, then there is no need for doing a Chi-square Test. If two sample means turn out equal, then there is no reason to do a t-Test. However, if the error bars are huge, I’d be skeptical of concluding that the sample means are at all representative of the populations from which they were taken.
If the probability of getting the test statistic (e.g. Sum Chi-square) by accident is greater than 0.05, then the null statistical hypothesis cannot be confidently rejected. What I would say is this:
“I cannot reject the null statistical hypothesis given the greater than 0.05 probability that the differences between the observed distribution and the expected distribution occurred by accident.”
Thanks for sharing!
I feel that we all need this! Not only would it vastly improve vaccination rates, it will also shed some much needed light to the alarming fall in prevailing climate change denier/skepticism!
This is excellent – I plan on posting it for my philosophy students to read. One comment and question: you mention that scientific vocabulary is unfamiliar to non-scientists, and therefore it is important to look up every unfamiliar word. I agree – in my teaching I very much emphasize the understanding of basic definitions of concepts. However, as in science, much of philosophy vocabulary is specialized, meaning that you will not find the correct definition in regular dictionaries and encyclopedias. For philosophy, I recommend to my students the Stanford Online Encyclopedia of Philosophy (https://plato.stanford.edu), the Internet Encyclopedia of Philosophy (http://www.iep.utm.edu/eds/), and the hard-bound Encyclopedia of Philosophy available in most good research libraries. What science-specific dictionaries and encyclopedias would you recommend?
Excellent. I’ve added a link to this post in the online errata and additions section of my book (“Writing for Science Journals”; http://www.geoff-hart.com/books/journals/journal-book.htm). In addition to being an excellent primer for nonscientists, it’s a great article for teaching young scientists how to read — and write! — journal manuscripts. When I finally get around to writing the 2nd edition, I plan to integrate several of your points into the revised text.
I’d say that the title isn’t quite correct, it’s not just for non-scientists, it’s for non-academics. There are many people who would regard themselves as scientists but don’t read papers.
Reblogged this on Quaerere Propter Vērum and commented:
An absolute must-read.
A side note for people using Google Chrome or Opera browsers:
You can right click on any blog post or most article, select “Print” … and under “Destination” click the “Change” button.
In the menu, you should have an option to “Save as PDF.” It won’t be the neatly formatted pdf that Jennifer already supplied, but this can be handy for future use.
It can also save a blog post plus comments for those good discussions you might want to look at later.
Great. I’m sharing with our members via our latest magazine issue and will be crediting you as the author. Thanks for sharing.
Reblogged this on SCIANS and commented:
A nice guide for scientific publications reading !