How to read a vaccine safety study: an example.

I want to expand upon my guide on how to read a scientific paper by working through an example. You may not have the time or ability to read a paper in as much depth as I outline below. But my goal with this series of posts is to help people feel less intimidated by research papers by giving them a framework for reading them. I want to make these tools available, should you wish to use them.

I decided to choose as my first example a paper that would also be useful for people grappling with the question of vaccine safety, so that they can become familiar with how these studies are conducted and better understand some of the terminology.

Think hard. Go deeper into the paper.
Think hard. Go deeper into the paper.

The paper I’ll go through is:

Toback et al., 2013.
“A postlicensure evaluation of the safety of Ann Arbor strain live attenuated
influenza vaccine in children 24–59 months of age” Vaccine 31: 1812– 1818

It’s freely accessible here.

Before you go any further, you’ll find it most useful to try to read through the paper on your own first and take notes. If this is your first time going through a research paper, you may find it difficult to judge the quality of the results and whether the authors are interpreting them correctly. My guide should help if you go through it step-by-step, but if it’s overwhelming, start small. At minimum, try to identify the Big Question, the Specific Question(s), and the Approach from the Introduction section. While the methods might be murky for you, read them over anyway and try to figure out answers to these questions:

–How many subjects participated in this research? What characteristics did they have that made the authors choose them for this study?

What exactly did the authors measure, and how?

What were the controls for this study?

Be sure to look up terminology you don’t understand. There’s a list of helpful abbreviations on the first page.

Next, try to identify and write down the main results of each experiment. Remember that some are likely to be in the tables, so make sure you look at them.

Finally think about what the results mean before you read the authors’ discussion of them.

Done? Okay, here’s what I did with the paper, following my own guide. See if you agree or disagree with me.

1. I took note of the authors and their institutional affiliations.

Despite what the Discovery Institute says, I don’t think you should necessarily discount papers solely on the institutional affiliations of the authors. However, you should be alert for potential bias if you see that they do come from institutions with a history of advocating for a specific position, or with a vested interest in a particular outcome for the experiments.

The first author and the last authors are (at least in biomedical fields) generally the most important. The first author tends to be the person who did most of the work, and the last author tends to be the person who was the overall supervisor (or Principal Investigator) of the research. In this paper, the first author worked for the company MedImmune at the time of this study (and now works for Gilead Sciences). The last author (and many of the other authors) works for the Kaiser Permanente Vaccine Study Center. The research was funded by MedImmune. The authors of this paper also provide disclosures as to their financial interests in the research. You can find that information at the very end of the paper, under the Acknowledgments. Think about how these affiliations might affect the research

2. I moved next to the Introduction (skipping the abstract!), to find the Big Question.

Remember that the Big Question is very broad, bigger than the scope of a single paper. In this case it could be: Are live-attenuated vaccines safe for children?

3. Next, I looked up some terms, and summarized the background (albeit a bit oversimplified) in a few sentences:

Important definitions for this paper:

Post-licensure vaccine safety studies
Cohort study
Case-control study
The CDC’s fact sheet on nasal vaccine
Live attenuated vaccine

Background summary:

a. The Ann Arbor strain live attenuated influenza vaccine (LAIV) has been approved for use in healthy children between 24–59.

b. One previous study showed an increase in wheezing in children of this age who received this vaccine.

c. It’s necessary to test whether children who received this vaccine had any adverse reactions.

4. From the abstract, I also determined the Specific Question for this research.

Remember that the Specific Question is what the paper is actually testing.

What are all adverse events reported for 24-59 month old children who received the LAIV?

5. From the Introduction and the Study Design, I determined the approach:

Compare rates of medically attended events (MAE), serious adverse events (SAE), perspecified diagnoses of interest (PSDI), and “rare events potentially related to wild-type influenza (WTI)” between children who received the vaccine and controls

6. Next, I went to the methods section to outline what they did.
(Given that this is a blog intended for a general audience, I’m going to simplify the summary of their methods somewhat.)

A. Identified a pool of 25,000 children between 24-59 months who were immunized with LAIV (= “analysis cohort”).

B. Removed all children with high-risk medical conditions from the cohort. This included children with asthma. This is important to note, because “wheezing” was an adverse effect reported to be higher among LAIV recipients in a previous study. (How do you think that might affect the results of this current study?)

C. Created three control groups for comparison:

1. Within-cohort (this was a division of vaccinated children into different risk intervals to compare against each other: 0-3 days post-vaccination was compared to 4-42 days post-vaccination; 0-21 days post-vaccination was compared to 22-42 days post-vaccination).

2. Unvaccinated children

3. Children vaccinated with a different type of influenza vaccine (trivalent inactivated influenza vaccine, or TIV).Here is the difference between the two types.

D. The control groups were matched with the test group by age (in months) and geographic distribution. Matching variables is important because it removes certain things that could complicate determining whether an association is real or not. Here’s one explanation of it. (If someone has a better one, please share it in the comments!)

E. The authors then identified medically attended adverse events (MAEs) for all cases and controls from medical records. Based on previous work (cited), they defined some MAEs which they hypothesized were possibly caused by vaccination (including respiratory and gastrointestinal tract issues, asthma, wheezing, and bacterial infections). They gave patients with these issues special scrutiny.

F. Rates of adverse events were statistically compared between cohorts.

7. Next, I went to the Results section to summarize the findings of the authors.

There are many results, and the comparisons are pretty intricate. I encourage you to study Figures 1-2 (page 1814-5), to get an overall picture of statistically significant differences in events between the cohorts. As with the methods, I simplified this a bit for a general audience below. (Please let me know in the comments if I left out anything you think is important)

This study included:

-28,226 children between the ages of 24-59 months who were vaccinated with LAIV

-27,937 children vaccinated with TIV (the alternative vaccine)

-25,981 unvaccinated children.

Do you think this sample size is large enough?

The authors divided the results into three categories:

A. Serious Adverse Events

The authors looked at the medical records of all 25,000+ children in each category and compared rates of ALL serious medical issues between them during the study period following vaccination. The rates of serious adverse events (SAEs) through 42 days post-vaccination were not significantly different between the LAIV children and any of the control groups.

The SAEs that did occur multiple times in the group of LAIV-vaccinated children were:

-4 cases of trauma
-4 cases of pneumonia
-3 cases of dehydration
-2 cases of congenital anomaly
-2 cases of urinary tract infection.

The authors thought that these particular events were not likely related to the vaccine. Do you agree or disagree?

Two cases occurred which the authors did hypothesize were linked to the immunization: a boy developed a case of respiratory syncytial virus infection, and another case of a boy with intussusception and viral infection. Both recovered completely with medical attention.

B. Hospitalization

The authors compared rates of hospitalization among all children in the study, and broke them down by diagnosis. A significant increase in events diagnosed in hospitalized children of the LAIV recipient group (vs. unvaccinated) included sleep apnea (5 cases), tonsillitis (5 cases), and any event (n=21). The authors report that “sleep apnea and tonsillitis occurred in 5 children who were vaccinated on the same day as their preoperative examination for an elective tonsillectomy and adenoidectomy procedure and were hospitalized after surgery with those diagnoses” (pp 1814-1815).

A significant decrease in events diagnosed in hospitalized children of the LAIV recipient group in some conditions:

The overall rate of hospitalization within 180 days postvaccination with LAIV was significantly lower compared to:

Unvaccinated controls in 24-35 month cohort
TIV recipients of all ages
24-35 month cohort
Asthma/RAD events within 21, 42 days postvaccination with LAIV were decreased compared to TIV-vaccinated controls and “any event in comparison to TIV-vaccinated and within-cohort.” (p1815).

C. All medically attended events (MAE)

One MAE rate was significantly higher among LAIV recipients when compared to all 3 control groups:

Urticaria (hives):

One MAE rate was significantly decreased among LAIV recipients across all comparison groups:

asthma/RAD (within 21 days postvaccination in all subjects, and 36-59 months old)

Children who received the LAIV vaccine had significantly higher frequencies of influenza infection than controls. Nearly all (99%) of these cases were the H1N1 strain (which LAIV wasn’t designed to protect against). H1N1 vaccination frequency was the lowest in the LAIV group.

The authors also adjusted for the fact that they were performing multiple comparisons (I won’t get into the statistical rational here, but please feel free to discuss the details in the comments if you like!). When they did this, 2 wheezing/SOB rates were increased among the LAIV group vs unvaccinated controls: in 36-59 months and 24-59 months. 42 AW events were lower in LAIV compared with TIV-vaccinated and unvaccinated controls (this included “respiratory tract events, AW events, any events, asthma/RAD, cough, otitis media, upper respiratory tract infection, viral syndrom, and wheezing/SOB” p1815).

Have you made it through all of that? Good!

8. Now it’s time to think hard about what all of this means. From this point forward, I’m not going to give you my answers—I want you to come up with your own (Yes, I know that’s infuriating, but I am a teacher, after all).

Go back to the Specific Question. Did the results answer it? Can you say that the vaccine did or did not adversely affected children’s health, according to this study? Do you think that the number of children included in this study was large enough?

9. Now read the Discussion and Conclusion sections.

What did the authors say the results mean? (Do you agree? Opinions might differ here, and I encourage you to discuss them in the comments)

For example, the authors provide a hypothesis why the LAIV children had higher rates of H1N1 infection. What is it? Does it seem plausible to you? How would you go about testing that hypothesis in a follow-up study?

Did the authors identify any weaknesses in their study design? Can you see any?

What do they/you think should be done next?

Would you feel confident giving your own child this vaccine, based on this safety study? Why or why not?

Do you see any evidence of bias in the authors’ research?

10. Now you can read the abstract.

Did it fairly represent the paper?

11. I encourage you (if you’re really interested in the topic of childhood influenza vaccines) to now go read what other people have to say about this (and other) papers.

This is the part where you “do your own research”. But save it for the final step, so that you’re better informed than when you started.

Final note:
If you’re just a casual reader, I understand that you might not be able to do every single step here. However, the more important this subject is to you, the more in-depth you should try to go. You now have a framework for how to go about doing this, should you need it.

Please understand that the experts in the field are reading these papers in even more detail than I’ve given you. They may be repeating these studies, or following up on them to test new ideas that they’ve gleaned from the results. Now that you have a glimpse into the process, does that alter your perception of scientific research? Nothing irritates me more than when someone writes a forum post, or creates a meme for Facebook trying to undermine a scientific study without actually reading beyond the title or abstract. Even if you don’t care to do it yourself, I hope that this series on my blog will help you see the need to look beyond surface critiques, in order to identify the real strengths/weaknesses of research.

Many thanks to everyone who have commented and shared my posts! I really appreciate your participation in these discussions, and hope that we can continue them. Feel free to share your thoughts/questions/concerns in the comments section.


Edited to add: Here are two more excellent resources for reading and understanding scientific papers. If you still don’t feel like you understand the process after reading this post, I encourage you to go read these powerpoints from the American Society of Plant Biology:

Click to access How%20to%20Read%20a%20Scientific%20Paper%20M%20Williams%20Mar%202013.pdf

Click to access CaseStudy%20for%20How%20to%20Read%20a%20Sci%20Paper%20M%20Williams%20Mar%202013.pdf

30 thoughts on “How to read a vaccine safety study: an example.

  1. Marni September 8, 2013 / 11:00 pm

    I don’t know how useful this site might be to those learning about medical research for the first time, but it seems to me to have potential for promoting understanding and I think it is more easily understood since people can click on links to get terms explained then return to the basic article to continue reading. You might give it a look:

    Keep up the great work!

      • Ruth Edlund September 12, 2013 / 1:51 pm

        Also: “gastrointenstinal”

    • Anonymous July 26, 2015 / 10:03 am

      Wonderful job, especially the way you go into detail with an actual linked example. Thanks.

  2. Ruth Edlund September 10, 2013 / 9:56 am

    Pet peeve: it’s principal investigator, not “principle.” It would make me happy if that was corrected in this fine article.

    • Jennifer Raff September 10, 2013 / 10:00 am

      Corrected! Yikes, thanks for catching that.

  3. SkepticalRaptor September 10, 2013 / 6:02 pm

    I cannot begin to start pointing out all of your logical fallacies and, to be honest, lack of critical thinking skills towards science.

    Let’s start with the Authors working for Kaiser Permanente, one of the most innovative, scientifically managed health care organizations in the USA. And what makes them so great as a resource for investigating vaccines and potential adverse events is that all patient data is centrally stored, from lab results, to physician’s visits, to emergency room visits, to whatever. You can point out precisely when a patient received a vaccine, then track any potential causal relationships with an adverse effect.

    Your implied, but fallacious, passive-aggressive ad hominem “think about what that means” made me laugh hysterically. What it means is that we’ve got some of the most accurate, in-depth, important information available to us to put together a retrospective study.

    Good luck with your Dunning-Kruger effect, because there are real people with outstanding educations who got off their lazy behind and learned real science who would disagree with you. Like me.

    • Jennifer Raff September 10, 2013 / 6:20 pm

      I believe you’ve completely misread my intention. I know exactly what Kaiser Permanente is…my suggestion that the reader think about what working for them means ISN’T a passive-aggressive attack, but rather my attempt at getting the casual critic of vaccines to look beyond the typical criticisms that pseudoscience makes and think about how much work and data are required for studies of this nature. If I worded this badly (and if my logic is poor), I apologize.

      I am pro-research, pro-vaccine. I’m surprised that didn’t come across in this post.

    • Miles September 18, 2013 / 12:19 pm

      This comment is a mess. You say you can’t start pointing out all of the author’s logical fallacies, and then you proceed to not point out any in a relatively lengthy comment, except one accusation of an ad-hominem. With regards to the one accused logical fallacy you actually do make, it is clear the author is merely recommending to the reader (which would be a non-scientist interested in learning how to do a little basic analysis of a scientific paper) to consider who is doing the study and what potential bias might exist. In what universe is this not reasonable to mention in an article on some basic analysis a non-scientist can apply to a study? Yes, some people who are just eager to find what they are looking for will dismiss the work outright as soon as they identify such a bias (or perceived bias) but the author clearly recommends against this, and besides, such people wouldn’t be reading this article in the first place. Everything you’re reading into it is simply not there and actually reading the article makes this clear.

      It would be useful to get criticism from scientists on this article, and your comment starts off in such a way that suggests that such criticism is about to be presented, but then you proceed with a one-item list of complaints that has absolutely nothing to do with the actual subject of this article. It instead suggests either how sensitive and fragile your ego seems to be in the face of this completely innocuous (no pun intended) article (I’d hate to see how it holds up to the scrutiny of actual peer-review) and/or how amazingly bad your reading comprehension is (which must be an absolute joy to those on the receiving end of your review.) I injected (no pun intended) some actual ad-hominems there for you since you’re such a fan.

    • Avery Davis June 25, 2014 / 2:50 pm

      “Your implied, but fallacious, passive-aggressive ad hominem…”

      This was a faulty inference on your part.

  4. Anonymous September 23, 2013 / 1:36 am

    I found it very useful, especially for someone without patient ppl like me. Thanks a lot!

  5. Barbara November 17, 2013 / 1:26 pm

    The results of these studies would have a different outcome IF the researchers were aware that the vaccines included in this study were contaminated. Was THAT a consideration? If not, they need to do the studies over again….

    UPDATED: FDA ravages Sanofi vaccine sterility process in warning letter

    Recent FDA inspections in Canada, France list more than two dozen issues
    July 24, 2012

    “An FDA warning letter that lays out two dozen observations at a Sanofi Pasteur plant in Canada explains some of the mold and contamination problems that led the company to close some operations, a move that it says will lead to a shortage of BCG tuberculosis vaccine.
    Sanofi Pasteur, the largest operation devoted to human vaccines in the world, in June recalled four batches of the tuberculosis vaccine BCG and suspended production at a plant in Canada after Australian regulators found problems with sterility. The result will be a shortage there, as well as perhaps in Canada. There were also worries that the halt in production could lead to a global shortage of the bladder cancer drug ImmuCyst, but spokesman Len Lavenda said that “As of today, this notification does not impact our ability to supply the market, except for our BCG products manufactured in Toronto.”
    An FDA warning letter posted Tuesday on the agency’s website lays out a particularly long list of observations, 24 at the plant in Toronto inspected in April, as well as a couple more found during a routine March inspection of a plant in Marcy l’Etoile, France. Among other problems at the Toronto plant, the FDA says, “there have been no less than 58 documented non-conformances relating to the isolation of mold within the BCG aseptic processing areas” since August 2010.
    In addition, FDA investigators had issues with vial handling in the aseptic processing area in the filling and packaging building, the fact that employees were allowed unchallenged from the live vaccine area into the washing and sterilizing area, had questions about its disinfection practices, and observed on April 18 that there were “nesting birds” in the air handling units.
    In France, problems included not thoroughly investigating what caused some lots of Conjugated Haemophilus Vaccine and typhoid vaccines to test out of specifications.”

    UPDATED: FDA ravages Sanofi vaccine sterility process in warning letter – FiercePharma Manufacturing

    Another excellent source for educating yourself about how the drug manufacturers operate can be found in the book:

    ‘Our Daily Meds’ by Melody Peterson.

  6. Barbara November 17, 2013 / 1:40 pm

    One more point.

    “In addition, FDA investigators had issues with vial handling in the aseptic processing area in the filling and packaging building, the fact that employees were allowed unchallenged from the live vaccine area into the washing and sterilizing area, had questions about its disinfection practices, and observed on April 18 that there were “NESTING BIRDS” in the AIR HANDLING UNITS”

    The bird droppings were found in the AIR handling units of this plant.

    There are over 60 other diseases that birds and their droppings can carry. Many of them are airborne and can be transferred to humans just by being around droppings.

    Would the researchers have ended the study (or changed the results) if they had been aware of this?

    THAT would be interesting to know….

  7. Hugo January 5, 2014 / 7:13 am

    This is very important to me, I’m preparing for a graduate school, I should have known all of these early.

  8. Robin Broyles April 10, 2014 / 2:24 am

    Jennifer, this is a very informative piece. I clicked to this from your “You are being lied to” article.

    I did have the very real experience of seeing my son, perfectly healthy one minute, regress immediately after receiving vaccinations at age 2 1/2, and thousands of other parents saw the very same thing with their children. (Please see my comments under your article for more details on my experience and those of others and how this, as well as research I have done, has shaped my views).

    I do not wish to insult you or anyone else who shares your view, as I believe too many insults have been hurled from people on both sides of this issue. I believe your motives are good, as you sincerely believe the science proves that vaccinations are completely safe, and you want to save children from horrible diseases making a comeback. On the other side, parents who share my experience want to save as many children as possible from suffering, as our children have suffered. We believe too many vaccinations too early in life played a role in our children’s ill health, and we are concerned about what life holds for our children, who will take care of them after we die, etc. Yes, measles is frightening, but autism is far more frightening.

    I am not a scientist but am educated and logical and believe in looking at studies for myself instead of trusting what others say. I read the study by Dr. Wakefield, as well as the vax research on monkeys, and I agree that these were seriously flawed and questionable. I also agree that no one should take medical advice from an actress like Jenny McCarthy.

    On the other hand, Jenny McCarthy shares the experience that many parents, and I, have had. Of course this is anecdotal, but it’s illogical to discount us by saying we really didn’t see what we saw.

    I am sure that the studies you and others have cited as proof that there is no connection between vaccinations and autism are scientifically valid and peer-reviewed. Each of them, separately, may prove that one aspect of vaccinations, or one vaccination (MMR) is safe, but none of them examine what happens when children get multiple vaccinations, as happens in the real world.

    I am going to share a link below to a critique of those studies and would love to have your opinions on this (whether you feel the arguments are valid and would stand up to scientific scrutiny).

    Here is also an article about recent vaccine court cases rewarding children.

    Thank you! I will look forward to your response.

  9. bhallmar April 12, 2014 / 3:59 pm

    Thank you so much for writing this! Great proactive way to encourage people. And a great explanation of why just reading “the internet” is not sufficient to understand a subject.

  10. ptxinhua April 14, 2016 / 8:09 am

    Thx a lot, it really helps!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s