Violent metaphors

I want to expand upon my guide on how to read a scientific paper by working through an example. You may not have the time or ability to read a paper in as much depth as I outline below. But my goal with this series of posts is to help people feel less intimidated by research papers by giving them a framework for reading them. I want to make these tools available, should you wish to use them.

I decided to choose as my first example a paper that would also be useful for people grappling with the question of vaccine safety, so that they can become familiar with how these studies are conducted and better understand some of the terminology.

The paper I’ll go through is:

Toback et al., 2013.
“A postlicensure evaluation of the safety of Ann Arbor strain live attenuated
influenza vaccine in children 24–59 months of age” Vaccine 31: 1812– 1818

It’s freely accessible here.

Before you go any further, you’ll find it most useful to try to read through the paper on your own first and take notes. If this is your first time going through a research paper, you may find it difficult to judge the quality of the results and whether the authors are interpreting them correctly. My guide should help if you go through it step-by-step, but if it’s overwhelming, start small. At minimum, try to identify the Big Question, the Specific Question(s), and the Approach from the Introduction section. While the methods might be murky for you, read them over anyway and try to figure out answers to these questions:

–How many subjects participated in this research? What characteristics did they have that made the authors choose them for this study?

—What exactly did the authors measure, and how?

—What were the controls for this study?

Be sure to look up terminology you don’t understand. There’s a list of helpful abbreviations on the first page.

Next, try to identify and write down the main results of each experiment. Remember that some are likely to be in the tables, so make sure you look at them.

Finally think about what the results mean before you read the authors’ discussion of them.

Done? Okay, here’s what I did with the paper, following my own guide. See if you agree or disagree with me.

1. I took note of the authors and their institutional affiliations.

Despite what the Discovery Institute says, I don’t think you should necessarily discount papers solely on the institutional affiliations of the authors. However, you should be alert for potential bias if you see that they do come from institutions with a history of advocating for a specific position, or with a vested interest in a particular outcome for the experiments.

The first author and the last authors are (at least in biomedical fields) generally the most important. The first author tends to be the person who did most of the work, and the last author tends to be the person who was the overall supervisor (or Principal Investigator) of the research. In this paper, the first author worked for the company MedImmune at the time of this study (and now works for Gilead Sciences). The last author (and many of the other authors) works for the Kaiser Permanente Vaccine Study Center. The research was funded by MedImmune. The authors of this paper also provide disclosures as to their financial interests in the research. You can find that information at the very end of the paper, under the Acknowledgments. Think about how these affiliations might affect the research

2. I moved next to the Introduction (skipping the abstract!), to find the Big Question.

Remember that the Big Question is very broad, bigger than the scope of a single paper. In this case it could be: Are live-attenuated vaccines safe for children?

3. Next, I looked up some terms, and summarized the background (albeit a bit oversimplified) in a few sentences:

Important definitions for this paper:

—Post-licensure vaccine safety studies
—Cohort study
—Case-control study
—The CDC’s fact sheet on nasal vaccine
—Live attenuated vaccine

Background summary:

a. The Ann Arbor strain live attenuated influenza vaccine (LAIV) has been approved for use in healthy children between 24–59.

b. One previous study showed an increase in wheezing in children of this age who received this vaccine.

c. It’s necessary to test whether children who received this vaccine had any adverse reactions.

4. From the abstract, I also determined the Specific Question for this research.

Remember that the Specific Question is what the paper is actually testing.

What are all adverse events reported for 24-59 month old children who received the LAIV?

5. From the Introduction and the Study Design, I determined the approach:

Compare rates of medically attended events (MAE), serious adverse events (SAE), perspecified diagnoses of interest (PSDI), and “rare events potentially related to wild-type influenza (WTI)” between children who received the vaccine and controls

6. Next, I went to the methods section to outline what they did.
(Given that this is a blog intended for a general audience, I’m going to simplify the summary of their methods somewhat.)

A. Identified a pool of 25,000 children between 24-59 months who were immunized with LAIV (= “analysis cohort”).

B. Removed all children with high-risk medical conditions from the cohort. This included children with asthma. This is important to note, because “wheezing” was an adverse effect reported to be higher among LAIV recipients in a previous study. (How do you think that might affect the results of this current study?)

C. Created three control groups for comparison:

1. Within-cohort (this was a division of vaccinated children into different risk intervals to compare against each other: 0-3 days post-vaccination was compared to 4-42 days post-vaccination; 0-21 days post-vaccination was compared to 22-42 days post-vaccination).

2. Unvaccinated children

3. Children vaccinated with a different type of influenza vaccine (trivalent inactivated influenza vaccine, or TIV).Here is the difference between the two types.

D. The control groups were matched with the test group by age (in months) and geographic distribution. Matching variables is important because it removes certain things that could complicate determining whether an association is real or not. Here’s one explanation of it. (If someone has a better one, please share it in the comments!)

E. The authors then identified medically attended adverse events (MAEs) for all cases and controls from medical records. Based on previous work (cited), they defined some MAEs which they hypothesized were possibly caused by vaccination (including respiratory and gastrointestinal tract issues, asthma, wheezing, and bacterial infections). They gave patients with these issues special scrutiny.

F. Rates of adverse events were statistically compared between cohorts.

7. Next, I went to the Results section to summarize the findings of the authors.

There are many results, and the comparisons are pretty intricate. I encourage you to study Figures 1-2 (page 1814-5), to get an overall picture of statistically significant differences in events between the cohorts. As with the methods, I simplified this a bit for a general audience below. (Please let me know in the comments if I left out anything you think is important)

This study included:

-28,226 children between the ages of 24-59 months who were vaccinated with LAIV

-27,937 children vaccinated with TIV (the alternative vaccine)

-25,981 unvaccinated children.

Do you think this sample size is large enough?

The authors divided the results into three categories:

A. Serious Adverse Events

The authors looked at the medical records of all 25,000+ children in each category and compared rates of ALL serious medical issues between them during the study period following vaccination. The rates of serious adverse events (SAEs) through 42 days post-vaccination were not significantly different between the LAIV children and any of the control groups.

The SAEs that did occur multiple times in the group of LAIV-vaccinated children were:

-4 cases of trauma
-4 cases of pneumonia
-3 cases of dehydration
-2 cases of congenital anomaly
-2 cases of urinary tract infection.

The authors thought that these particular events were not likely related to the vaccine. Do you agree or disagree?

Two cases occurred which the authors did hypothesize were linked to the immunization: a boy developed a case of respiratory syncytial virus infection, and another case of a boy with intussusception and viral infection. Both recovered completely with medical attention.

B. Hospitalization

The authors compared rates of hospitalization among all children in the study, and broke them down by diagnosis. A significant increase in events diagnosed in hospitalized children of the LAIV recipient group (vs. unvaccinated) included sleep apnea (5 cases), tonsillitis (5 cases), and any event (n=21). The authors report that “sleep apnea and tonsillitis occurred in 5 children who were vaccinated on the same day as their preoperative examination for an elective tonsillectomy and adenoidectomy procedure and were hospitalized after surgery with those diagnoses” (pp 1814-1815).

A significant decrease in events diagnosed in hospitalized children of the LAIV recipient group in some conditions:

The overall rate of hospitalization within 180 days postvaccination with LAIV was significantly lower compared to:

Unvaccinated controls in 24-35 month cohort
TIV recipients of all ages
24-35 month cohort
Asthma/RAD events within 21, 42 days postvaccination with LAIV were decreased compared to TIV-vaccinated controls and “any event in comparison to TIV-vaccinated and within-cohort.” (p1815).

C. All medically attended events (MAE)

One MAE rate was significantly higher among LAIV recipients when compared to all 3 control groups:

Urticaria (hives): http://emedicine.medscape.com/article/137362-overview

One MAE rate was significantly decreased among LAIV recipients across all comparison groups:

asthma/RAD (within 21 days postvaccination in all subjects, and 36-59 months old)

Children who received the LAIV vaccine had significantly higher frequencies of influenza infection than controls. Nearly all (99%) of these cases were the H1N1 strain (which LAIV wasn’t designed to protect against). H1N1 vaccination frequency was the lowest in the LAIV group.

The authors also adjusted for the fact that they were performing multiple comparisons (I won’t get into the statistical rational here, but please feel free to discuss the details in the comments if you like!). When they did this, 2 wheezing/SOB rates were increased among the LAIV group vs unvaccinated controls: in 36-59 months and 24-59 months. 42 AW events were lower in LAIV compared with TIV-vaccinated and unvaccinated controls (this included “respiratory tract events, AW events, any events, asthma/RAD, cough, otitis media, upper respiratory tract infection, viral syndrom, and wheezing/SOB” p1815).

Have you made it through all of that? Good!

8. Now it’s time to think hard about what all of this means. From this point forward, I’m not going to give you my answers—I want you to come up with your own (Yes, I know that’s infuriating, but I am a teacher, after all).

Go back to the Specific Question. Did the results answer it? Can you say that the vaccine did or did not adversely affected children’s health, according to this study? Do you think that the number of children included in this study was large enough?

9. Now read the Discussion and Conclusion sections.

What did the authors say the results mean? (Do you agree? Opinions might differ here, and I encourage you to discuss them in the comments)

For example, the authors provide a hypothesis why the LAIV children had higher rates of H1N1 infection. What is it? Does it seem plausible to you? How would you go about testing that hypothesis in a follow-up study?

Did the authors identify any weaknesses in their study design? Can you see any?

What do they/you think should be done next?

Would you feel confident giving your own child this vaccine, based on this safety study? Why or why not?

Do you see any evidence of bias in the authors’ research?

10. Now you can read the abstract.

Did it fairly represent the paper?

11. I encourage you (if you’re really interested in the topic of childhood influenza vaccines) to now go read what other people have to say about this (and other) papers.

This is the part where you “do your own research”. But save it for the final step, so that you’re better informed than when you started.

Final note:
If you’re just a casual reader, I understand that you might not be able to do every single step here. However, the more important this subject is to you, the more in-depth you should try to go. You now have a framework for how to go about doing this, should you need it.

Please understand that the experts in the field are reading these papers in even more detail than I’ve given you. They may be repeating these studies, or following up on them to test new ideas that they’ve gleaned from the results. Now that you have a glimpse into the process, does that alter your perception of scientific research? Nothing irritates me more than when someone writes a forum post, or creates a meme for Facebook trying to undermine a scientific study without actually reading beyond the title or abstract. Even if you don’t care to do it yourself, I hope that this series on my blog will help you see the need to look beyond surface critiques, in order to identify the real strengths/weaknesses of research.

Many thanks to everyone who have commented and shared my posts! I really appreciate your participation in these discussions, and hope that we can continue them. Feel free to share your thoughts/questions/concerns in the comments section.

——————————————————————————————————————————————

Edited to add: Here are two more excellent resources for reading and understanding scientific papers. If you still don’t feel like you understand the process after reading this post, I encourage you to go read these powerpoints from the American Society of Plant Biology:

Click to access How%20to%20Read%20a%20Scientific%20Paper%20M%20Williams%20Mar%202013.pdf

Click to access CaseStudy%20for%20How%20to%20Read%20a%20Sci%20Paper%20M%20Williams%20Mar%202013.pdf