Analysis Framework

Vaccine Quality

How high-quality
is this information?

Last updated: June 30, 2022 (Version 1)

This framework was developed to assess the quality of a given news article — vaccine reporting in particular. The framework is grounded in a questionnaire: recognizing that specialized topics benefit from the specificity of answerable questions, we developed a questionnaire that evaluates news quality through a layered approach.

This layered framework for evaluating news quality is visualized as three concentric circles; the circles represent how each dimension of quality evaluation builds upon the other.

Dimension 1: General Journalism Quality: This foundational dimension assesses the general journalism quality when the piece aims to be journalism.
Dimension 2: Science/Health Journalism Quality: The second dimension assesses the science/health journalism quality when the piece mentions health and/or well-being, and/or makes mention of science or scientific methodology,
Dimension 3: Vaccine Journalism Quality: The third dimension assesses the quality of vaccine journalism when the piece is about or discusses vaccines.

Then, each dimension is associated with a set of indicators, which are determined through questions. These questions capture characteristics commonly found in high-quality journalism, and encourages the reader to consider the accuracy, bias, genre, language, and transparency of the content and the content’s source. The questions provide general guidelines for reliability and credibility, and highlights key points for a reader to keep in mind when assessing the general quality of a news article.

Overall, this evaluation should serve as a broad guide to helping writers, editors and readers feel confident in their vaccine news quality assessment skills. There is also a possibility this questionnaire can be used as a guide for new research in order to build news quality datasets through annotation and labeling, and to crowd-source automated techniques that can detect key quality indicators. Additional uses will be determined through further testing and future work in this area.

Indicators and questions can be seen in the expandable sections below. The formal questionnaire can be accessed either in an online version or as a downloadable PDF.

Vaccine Quality Analysis Framework

General Journalism Quality

This foundational category of news quality is largely derived from traditional journalistic standards, and can be applied to news of any topic or genre. This first dimension represents an integrated approach to quality, emphasizing how the areas of accuracy, transparency, impartiality, reliability of sourcing, and language work together to create high-standard public information.

Science/Health Journalism Quality

Science and health reporting, which deals with scientific issues and public issues involving science, can be understood broadly as “how society talks about science.” In this type of specialized journalism, it is critical to communicate often-complex scientific information in an accurate yet accessible way. Accordingly, this second dimension of quality includes questions regarding subject matter expertise, the inclusion of context, and the handling of uncertainty.

Vaccine Journalism Quality

Vaccine journalism, as a highly unique type of science communication, requires additional considerations.This third dimension of quality considers the use of specialized sources, the avoidance of equal presentation of scientifically unequal claims (“false balance”), and the representation of (un)certainty about the strength of scientific evidence for or against a particular risk.These indicators are new contributions to news quality standards. They were derived from conversations with science and public health journalists and members of the World Health Organization’s Vaccine Safety Net (VSN), and validated through review workshops. We plan to continue to validate these indicators through additional testing and research.

Methodology

The development of this framework for evaluating vaccine journalism quality was informed through academic research, consultation with journalists and technologists, and input from collaborative workshops. The structure was first derived from earlier work on classification around news taxonomies related to scientific reporting, which emerged from panels hosted by the News Quality Initiative in 2020 that built upon earlier work in the field.

Drawing on these findings, the ARTT team determined an initial set of indicators and drafted a corresponding questionnaire. Experts in a variety of related fields were then consulted for review and feedback, either directly or in semi-structured workshops. This group included science and public health journalists, members of the World Health Organization’s Vaccine Safety Net (VSN), experienced editors within the Wikipedia community, and a stakeholder from a technology platform. Related questions were also reviewed by participating Wikipedians in two workshops in early 2022. The questionnaire was also presented in a workshop at the International Journalism Festival (IJF) in Perugia, Italy in April 2022. Throughout this iterative process, questionnaire categories and indicators were continually revised and validated, resulting in the first version of this framework.

We would like to thank our partners at MuckRock Foundation, the Vaccine Safety Net (VSN) of the World Health Organization, and Wikimedia DC for their support and input. We would also like to thank the individual participants of our workshops and consultations, as well as the attendees at IJF Perugia.
‍
‍
‍

Note: We are open to feedback regarding these indicators and questions. Please contact us at artt [dot] hackshackers [dot] com with the subject line “Vaccine News Questionnaire.”

References

[1] Anderson, Peter James. 2013. “Application of a News Quality Monitoring Methodology.” https://clok.uclan.ac.uk/7825/.

[2] Bhuiyan, Md Momen, Hayden Whitley, Michael Horning, Sang Won Lee, and Tanushree Mitra. 2021. “Designing Transparency Cues in Online News Platforms to Promote Trust: Journalists’ & Consumers’ Perspectives.” Proceedings of the ACM on Human-Computer Interaction 5 (CSCW2): 1–31. https://doi.org/10.1145/3479539.

[3] Bogart, Leo. 2004. “Reflections on Content Quality in Newspapers.” Newspaper Research Journal 25 (1): 40–53. https://doi.org/10.1177/073953290402500104.

[4] Chen, Caroline, Randi Hutter Epstein, Jennifer Kahn, Katherina Thomas, Rick Weiss, Wudan Yan, and Yemile Bucay. 2021. “Towards Healthier Science and Health News Feeds: NewsQ Panel on Science and Health Journalism.” NewsQ (blog). January 26, 2021.https://newsq.net/newsq-review-panel-reports-2020/towards-healthier-science-and-health-news-feeds-newsq-panel-on-science-and-health-journalism/

[5] Fleerackers, Alice, Michelle Riedlinger, Laura Moorhead, Rukhsana Ahmed, and Juan Pablo Alperin. 2021. “Communicating Scientific Uncertainty in an Age of COVID-19: An Investigation into the Use of Preprints by Digital Media Outlets.” Health Communication 0 (0): 1–13. https://doi.org/10.1080/10410236.2020.1864892.

[6] Gladney, George Albert. 1996. “How Editors and Readers Rank and Rate the Importance of Eighteen Traditional Standards of Newspaper Excellence.” Journalism & Mass Communication Quarterly 73 (2): 319–31. https://doi.org/10.1177/107769909607300204.

[7] Harloe, Kate, Nancy Ancrum, Karen Attiah, Maria Bustillos, Sewell Chan, Elena Gooray, and Arwa Mahdawi. 2021. “Separating Quality News from Quality Opinion,” NewsQ. https://newsq.net/wp-content/uploads/2021/11/Separating-Quality-News-from-Quality-Opinion.pdf.

[8] “JTI | Journalism Trust Initiative.” JTI Website. Accessed June 2, 2022. https://www.journalismtrustinitiative.org/.

[9] Kahn, Gabriel, Meredith Clark, Al Cross, Claudia Irizarry Aponte, Mandy Jenkins, and David Kroman. 2020. “The Local News Landscape Is Broken: NewsQ Panel Review of Platform News Products.” NewsQ. https://newsq.net/wp-content/uploads/2020/11/NewsQ-Local-Panel-2020-nov30FINAL.pdf.

[10] Liotsiou, Dimitra, Bence Kollanyi, and Philip N. Howard. 2019. “The Junk News Aggregator: Examining Junk News Posted on Facebook, Starting with the 2018 US Midterm Elections.” ArXiv:1901.07920 [Cs], April. http://arxiv.org/abs/1901.07920.

[11] Ordway, Denise-Marie. 2020. “Covering COVID-19 and the Coronavirus: Harvard Professor Offers 5 Tips.” The Journalist’s Resource (blog). March 6, 2020. https://journalistsresource.org/economics/covid-19-coronavirus-epidemiology/.

[12] Secko, David M., Elyse Amend, and Terrine Friday. 2013. “Four Models of Science Journalism.” Journalism Practice 7 (1): 62–80. https://doi.org/10.1080/17512786.2012.691351.

[13] Shapiro, Ivor, Patrizia Albanese, and Leigh Doyle. 2006 “What Makes Journalism ‘Excellent’? Criteria Identified by Judges in Two Leading Awards Programs.” Canadian Journal of Communication 31: 21.

[14] Shapiro, Ivor. 2010. “Evaluating Journalism.” Journalism Practice 4 (2): 143–62. https://doi.org/10.1080/17512780903306571.

[15] Vincent, Subramaniam, Patricia Lopez, David Agraz, Leona Allen Ford, Jon Allsop, Rochelle Riley, and Rebecca Traister. 2020. “Our Opinion: Recommendations for Publishing Opinion Journalism on Digital Platforms.” NewsQ. https://newsq.net/wp-content/uploads/2020/11/NewsQ-Opinion-Panel-2020-nov30-FINAL.pdf.

[16] Vir, Jason. 2021. “The Relevance of Impartial News in a Polarised World.” Reuters Institute for the Study of Journalism, University of Oxford. https://reutersinstitute.politics.ox.ac.uk/sites/default/files/2021-10/Vir_the_relevance_of_impartial_news_in_a_polarised_world_FINAL_0.pdf.