Analysis Framework

Vaccine News Quality

Last Updated: June 30, 2022 (Version 1)

References ↓

This framework was developed as part of the News Quality (NewsQ) Initiative to assess the quality of a given news article -- vaccine reporting in particular. Over the course of two years, NewsQ worked with journalists and health communicators to better understand what elements make for a better news article generally, for science/health journalism, and for vaccine-related news.

The framework is grounded in a questionnaire: recognizing that specialized topics benefit from the specificity of answerable questions, we developed a questionnaire that evaluates news quality through a layered approach.

We’ve arranged these markers into three categories of quality:

This layered framework for evaluating news quality is visualized as three concentric circles; the circles represent how each dimension of quality evaluation builds upon the other.

This figure is a visualization of the Vaccine News Quality Framework. The three circles on the left represent the three quality dimensions of general journalism, science/health journalism, and vaccine journalism. The boxes on the right represent the indicators associated with each dimension. Finally, the arrow on the far right shows how as the dimensions become more specialized, so do their associated indicators.

Then, each dimension is associated with a set of indicators, which are determined through questions. These questions capture characteristics commonly found in high-quality journalism, and encourage the reader to consider the accuracy, bias, genre, language, and transparency of the content and the content’s source. The questions provide general guidelines for reliability and credibility, and highlight key points for a reader to keep in mind when assessing the general quality of a news article.

Overall, this evaluation should serve as a broad guide to helping writers, editors, and readers feel confident in their vaccine news quality assessment skills. There is also a possibility this questionnaire can be used as a guide for new research in order to build news quality datasets through annotation and labeling, and to crowd-source automated techniques that can detect key quality indicators. Additional uses will be determined through further testing and future work in this area.

Indicators and questions can be seen in the expandable sections below. The formal questionnaire can be accessed either in an online version or as a downloadable PDF.

General Journalism Quality
This foundational category of news quality is largely derived from traditional journalistic standards, and can be applied to news of any topic or genre. This first dimension represents an integrated approach to quality, emphasizing how the areas of accuracy, transparency, impartiality, reliability of sourcing, and language work together to create high-standard public information.
Science/Health Journalism Quality
Science and health reporting, which deals with scientific issues and public issues involving science, can be understood broadly as “how society talks about science.” In this type of specialized journalism, it is critical to communicate often-complex scientific information in an accurate yet accessible way. Accordingly, this second dimension of quality includes questions regarding subject matter expertise, the inclusion of context, and the handling of uncertainty.
Vaccine Journalism Quality
Vaccine journalism, as a highly unique type of science communication, requires additional considerations. This third dimension of quality considers the use of specialized sources, the avoidance of equal presentation of scientifically unequal claims (“false balance”), and the representation of (un)certainty about the strength of scientific evidence for or against a particular risk. These indicators are new contributions to news quality standards. They were derived from conversations with science and public health journalists and members of the World Health Organization’s Vaccine Safety Net (VSN), and validated through review workshops. We plan to continue to validate these indicators through additional testing and research.


The development of this framework for evaluating vaccine journalism quality was informed through academic research, consultation with journalists and technologists, and input from collaborative workshops. The structure was first derived from earlier work on classification around news taxonomies related to scientific reporting, which emerged from panels hosted by the News Quality Initiative in 2020 that built upon earlier work in the field.

Drawing on these findings, the ARTT team determined an initial set of indicators and drafted a corresponding questionnaire. Experts in a variety of related fields were then consulted for review and feedback, either directly or in semi-structured workshops. This group included science and public health journalists, members of the World Health Organization’s Vaccine Safety Net (VSN), experienced editors within the Wikipedia community, and a stakeholder from a technology platform. Related questions were also reviewed by participating Wikipedians in two workshops in early 2022. The questionnaire was also presented in a workshop at the International Journalism Festival (IJF) in Perugia, Italy in April 2022. Throughout this iterative process, questionnaire categories and indicators were continually revised and validated, resulting in the first version of this framework.

We would like to thank our partners at MuckRock Foundation, the Vaccine Safety Net (VSN) of the World Health Organization, and Wikimedia DC for their support and input. We would also like to thank the individual participants of our workshops and consultations, as well as the attendees at IJF Perugia.

Note: We are open to feedback regarding these indicators and questions. Please contact us at artt [dot] hackshackers [dot] com with the subject line “Vaccine News Questionnaire.”

View All References
[1] Anderson, Peter James. 2013. “Application of a News Quality Monitoring Methodology.”

[2] Bhuiyan, Md Momen, Hayden Whitley, Michael Horning, Sang Won Lee, and Tanushree Mitra. 2021. “Designing Transparency Cues in Online News Platforms to Promote Trust: Journalists’ & Consumers’ Perspectives.” Proceedings of the ACM on Human-Computer Interaction 5 (CSCW2): 1–31.

[3] Bogart, Leo. 2004. “Reflections on Content Quality in Newspapers.” Newspaper Research Journal 25 (1): 40–53.

[4] Chen, Caroline, Randi Hutter Epstein, Jennifer Kahn, Katherina Thomas, Rick Weiss, Wudan Yan, and Yemile Bucay. 2021. “Towards Healthier Science and Health News Feeds: NewsQ Panel on Science and Health Journalism.” NewsQ (blog). January 26, 2021.

[5] Fleerackers, Alice, Michelle Riedlinger, Laura Moorhead, Rukhsana Ahmed, and Juan Pablo Alperin. 2021. “Communicating Scientific Uncertainty in an Age of COVID-19: An Investigation into the Use of Preprints by Digital Media Outlets.” Health Communication 0 (0): 1–13.

[6] Gladney, George Albert. 1996. “How Editors and Readers Rank and Rate the Importance of Eighteen Traditional Standards of Newspaper Excellence.” Journalism & Mass Communication Quarterly 73 (2): 319–31.

[7] Harloe, Kate, Nancy Ancrum, Karen Attiah, Maria Bustillos, Sewell Chan, Elena Gooray, and Arwa Mahdawi. 2021. “Separating Quality News from Quality Opinion,” NewsQ.

[8] “JTI | Journalism Trust Initiative.” JTI Website. Accessed June 2, 2022.

[9] Kahn, Gabriel, Meredith Clark, Al Cross, Claudia Irizarry Aponte, Mandy Jenkins, and David Kroman. 2020. “The Local News Landscape Is Broken: NewsQ Panel Review of Platform News Products.” NewsQ.

[10] Liotsiou, Dimitra, Bence Kollanyi, and Philip N. Howard. 2019. “The Junk News Aggregator: Examining Junk News Posted on Facebook, Starting with the 2018 US Midterm Elections.” ArXiv:1901.07920 [Cs], April.

[11] Ordway, Denise-Marie. 2020. “Covering COVID-19 and the Coronavirus: Harvard Professor Offers 5 Tips.” The Journalist’s Resource (blog). March 6, 2020.

[12] Secko, David M., Elyse Amend, and Terrine Friday. 2013. “Four Models of Science Journalism.” Journalism Practice 7 (1): 62–80.

[13] Shapiro, Ivor, Patrizia Albanese, and Leigh Doyle. 2006 “What Makes Journalism ‘Excellent’? Criteria Identified by Judges in Two Leading Awards Programs.” Canadian Journal of Communication 31: 21.

[14] Shapiro, Ivor. 2010. “Evaluating Journalism.” Journalism Practice 4 (2): 143–62.

[15] Vincent, Subramaniam, Patricia Lopez, David Agraz, Leona Allen Ford, Jon Allsop, Rochelle Riley, and Rebecca Traister. 2020. “Our Opinion: Recommendations for Publishing Opinion Journalism on Digital Platforms.” NewsQ.

[16] Vir, Jason. 2021. “The Relevance of Impartial News in a Polarised World.” Reuters Institute for the Study of Journalism, University of Oxford.

[17] Zhang, Amy X., Aditya Ranganathan, Sarah Emlen Metz, Scott Appling, Connie Moon Sehat, Norman Gilmore, Nick B. Adams et al. 2018. “A structured response to misinformation: Defining and annotating credibility indicators in news articles.” In Companion Proceedings of the The Web Conference 2018, pp. 603-612. https://doi/abs/10.1145/3184558.3188731.