In Validations We Trust? The Impact of Imperfect Human Annotations as a Gold Standard on the Quality of Validation of Automated Content Analysis

Author(s)
Hyunjin Song, Petro Tolochko, Jakob-Moritz Eberl, Olga Eisele, Esther Greussing, Tobias Heidenreich, Fabienne Lind, Sebastian Galyga, Hajo Boomgaarden
Abstract

Political communication has become one of the central arenas of innovation in the application of automated analysis approaches to ever-growing quantities of digitized texts. However, although researchers routinely and conveniently resort to certain forms of human coding to validate the results derived from automated procedures, in practice the actual “quality assurance” of such a “gold standard” often goes unchecked. Contemporary practices of validation via manual annotations are far from being acknowledged as best practices in the literature, and the reporting and interpretation of validation procedures differ greatly. We systematically assess the connection between the quality of human judgment in manual annotations and the relative performance evaluations of automated procedures against true standards by relying on large-scale Monte Carlo simulations. The results from the simulations confirm that there is a substantially greater risk of a researcher reaching an incorrect conclusion regarding the performance of automated procedures when the quality of manual annotations used for validation is not properly ensured. Our contribution should therefore be regarded as a call for the systematic application of high-quality manual validation materials in any political communication study, drawing on automated text analysis procedures.

Organisation(s)
Department of Communication, Department of Political Science
External organisation(s)
Technische Universität Carolo-Wilhelmina zu Braunschweig
Journal
Political Communication
Volume
37
Pages
550-572
No. of pages
23
ISSN
1058-4609
DOI
https://doi.org/10.1080/10584609.2020.1723752
Publication date
2020
Peer reviewed
Yes
Austrian Fields of Science 2012
508007 Communication science
Keywords
ASJC Scopus subject areas
Communication, Sociology and Political Science
Portal url
https://ucris.univie.ac.at/portal/en/publications/in-validations-we-trust-the-impact-of-imperfect-human-annotations-as-a-gold-standard-on-the-quality-of-validation-of-automated-content-analysis(f160e180-3cdf-4cab-adce-fe68e151ea16).html