In validation we trust? Imperfect Gold Standards and the Validation of Automated Content Analysis

Autor(en)

Hyunjin Song, Petro Tolochko, Jakob-Moritz Eberl, Tobias Heidenreich, Fabienne Lind, Olga Eisele, Esther Greussing, Hajo Boomgaarden

Abstrakt

Political communication has become one of the central arenas of innovation in the application of automated analysis approaches to ever-growing quantities of digitized texts. However, although researchers routinely and conveniently resort to certain forms of human coding to validate the results derived from automated procedures, in practice the actual “quality assurance” of such a “gold standard” often goes unchecked. Contemporary practices of validation via manual annotations are far from being acknowledged as best practices in the literature, and the reporting and interpretation of validation procedures differ greatly. We systematically assess the connection between the quality of human judgment in manual annotations and the relative performance evaluations of automated procedures against true standards by relying on large-scale Monte Carlo simulations. The results from the simulations confirm that there is a substantially greater risk of a researcher reaching an incorrect conclusion regarding the performance of automated procedures when the quality of manual annotations used for validation is not properly ensured. Our contribution should therefore be regarded as a call for the systematic application of high-quality manual validation materials in any political communication study, drawing on automated text analysis procedures.

Organisation(en)

Institut für Publizistik- und Kommunikationswissenschaft, Institut für Politikwissenschaft

Seiten

550-572

Anzahl der Seiten

23

Publikationsdatum

2019

Peer-reviewed

Ja

ÖFOS 2012

508007 Kommunikationswissenschaft

Schlagwörter

ASJC Scopus Sachgebiete

Communication, Sociology and Political Science

Link zum Portal

https://ucrisportal.univie.ac.at/de/publications/7b8e6e1a-6942-4dea-ab6d-b2ca003a38b8