Three Dimensions of Reproducibility in Natural Language Processing

Kevin Cohen (1, 2) , Jingbo Xia (3) , Pierre Zweigenbaum (1) , Tiffany Callahan , O. Hargraves , Foster Goss , Nancy Ide , Aurélie Névéol (1) , Cyril Grouin (1) , Larry E Hunter (2)

1 ILES - Information, Langue Ecrite et Signée
2 Computational Bioscience Program. University of Colorado Denver
3 HZAU - Huazhong Agricultural University [Wuhan]

Despite considerable recent attention to problems with reproducibility of scientific research, there is a striking lack of agreement about even the definition of the term. That is a problem, because the lack of a consensus definition makes it difficult to compare studies of reproducibility, and thus to have even a broad overview of the state of the issue in natural language processing. This paper proposes an ontology of reproducibility in that field. We show that three dimensions of reproducibility, corresponding to three kinds of claims in natural language processing papers, can account for a variety of types of research reports. These dimensions are reproducibility of a conclusion, of a finding, and of a value. Three biomedical natural language processing papers by the authors of this paper are analyzed with respect to these dimensions.

Mots clés: replicatability; reproducibility; replicability; methodology; repeatability

Domaines: Informatique [cs]; Informatique et langage [cs.CL]

Kevin Cohen, Jingbo Xia, Pierre Zweigenbaum, Tiffany Callahan, O. Hargraves, et al.. Three Dimensions of Reproducibility in Natural Language Processing. LREC 2018, Eleventh International Conference on Language Resources and Evaluation, May 2018, Miyazaki, Japan. pp.156-165. ⟨hal-01842490⟩