Keywords

Annotated Corpora, Annotation, User Study

Abstract

Manual annotation of large textual corpora can be cost-prohibitive, especially for rare and under-resourced languages. One potential solution is pre-annotation: asking human annotators to correct sentences that have already been annotated, usually by a machine. Another potential solution is correction propagation: using annotator corrections to dynamically improve to the remaining pre-annotations within the current sentence. The research presented in this paper employs a controlled user study to discover under what conditions these two machine-assisted annotation techniques are effective in increasing annotator speed and accuracy and thereby reducing the cost for the task of morphologically annotating texts written in classical Syriac. A preliminary analysis of the data indicates that pre-annotations improve annotator accuracy when they are at least 60% accurate, and annotator speed when they are at least 80% accurate. This research constitutes the first systematic evaluation of pre-annotation and correction propagation together in a controlled user study.

Original Publication Citation

Paul Felt, Eric Ringger, Kevin Seppi, Kristian Heal, Robbie Haertel, Deryle Lonsdale (2012). First Results in a Study Evaluating Pre-labeling and Correction Propagation for Machine-Assisted Syriac Morphological Analysis. In (Nicoletta Calzolari. Khalid Choukri, ThierryDeclerck, Mehmet Uğur Doğan, Bente Maegaard, Joseph Mariani, Jan Odijk, and SteliosPiperidis, Eds.) Proceedings of the Eighth International Conference on Language Resources andEvaluation (LREC '12), European Language Resources Association (ELRA); pp. 878-885, ISBN 978-2-9517408-7-7.

Document Type

Conference Paper

Publication Date

2012

Publisher

European Language Resources Association

Language

English

College

Humanities

Department

Linguistics

University Standing at Time of Publication

Associate Professor

Download

Included in

Linguistics Commons

Share

COinS