Examining Automated Writing Evaluation Error Coverage in Relation to Uptake and Retention

Authors

• Mohaddeseh Mehrzad, Johnson County Community CollegeFollow
• Mohammad Rahimi, Cégep de l’OutaouaisFollow
• Stephanie Link, Iowa State UniversityFollow

Keywords

automated writing evaluation, recall, feedback uptake, revision acts, retention

Abstract

Despite the current widespread use of Automated Writing Evaluation (AWE) feedback, many issues regarding its efficacy still remain unresolved. Recent studies mainly focus on correctly detected errors with a lack of attention on the comprehensiveness of error detection, or error coverage. Error coverage is interesting because little is known about the capacity of AWE systems to fully detect common second language (L2) errors. It is also important to investigate the potential effect of such capacity on student uptake and retention, which are important constructs in fostering L2 writing development. To this end, the present study compared teacher feedback and AWE error coverage in L2 writing classes. The findings suggest that both the AWE system and the teacher demonstrated low error coverage across grammar, usage, and mechanics error categories. However, they indicated differences in the types of errors they identified most frequently. The AWE system flagged more mechanical errors, whereas the teacher provided twice as many corrections for grammar errors, including wrong/missing words, prepositions, and incorrect word forms. While the AWE system performed moderately in flagging articles and comma errors, it struggled with more nuanced grammatical errors, suggesting it may not be a reliable standalone tool for addressing specific needs of L2 learners’ writing challenges. Interestingly, coverage was positively associated with successful uptake, with students utilizing a wider variety of revision acts (i.e., change, add, delete, remove) on AWE errors identified compared to errors not identified. However, error coverage did not correlate with short- or long-term retention of accuracy, implying that retention may result from the interplay of error coverage with other factors. Findings provide implications for writing teachers regarding the employment of AWE systems and for AWE developers regarding the future optimizations of the AWE systems.

Recommended Citation

Mehrzad, Mohaddeseh; Rahimi, Mohammad; and Link, Stephanie (2026) "Examining Automated Writing Evaluation Error Coverage in Relation to Uptake and Retention," Journal of Response to Writing: Vol. 12: Iss. 1, Article 3.
Available at: https://scholarsarchive.byu.edu/journalrw/vol12/iss1/3