Abstract
In the research regarding rating ESL writing assessments, borderline writing samples are mentioned, but a solution has yet to be addressed. Borderline samples are writing samples that do not perfectly fit a set level within the rubric, but instead have characteristics from multiple levels. The aim of this thesis is to provide an improved training module in the setting of an Intensive English Program by exposing new raters to borderline samples and rating rationale from experienced raters. The purpose of this training is to increase the confidence, consistency, and accuracy of novice raters when rating borderline samples of writing. The training consists of a workbook with a rubric and instructions for use, benchmark examples of writing, borderline examples of writing with comments from experienced raters defending the established scores, then a variety of writing samples for practice. The selection of the benchmark and the borderline examples of writing was informed by the fit statistic from existing datasets that had been analyzed with many-facet Rasch measurement. Eight experienced raters participated in providing rationale based upon the rubric explaining why each borderline sample was rated with its established score, and describing why the sample could be considered at a different level. In order to assess the effectiveness of the training workbook, it was piloted by 10 novice raters who rated a series of essays and responded to a survey. Results of the survey demonstrated that rater confidence increased following the training, but that they needed more time with the training materials to use them properly. The statistical analyses showed insignificant changes, which could be due to the limitations of the data collection. Further research regarding the effectiveness of this training workbook is necessary, as well as an increased discussion in the field regarding the prevalent issue of rating borderline samples of writing.
Degree
MA
College and Department
Humanities; Linguistics and English Language
Rights
http://lib.byu.edu/about/copyright/
BYU ScholarsArchive Citation
Greer, Brittney, "Assisting Novice Raters in Addressing the In-Between Scores When Rating Writing" (2013). Theses and Dissertations. 4066.
https://scholarsarchive.byu.edu/etd/4066
Date Submitted
2013-06-16
Document Type
Thesis
Handle
http://hdl.lib.byu.edu/1877/etd6293
Keywords
borderline, benchmark, writing assessment, rater error, rater training, language assessment, experienced raters, novice raters, English as a second language, intensive English program, L2
Language
English