Keywords
Many-Facets-Rasch Model, incomplete rating designs, incomplete block design, design linkage, design connectivity, disconnected subsets, rater-mediated assessment, performance assessment
Description
This doctoral research project created a software prototype to generate, classify, and visualize all possible (a priori unique) incomplete rating designs for eight raters and 24 rating objects, with 25%, 50%, and 75% rater coverage, for sub-designs that repeat over four, six, and eight rating objects. Additionally it produced a representative rater schedule (incidence matrix), concurrence matrix, and network graph visualization for each design class. Lastly, the project report included recommendations for processes researchers can use when creating an incomplete rating design.
The main conclusions of this study are as follows:
1. Incomplete designs can vary greatly in their connectivity (linkage) and structure even when the number of raters, objects, and the number of raters-per-object is identical.
2. Visualization is extremely valuable for revealing details about the linkage or connectivity of an incomplete design. When the number of raters is not prohibitively large, a network graph of a proposed rating design should always be produced and evaluated before the rating data is collected. This enables the researcher to identify any disconnected subsets, as well as to more fully understand the structure and connectivity of their incomplete design.
3. More research is needed regarding the effect of an incomplete design (and its attributes) on the resulting measures from of a Many-Facets Rasch Model analysis, as compared to a complete, fully-crossed design.
4. Software is well suited to advance research in many of these issues, as well as to help the practitioner choose an optimal design for their specific purposes. More efforts should be made to develop and provide this sort of tool for those administrating or researching rater-mediated assessments.
BYU ScholarsArchive Citation
McEwen, M. R. (2015). Development of a Software Prototype for Generating and Classifying Incomplete Many-Facet-Rasch Model Rating Designs. Unpublished doctoral project manuscript, Department of Instructional Psychology and Technology, Brigham Young University, Provo, Utah. Retrieved from https://scholarsarchive.byu.edu/ipt_projects/1
Project Type
Research Project
Publication Date
2015-05-29
College
David O. McKay School of Education
Department
Instructional Psychology and Technology
Client
College/University
Master's Project or PhD Project
PhD Project
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 4.0 International License.