Keywords

Many-Facets-Rasch Model, incomplete rating designs, incomplete block design, design linkage, design connectivity, disconnected subsets, rater-mediated assessment, performance assessment

Description

This doctoral research project created a software prototype to generate, classify, and visualize all possible (a priori unique) incomplete rating designs for eight raters and 24 rating objects, with 25%, 50%, and 75% rater coverage, for sub-designs that repeat over four, six, and eight rating objects. Additionally it produced a representative rater schedule (incidence matrix), concurrence matrix, and network graph visualization for each design class. Lastly, the project report included recommendations for processes researchers can use when creating an incomplete rating design.

The main conclusions of this study are as follows:

1. Incomplete designs can vary greatly in their connectivity (linkage) and structure even when the number of raters, objects, and the number of raters-per-object is identical.

2. Visualization is extremely valuable for revealing details about the linkage or connectivity of an incomplete design. When the number of raters is not prohibitively large, a network graph of a proposed rating design should always be produced and evaluated before the rating data is collected. This enables the researcher to identify any disconnected subsets, as well as to more fully understand the structure and connectivity of their incomplete design.

3. More research is needed regarding the effect of an incomplete design (and its attributes) on the resulting measures from of a Many-Facets Rasch Model analysis, as compared to a complete, fully-crossed design.

4. Software is well suited to advance research in many of these issues, as well as to help the practitioner choose an optimal design for their specific purposes. More efforts should be made to develop and provide this sort of tool for those administrating or researching rater-mediated assessments.

Project Type

Research Project

Publication Date

2015-05-29

College

David O. McKay School of Education

Department

Instructional Psychology and Technology

Client

College/University

Master's Project or PhD Project

PhD Project

Creative Commons License

Creative Commons Attribution-Noncommercial 4.0 License
This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 4.0 License.

Share

COinS