Abstract
To use a Cronbach's alpha coefficient or IRT methods on measuring instruments, both unidimensionality and local item dependence (LID) conditions need to be met. Moreover, LID can potentially introduce an unintended additional dimension into the test at the expense of the constructs of interest. Situational judgment tests use testlet structures, and thus likely present degrees of LID. We explored the latent structure and degrees of local item dependence in a situational judgment test (SJT) used by a Western United States medical school (WMS) for admission purposes. I used a fully deidentified data set from a group of 560 examinees who took the WMS SJT in 2020-2021 for admission purposes and who gave consent for use of their responses in future studies. Our inquiry included statistical analysis using exploratory structural equation modeling (ESEM) parameterization, multitrait multimethod modeling and computation of modification indices, as well as content analysis of items. When using the directional distance scoring method, the data set provided by the WMS yields a 4-dimensional substantive structure. I named these factors (a) authority, (b) avoidance, (c) integrity, and (d) communication. Each of these names refer to the coping mechanism chosen to respond to the issue related in the scenarios. These four factors are somewhat different from the four initial themes the WMS used to write their SJT items. The analysis results show that there is little method or testlet interference with the instrument and that degrees of LID within the WMS SJT are low. I posit that both the rating task given to examinees, as well as the interpretability of each item outside of the scenario are functions of LID, and that both the rating task used by the WMS in their admission SJT, as well as the way they wrote their items influenced the surprisingly low degrees of LID interference. Further research to support or disprove this theory is needed. If supported by future analysis, practitioners and SJT designers may leverage on both item tasks and items interpretability to avoid method interference and LID.
Degree
PhD
College and Department
David O. McKay School of Education; Educational Inquiry, Measurement, and Evaluation
Rights
https://lib.byu.edu/about/copyright/
BYU ScholarsArchive Citation
Sampson, Ester, "An Exploratory Analysis of Within and Between-Testlet Local Item Dependence in a Medical School Admission Situational Judgment Test Using Exploratory Structural Equation Modeling" (2024). Theses and Dissertations. 11061.
https://scholarsarchive.byu.edu/etd/11061
Date Submitted
2024-12-20
Document Type
Dissertation
Keywords
situational tests, local item dependence, exploratory structural equation models, testlets, rating task, scenarios dependence
Language
english