An IRT Mixture Model for Rating Scale Confusion Associated with Negatively Worded Items in Measures of Social-Emotional Learning
Daniel Bolt, Yang Caroline Wang, Robert H. Meyer, & Libby Pier
We illustrate the application of mixture IRT models to evaluate respondent confusion due to the negative wording of certain items on a social-emotional learning (SEL) assessment. Using actual student self-report ratings on four social-emotional learning scales collected from students in grades 3–12 from CORE Districts in the state of California, we also evaluate the consequences of the potential confusion in biasing student- and school-level scores as well as the estimated correlational relationships between SEL constructs and student-level variables. Models of both full and partial confusion are examined. Our results suggest that (1) rating scale confusion due to negatively worded items does appear to be present; (2) the confusion is most prevalent at lower grade levels (third–fifth); and (3) the occurrence of confusion is positively related to both reading proficiency and ELL status, as anticipated, and consequently biases estimates of SEL correlations with these student-level variables. For these reasons, we suggest future iterations of the SEL measures use only positively oriented items.