Improving reliability estimation in cognitive diagnosis modeling

Schames Kreitchmann, Rodrigo, Torre, Jimmy de la, Sorrel, Miguel A., Nájera, pablo y Abad, Francisco J. . (2023) Improving reliability estimation in cognitive diagnosis modeling. Behavior Research Methods (2023) 55:3446–3460

Ficheros (Some files may be inaccessible until you login with your e-spacio credentials)
Nombre Descripción Tipo MIME Size
SCHAMES_KREITCHMANN_Rodrigo_Improving_reliability.pdf SCHAMES_KREITCHMANN_Rodrigo_Improving_reliability.pdf application/pdf 2.03MB

Título Improving reliability estimation in cognitive diagnosis modeling
Autor(es) Schames Kreitchmann, Rodrigo
Torre, Jimmy de la
Sorrel, Miguel A.
Nájera, pablo
Abad, Francisco J.
Materia(s) Psicología
Abstract Cognitive diagnosis models (CDMs) are used in educational, clinical, or personnel selection settings to classify respondents with respect to discrete attributes, identifying strengths and needs, and thus allowing to provide tailored training/treatment. As in any assessment, an accurate reliability estimation is crucial for valid score interpretations. In this sense, most CDM reliability indices are based on the posterior probabilities of the estimated attribute profiles. These posteriors are traditionally computed using point estimates for the model parameters as approximations to their populational values. If the uncertainty around these parameters is unaccounted for, the posteriors may be overly peaked, deriving into overestimated reliabilities. This article presents a multiple imputation (MI) procedure to integrate out the model parameters in the estimation of the posterior distributions, thus correcting the reliability estimation. A simulation study was conducted to compare the MI procedure with the traditional reliability estimation. Five factors were manipulated: the attribute structure, the CDM model (DINA and G-DINA), test length, sample size, and item quality. Additionally, an illustration using the Examination for the Certificate of Proficiency in English data was analyzed. The effect of sample size was studied by sampling subsets of subjects from the complete data. In both studies, the traditional reliability estimation systematically provided overestimated reliabilities, whereas the MI procedure offered more accurate results. Accordingly, practitioners in small educational or clinical settings should be aware that the reliability estimation using model parameter point estimates may be positively biased. R codes for the MI procedure are made available
Palabras clave Cognitive diagnosis
Diagnostic classification
Reliability
Classification accuracy
Multiple imputation
Editor(es) Springer
Fecha 2023-10-01
Formato application/pdf
Identificador bibliuned:DptoMCC-FPSI-Articulos-Rschames-0002
http://e-spacio.uned.es/fez/view/bibliuned:DptoMCC-FPSI-Articulos-Rschames-0002
DOI - identifier https://doi.org/10.3758/s13428-022-01967-5
ISSN - identifier 1554-351X; eISSN: 1554-3528
Nombre de la revista Behavior Research Methods
Número de Volumen 55
Página inicial 3446
Página final 3460
Publicado en la Revista Behavior Research Methods (2023) 55:3446–3460
Idioma eng
Versión de la publicación publishedVersion
Tipo de recurso Article
Derechos de acceso y licencia http://creativecommons.org/licenses/by/4.0
info:eu-repo/semantics/openAccess
Tipo de acceso Acceso abierto
Notas adicionales The registered version of this article, first published in Behavior Research Methods (2023) 55:3446–3460, is available online at the publisher's website: Springer, https://doi.org/10.3758/s13428-022-01967-5
Notas adicionales La versión registrada de este artículo, publicado por primera vez en Behavior Research Methods (2023) 55:3446–3460, está disponible en línea en el sitio web del editor: Springer, https://doi.org/10.3758/s13428-022-01967-5

 
Versiones
Versión Tipo de filtro
Contador de citas: Google Scholar Search Google Scholar
Estadísticas de acceso: 34 Visitas, 7 Descargas  -  Estadísticas en detalle
Creado: Tue, 06 Feb 2024, 19:15:00 CET