Distilling vector space model scores for the assessment of constructed responses with bifactor Inbuilt Rubric method and latent variables

Martínez‑Huertas, José Ángel, Olmos, Ricardo, Jorge‑Botana, Guillermo y León, José A. . (2022) Distilling vector space model scores for the assessment of constructed responses with bifactor Inbuilt Rubric method and latent variables. Behavior Research Methods

Ficheros (Some files may be inaccessible until you login with your e-spacio credentials)
Nombre Descripción Tipo MIME Size
Martinez-Huertas_JA_Distillingv.pdf Martinez-Huertas_JA_Distillingv.pdf application/pdf 1.62MB

Título Distilling vector space model scores for the assessment of constructed responses with bifactor Inbuilt Rubric method and latent variables
Autor(es) Martínez‑Huertas, José Ángel
Olmos, Ricardo
Jorge‑Botana, Guillermo
León, José A.
Materia(s) Psicología
Abstract In this paper, we highlight the importance of distilling the computational assessments of constructed responses to validate the indicators/proxies of constructs/trins using an empirical illustration in automated summary evaluation. We present the validation of the Inbuilt Rubric (IR) method that maps rubrics into vector spaces for concepts’ assessment. Specifically, we improved and validated its scores’ performance using latent variables, a common approach in psychometrics. We also validated a new hierarchical vector space, namely a bifactor IR. 205 Spanish undergraduate students produced 615 summaries of three different texts that were evaluated by human raters and different versions of the IR method using latent semantic analysis (LSA). The computational scores were validated using multiple linear regressions and different latent variable models like CFAs or SEMs. Convergent and discriminant validity was found for the IR scores using human rater scores as validity riteria. While this study was conducted in the Spanish language, the proposed scheme is language-independent and applicable to any language. We highlight four main conclusions: (1) Accurate performance can be observed in topic-detection tasks without hundreds/thousands of pre-scored samples required in supervised models. (2) Convergent/discriminant validity can be improved using measurement models for computational scores as they adjust for measurement errors. (3) Nouns embedded in fragments of instructional text can be an affordable alternative to use the IR method. (4) Hierarchical models, like the bifactor IR, can increase the validity of computational assessments evaluating general and specific knowledge in vector space models. R code is provided to apply the classic and bifactor IR method.
Palabras clave Inbuilt Rubric
Vector space models
Bifactor
Measurement models
Validity
Constructed responses
Editor(es) Springer
Psychonomic Society
Fecha 2022-01-11
Formato application/pdf
Identificador bibliuned:DptoMCC-FPSI-Articulos-Jamartinez-0011
http://e-spacio.uned.es/fez/view/bibliuned:DptoMCC-FPSI-Articulos-Jamartinez-0011
DOI - identifier https://doi.org/10.3758/s13428-021-01764-6
ISSN - identifier 1554-351X eISSN 1554-3528
Nombre de la revista Behavior Research Methods
Número de Volumen 54
Página inicial 2579
Página final 2601
Publicado en la Revista Behavior Research Methods
Idioma eng
Versión de la publicación acceptedVersion
Tipo de recurso Article
Derechos de acceso y licencia http://creativecommons.org/licenses/by-nc-nd/4.0
info:eu-repo/semantics/openAccess
Tipo de acceso Acceso abierto
Notas adicionales La versión registrada de este artículo, publicado por primera vez en Behavior Research Methods (2022) 54, p. 2579–2601, está disponible en línea en el sitio web del editor: Springer; https://doi.org/10.3758/s13428-021-01764-6
Notas adicionales The copyrighted version of this article, first published in Behavior Research Methods (2022) 54, p. 2579–2601, is available online at the publisher's website: Springer; https://doi.org/10.3758/s13428-021-01764-6

 
Versiones
Versión Tipo de filtro
Contador de citas: Google Scholar Search Google Scholar
Estadísticas de acceso: 41 Visitas, 4 Descargas  -  Estadísticas en detalle
Creado: Thu, 29 Feb 2024, 22:39:58 CET