Publicación:
Distilling vector space model scores for the assessment of constructed responses with bifactor Inbuilt Rubric method and latent variables

Vista sencilla de ítem
dc.contributor.authorOlmos, Ricardo
dc.contributor.authorJorge Botana, Guillermo de
dc.contributor.authorLeón, José A.
dc.contributor.authorMartínez Huertas, José Ángel
dc.date.accessioned2024-05-20T11:49:10Z
dc.date.available2024-05-20T11:49:10Z
dc.date.issued2022-01-11
dc.description.abstractIn this paper, we highlight the importance of distilling the computational assessments of constructed responses to validate the indicators/proxies of constructs/trins using an empirical illustration in automated summary evaluation. We present the validation of the Inbuilt Rubric (IR) method that maps rubrics into vector spaces for concepts’ assessment. Specifically, we improved and validated its scores’ performance using latent variables, a common approach in psychometrics. We also validated a new hierarchical vector space, namely a bifactor IR. 205 Spanish undergraduate students produced 615 summaries of three different texts that were evaluated by human raters and different versions of the IR method using latent semantic analysis (LSA). The computational scores were validated using multiple linear regressions and different latent variable models like CFAs or SEMs. Convergent and discriminant validity was found for the IR scores using human rater scores as validity riteria. While this study was conducted in the Spanish language, the proposed scheme is language-independent and applicable to any language. We highlight four main conclusions: (1) Accurate performance can be observed in topic-detection tasks without hundreds/thousands of pre-scored samples required in supervised models. (2) Convergent/discriminant validity can be improved using measurement models for computational scores as they adjust for measurement errors. (3) Nouns embedded in fragments of instructional text can be an affordable alternative to use the IR method. (4) Hierarchical models, like the bifactor IR, can increase the validity of computational assessments evaluating general and specific knowledge in vector space models. R code is provided to apply the classic and bifactor IR method.en
dc.description.versionversión final
dc.identifier.doihttps://doi.org/10.3758/s13428-021-01764-6
dc.identifier.issn1554-351X eISSN 1554-3528
dc.identifier.urihttps://hdl.handle.net/20.500.14468/12593
dc.journal.titleBehavior Research Methods
dc.journal.volume54
dc.language.isoen
dc.publisher['Springer', 'Psychonomic Society']
dc.relation.centerFacultad de Psicología
dc.relation.departmentMetodología de las Ciencias del Comportamiento
dc.rightsAtribución-NoComercial-SinDerivadas 4.0 Internacional
dc.rightsinfo:eu-repo/semantics/openAccess
dc.rights.urihttp://creativecommons.org/licenses/by-nc-nd/4.0
dc.subject.keywordsInbuilt Rubric
dc.subject.keywordsVector space models
dc.subject.keywordsBifactor
dc.subject.keywordsMeasurement models
dc.subject.keywordsValidity
dc.subject.keywordsConstructed responses
dc.titleDistilling vector space model scores for the assessment of constructed responses with bifactor Inbuilt Rubric method and latent variableses
dc.typeartículoes
dc.typejournal articleen
dspace.entity.typePublication
relation.isAuthorOfPublicationca510876-0be8-438a-a565-ac5f8953fb78
relation.isAuthorOfPublication.latestForDiscoveryca510876-0be8-438a-a565-ac5f8953fb78
Archivos
Bloque original
Mostrando 1 - 1 de 1
Miniatura
Nombre:
Martinez-Huertas_JA_Distillingv.pdf
Tamaño:
1.62 MB
Formato:
Adobe Portable Document Format
Descargar
Colecciones
Artículos y papers