Enhancing topic-detection in computerized assessments of constructed responses with distributional models of language

Martínez-Huertas, José Ángel, Olmos, Ricardo y León, José A. . (2021) Enhancing topic-detection in computerized assessments of constructed responses with distributional models of language. Expert Systems with Applications

Ficheros (Some files may be inaccessible until you login with your e-spacio credentials)
Nombre Descripción Tipo MIME Size
Martinez-Huertas_JA_Enhancingto.pdf Martinez-Huertas_JA_Enhancingto.pdf application/pdf 568.28KB

Título Enhancing topic-detection in computerized assessments of constructed responses with distributional models of language
Autor(es) Martínez-Huertas, José Ángel
Olmos, Ricardo
León, José A.
Materia(s) Psicología
Abstract Usually, computerized assessments of constructed responses use a predictive-centered approach instead of a validity-centered one. Here, we compared the convergent and discriminant validity of two computerized assessment methods designed to detect semantic topics in constructed responses: Inbuilt Rubric (IR) and Partial Contents Similarity (PCS). While both methods are distributional models of language and use the same Latent Semantic Analysis (LSA) prior knowledge, they produce different semantic representations. PCS evaluates constructed responses using non-meaningful semantic dimensions (this method is the standard LSA assessment of constructed responses), but IR endows original LSA semantic space coordinates with meaning. In the present study, 255 undergraduate and high school students were allocated one of three texts and were tasked to make a summary. A topic- detection task was conducted comparing IR and PCS methods. Evidence from convergent and discriminant validity was found in favor of the IR method for topic-detection in computerized constructed response assessments. In this line, the multicollinearity of PCS method was larger than the one of IR method, which means that the former is less capable of discriminating between related concepts or meanings. Moreover, the semantic representations of both methods were qualitatively different, that is, they evaluated different concepts or meanings. The implications of these automated assessment methods are also discussed. First, the meaningful coordinates of the Inbuilt Rubric method can accommodate expert rubrics for computerized assessments of constructed responses improving computer-assisted language learning. Second, they can provide high-quality computerized feedback accurately detecting topics in other educational constructed response assessments. Thus, it is concluded that: (1) IR method can represent different concepts and contents of a text, simultaneously mapping a considerable variability of contents in constructed responses; (2) IR method semantic representations have a qualitatively different meaning than the LSA ones and present a desirable multicollinearity that promotes the discriminant validity of the scores of distributional models of language; and (3) IR method can extend the performance and the applications of current LSA semantic representations by endowing the dimensions of the semantic space with semantic meanings.
Palabras clave Inbuilt Rubric
constructed responses
summaries
topic detection
Latent Semantic Analysis
Automated Summary Evaluation
Editor(es) Elsevier
Fecha 2021-12-15
Formato application/pdf
Identificador bibliuned:DptoMCC-FPSI-Articulos-Jamartinez-0017
http://e-spacio.uned.es/fez/view/bibliuned:DptoMCC-FPSI-Articulos-Jamartinez-0017
DOI - identifier https://doi.org/10.1016/j.eswa.2021.115621
ISSN - identifier 0957-4174
Nombre de la revista Expert Systems with Applications
Número de Volumen 185
Publicado en la Revista Expert Systems with Applications
Idioma eng
Versión de la publicación acceptedVersion
Tipo de recurso Article
Derechos de acceso y licencia http://creativecommons.org/licenses/by-nc-nd/4.0
info:eu-repo/semantics/openAccess
Tipo de acceso Acceso abierto
Notas adicionales La versión registrada de este artículo, publicado por primera vez en Expert Systems with Applications (2021) 185, está disponible en línea en el sitio web del editor: Elsevier; https://doi.org/10.1016/j.eswa.2021.115621
Notas adicionales The copyrighted version of this article, first published in Expert Systems with Applications (2021) 185, is available online at the publisher's website: Elsevier; https://doi.org/10.1016/j.eswa.2021.115621

 
Versiones
Versión Tipo de filtro
Contador de citas: Google Scholar Search Google Scholar
Estadísticas de acceso: 44 Visitas, 9 Descargas  -  Estadísticas en detalle
Creado: Fri, 01 Mar 2024, 00:53:58 CET