Fuster Baggetto, AlejandroFresno Fernández, Víctor Diego2025-12-032025-12-032022-01-01Alejandro Fuster Baggetto and Victor Fresno. 2022. Is anisotropy really the cause of BERT embeddings not being semantic?. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4271–4281, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.978-1-959429-43-2https://doi.org/10.18653/v1/2022.findings-emnlp.314https://hdl.handle.net/20.500.14468/30998The registered version of this conference paper, first published in "Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4271–4281, Abu Dhabi, United Arab Emirates", is available online at the publisher's website: Association for Computational Linguistics, https://doi.org/10.18653/v1/2022.findings-emnlp.314La versión registrada de esta comunicación, publicada por primera vez en "Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4271–4281, Abu Dhabi, United Arab Emirates", está disponible en línea en el sitio web del editor: Association for Computational Linguistics, https://doi.org/10.18653/v1/2022.findings-emnlp.314In this paper we conduct a set of experiments aimed to improve our understanding of the lack of semantic isometry in BERT, i.e. the lack of correspondence between the embedding and meaning spaces of its contextualized word representations. Our empirical results show that, contrary to popular belief, the anisotropy is not the root cause of the poor performance of these contextual models’ embeddings in semantic tasks. What does affect both the anisotropy and semantic isometry is a set of known biases: frequency, subword, punctuation, and case. For each one of them, we measure its magnitude and the effect of its removal, showing that these biases contribute but do not completely explain the phenomenon of anisotropy and lack of semantic isometry of these contextual language models.eninfo:eu-repo/semantics/openAccess1203.04 Inteligencia artificialactas de congreso