Publicación: Arquitectura lambda aplicada a clustering de documentos en contextos Big Data
Cargando...
Fecha
2015-10-01
Autores
Editor/a
Director/a
Tutor/a
Coordinador/a
Prologuista
Revisor/a
Ilustrador/a
Derechos de acceso
Atribución-NoComercial-SinDerivadas 4.0 Internacional
info:eu-repo/semantics/openAccess
info:eu-repo/semantics/openAccess
Título de la revista
ISSN de la revista
Título del volumen
Editor
Universidad Nacional de Educación a Distancia (España). Escuela Técnica Superior de Ingeniería Informática. Departamento de Inteligencia Artificial.
Resumen
La aplicación de técnicas de minería de datos, concretamente de clustering, sobre grandes volúmenes de datos (Big Data) supone un desafío en cuanto a la escalabilidad y al tiempo de respuesta, ya que cantidades crecientes de datos implican tiempos mayores de computación. La arquitectura lambda es un conjunto de recomendaciones de propósito general para diseñar una arquitectura en escenarios Big Data de forma que se resuelva el problema de la latencia y se puedan obtener resultados en tiempo real. En este trabajo se presenta un estudio aplicando la arquitectura lambda sobre el clustering de documentos en contextos Big Data. La problemática que se desea resolver es la elevada latencia que tiene lugar cuando se introducen nuevos documentos en el sistema que realiza clustering. Uno de los puntos clave sugeridos por esta arquitectura es la separación del procesamiento en tres capas: batch layer, speed layer y serving layer. Una problemática adicional al tratar documentos es su elevada dimensionalidad y este problema se soslaya mediante reducción de dimensionalidad con Latent Dirichlet Allocation. Los experimentos se han llevado a cabo utilizando el framework Apache Spark y demuestran que esta combinación de capas permite realizar clustering sobre grandes volúmenes de datos y disponer de resultados actualizados en tiempo real, con calidad del clustering comparable a trabajos similares sobre contextos no Big Data.
The application of data mining techniques, specifically clustering on large volumes of data (such as Big Data), represents a challenge in terms of scalability and response time, given that a higher amount of data involves a higher computation time. The lambda architecture offers a set of general purpose recommendations to design an architecture in Big Data scenarios with the purpose of reducing latency and getting results in real time. This paper presents a study on applying the Lambda architecture on the clustering of documents in a Big Data environment. We focus on the problem of the high latency that occurs when new documents are ingested into the clustering system. One of the key points suggested by this architecture is a separation into three layers of processing: batch layer, serving layer and speed layer. An additional problem in dealing with documents is the high dimensionality when representing such documents, which is approached by means of dimensionality reduction using Latent Dirichlet Allocation. We have seen that the combination of layers proposed in the Lambda Architecture allows clustering of large data volumes and yields results updated in real-time, obtaining a clustering quality comparable to other approaches that do not work on Big Data.
The application of data mining techniques, specifically clustering on large volumes of data (such as Big Data), represents a challenge in terms of scalability and response time, given that a higher amount of data involves a higher computation time. The lambda architecture offers a set of general purpose recommendations to design an architecture in Big Data scenarios with the purpose of reducing latency and getting results in real time. This paper presents a study on applying the Lambda architecture on the clustering of documents in a Big Data environment. We focus on the problem of the high latency that occurs when new documents are ingested into the clustering system. One of the key points suggested by this architecture is a separation into three layers of processing: batch layer, serving layer and speed layer. An additional problem in dealing with documents is the high dimensionality when representing such documents, which is approached by means of dimensionality reduction using Latent Dirichlet Allocation. We have seen that the combination of layers proposed in the Lambda Architecture allows clustering of large data volumes and yields results updated in real-time, obtaining a clustering quality comparable to other approaches that do not work on Big Data.
Descripción
Categorías UNESCO
Palabras clave
arquitectura lambda, clustering de documentos, Big Data, LDA, streaming, Apache Spark, k-means, Lambda architecture, document clustering, Big Data, LDA, streaming, Apache Spark, k-means
Citación
Centro
Facultades y escuelas::E.T.S. de Ingeniería Informática
Departamento
Inteligencia Artificial