Publicación: Evaluación del rendimiento de algoritmos sobre frameworks Hadoop y Spark
Cargando...
Fecha
2024-09
Autores
Editor/a
Director/a
Tutor/a
Coordinador/a
Prologuista
Revisor/a
Ilustrador/a
Derechos de acceso
info:eu-repo/semantics/openAccess
Título de la revista
ISSN de la revista
Título del volumen
Editor
Universidad de Educación a Distancia (UNED)
Resumen
Big Data e inteligencia artificial son conceptos que están a la orden del día en la actualidad, pero cómo funcionan internamente es una incógnita para la mayoría de la sociedad, por ello en este documento se busca esclarecer dichos términos y como consecuencia entender y evaluar distintas metodologías que llevan a su utilización. Para ello, se cuenta con frameworks como Apache Hadoop o Apache Spark, los cuales brindan la posibilidad de procesar grandes cantidades de datos distribuyéndolos en diferentes máquinas para su procesamiento.
Como parte del análisis, se comparará la eficiencia de ambos frameworks, no solo en términos de tiempo de ejecución, sino también en el uso de recursos como CPU, memoria y red, proporcionando una evaluación más exhaustiva de su rendimiento en un entorno distribuido.
Como caso práctico en este trabajo, se utilizarán datos de imágenes por satélite, las cuales pueden alcanzar un tamaño considerable haciendo imposible siquiera su apertura en una configuración local, mucho menos su procesamiento. Estas imágenes, a diferencia de las que son utilizadas continuamente cuentan con más de tres bandas de color (RGB), es decir, son multibanda o multiespectrales, permitiendo realizar una serie de cálculos sobre ellas realmente útiles en el ámbito de la teledetección. Además, los resultados obtenidos tras el procesamiento de estas imágenes podrán ser visualizados y empleados para la toma de decisiones en áreas como la agricultura de precisión o el monitoreo ambiental, subrayando el valor práctico de la información extraída.
La metodología a seguir será el desarrollo y puesta en marcha de una infraestructura capaz de cargar ingentes cantidades de datos, distribuirlos en un clúster compuesto por varias máquinas, realizar el cálculo pertinente de manera distribuida y recuperar el resultado final.
Además de probar y evaluar los dos frameworks mencionados anteriormente, se utilizarán herramientas y plataformas extra como Apache Hive y Amazon Web Services, proporcionando un entorno robusto y escalable para manejar grandes volúmenes de datos.
Big Data and artificial intelligence are concepts that are highly relevant today, but how they work internally remains a mystery to most of society. Therefore, this document seeks to clarify these terms and, as a result, understand and evaluate different methodologies that lead to their use. For this purpose, frameworks such as Apache Hadoop and Apache Spark will be utilized, which offer the possibility of processing large amounts of data by distributing them across different machines for processing. As part of the analysis, the efficiency of both frameworks will be compared, not only in terms of execution time but also in resource usage, such as CPU, memory, and network, providing a more comprehensive assessment of their performance in a distributed environment. As a practical case study in this work, satellite imagery data will be used, which can reach a considerable size, making it impossible to even open them in a local configuration, let alone process them. These images, unlike those commonly used, have more than three color bands (RGB), that is, they are multiband or multispectral, allowing a series of calculations to be performed on them that are highly useful in the field of remote sensing. Additionally, the results obtained after processing these images can be visualized and employed for decision-making in areas such as precision agriculture or environmental monitoring, highlighting the practical value of the extracted information. The methodology to be followed will involve the development and implementation of an infrastructure capable of loading vast amounts of data, distributing them across a cluster composed of several machines, performing the relevant calculations in a distributed manner, and retrieving the final result. In addition to testing and evaluating the two frameworks mentioned earlier, additional tools and platforms such as Apache Hive and Amazon Web Services will be used, providing a robust and scalable environment for handling large volumes of data.
Big Data and artificial intelligence are concepts that are highly relevant today, but how they work internally remains a mystery to most of society. Therefore, this document seeks to clarify these terms and, as a result, understand and evaluate different methodologies that lead to their use. For this purpose, frameworks such as Apache Hadoop and Apache Spark will be utilized, which offer the possibility of processing large amounts of data by distributing them across different machines for processing. As part of the analysis, the efficiency of both frameworks will be compared, not only in terms of execution time but also in resource usage, such as CPU, memory, and network, providing a more comprehensive assessment of their performance in a distributed environment. As a practical case study in this work, satellite imagery data will be used, which can reach a considerable size, making it impossible to even open them in a local configuration, let alone process them. These images, unlike those commonly used, have more than three color bands (RGB), that is, they are multiband or multispectral, allowing a series of calculations to be performed on them that are highly useful in the field of remote sensing. Additionally, the results obtained after processing these images can be visualized and employed for decision-making in areas such as precision agriculture or environmental monitoring, highlighting the practical value of the extracted information. The methodology to be followed will involve the development and implementation of an infrastructure capable of loading vast amounts of data, distributing them across a cluster composed of several machines, performing the relevant calculations in a distributed manner, and retrieving the final result. In addition to testing and evaluating the two frameworks mentioned earlier, additional tools and platforms such as Apache Hive and Amazon Web Services will be used, providing a robust and scalable environment for handling large volumes of data.
Descripción
Categorías UNESCO
Palabras clave
Apache, Hadoop, Spark, Hive, AWS
Citación
Cañada Rostro, Carlos (2024) Evaluación del rendimiento de algoritmos sobre frameworks Hadoop y Spark. Trabajo Fin de Máster. Universidad de Educación a Distancia (UNED)
Centro
Facultades y escuelas::E.T.S. de Ingeniería Informática
Departamento
Sistemas de Comunicación y Control