Publicación:
Heterogeneous model parallelism for deep neural networks

Vista sencilla de ítem
dc.contributor.authorMoreno Álvarez, Sergio
dc.contributor.authorHaut, Juan M.
dc.contributor.authorPaoletti, Mercedes Eugenia
dc.contributor.authorRico Gallego, Juan Antonio
dc.contributor.orcidhttps://orcid.org/0000-0003-1030-3729
dc.contributor.orcidhttps://orcid.org/0000-0002-4264-7473
dc.date.accessioned2024-11-18T09:24:57Z
dc.date.available2024-11-18T09:24:57Z
dc.date.issued2021-06-21
dc.descriptionThe registered version of this article, first published in “Neurocomputing, Volume 441", is available online at the publisher's website: Elsevier, https://doi.org/10.1016/j.neucom.2021.01.125 La versión registrada de este artículo, publicado por primera vez en “Neurocomputing, Volume 441", está disponible en línea en el sitio web del editor: Elsevier, https://doi.org/10.1016/j.neucom.2021.01.125
dc.description.abstractDeep neural networks (DNNs) have transformed computer vision, establishing themselves as the current state-of-the-art for image processing. Nevertheless, the training of current large DNN models is one of the main challenges to be solved. In this sense, data-parallelism has been the most widespread distributed training strategy since it is easy to program and can be applied to almost all cases. However, this solution suffers from several limitations, such as its high communication requirements and the memory constraints when training very large models. To overcome these limitations model-parallelism has been proposed, solving the most substantial problems of the former strategy. However, describing and implementing the parallelization of the training of a DNN model across a set of processes deployed on several devices is a challenging task. Current proposed solutions assume a homogeneous distribution, being impractical when working with devices of different computational capabilities, which is quite common on high performance computing platforms. To address previous shortcomings, this work proposes a novel model-parallelism technique considering heterogeneous platforms, where a load balancing mechanism between uneven devices of an HPC platform has been implemented. Our proposal takes advantage of the Google Brain’s Mesh-TensorFlow for convolutional networks, splitting computing tensors across filter dimension in order to balance the computational load of the available devices. Conducted experiments show an improvement in the exploitation of heterogeneous computational resources, enhancing the training performance. The code is available on: https://github.com/mhaut/HeterogeneusModelDNN.en
dc.description.versionversión publicada
dc.identifier.citationSergio Moreno-Alvarez, Juan M. Haut, Mercedes E. Paoletti, Juan A. Rico-Gallego, Heterogeneous model parallelism for deep neural networks, Neurocomputing, Volume 441, 2021, Pages 1-12, ISSN 0925-2312, https://doi.org/10.1016/j.neucom.2021.01.125
dc.identifier.doihttps://doi.org/10.1016/j.neucom.2021.01.125
dc.identifier.issn0925-2312 | eISSN 1872-8286
dc.identifier.urihttps://hdl.handle.net/20.500.14468/24396
dc.journal.titleNeurocomputing
dc.journal.volume441
dc.language.isoen
dc.page.final12
dc.page.initial1
dc.publisherELSEVIER
dc.relation.centerFacultades y escuelas::E.T.S. de Ingeniería Informática
dc.relation.departmentLenguajes y Sistemas Informáticos
dc.rightsinfo:eu-repo/semantics/openAccess
dc.rights.urihttp://creativecommons.org/licenses/by-nc-nd/4.0/deed.es
dc.subject12 Matemáticas::1203 Ciencia de los ordenadores ::1203.17 Informática
dc.subject.keywordsdeep learningen
dc.subject.keywordshigh performance computingen
dc.subject.keywordsdistributed trainingen
dc.subject.keywordsheterogeneous platformsen
dc.subject.keywordsmodel parallelismen
dc.titleHeterogeneous model parallelism for deep neural networksen
dc.typeartículoes
dc.typejournal articleen
dspace.entity.typePublication
relation.isAuthorOfPublication3482d7bc-e120-48a3-812e-cc4b25a6d2fe
relation.isAuthorOfPublication.latestForDiscovery3482d7bc-e120-48a3-812e-cc4b25a6d2fe
Archivos
Bloque original
Mostrando 1 - 1 de 1
Miniatura
Nombre:
MorenoAlvarez_Sergio_2021HeterogeneousModelPa.pdf
Tamaño:
1.73 MB
Formato:
Adobe Portable Document Format
Descargar
Bloque de licencias
Mostrando 1 - 1 de 1
No hay miniatura disponible
Nombre:
license.txt
Tamaño:
3.62 KB
Formato:
Item-specific license agreed to upon submission
Descripción:
Descargar
Colecciones
Artículos y papers