Publicación:
Training deep neural networks: a static load balancing approach

Vista sencilla de ítem
dc.contributor.authorMoreno Álvarez, Sergio
dc.contributor.authorHaut, Juan Mario
dc.contributor.authorPaoletti, Mercedes Eugenia
dc.contributor.authorRico Gallego, Juan Antonio
dc.contributor.authorDíaz Martín, Juan Carlos
dc.contributor.authorPlaza, Javier
dc.contributor.orcidhttps://orcid.org/0000-0003-1030-3729
dc.contributor.orcidhttps://orcid.org/0000-0002-4264-7473
dc.contributor.orcidhttps://orcid.org/0000-0002-8435-3844
dc.contributor.orcidhttps://orcid.org/0000-0002-8908-1606
dc.date.accessioned2024-11-15T11:09:07Z
dc.date.available2024-11-15T11:09:07Z
dc.date.issued2020-03-02
dc.descriptionThe registered version of this article, first published in “Journal of Supercomputing 76", is available online at the publisher's website: Springer, https://doi.org/10.1007/s11227-020-03200-6 La versión registrada de este artículo, publicado por primera vez en “Journal of Supercomputing 76", está disponible en línea en el sitio web del editor: Springer, https://doi.org/10.1007/s11227-020-03200-6
dc.description.abstractDeep neural networks are currently trained under data-parallel setups on high-performance computing (HPC) platforms, so that a replica of the full model is charged to each computational resource using non-overlapped subsets known as batches. Replicas combine the computed gradients to update their local copies at the end of each batch. However, differences in performance of resources assigned to replicas in current heterogeneous platforms induce waiting times when synchronously combining gradients, leading to an overall performance degradation. Albeit asynchronous communication of gradients has been proposed as an alternative, it suffers from the so-called staleness problem. This is due to the fact that the training in each replica is computed using a stale version of the parameters, which negatively impacts the accuracy of the resulting model. In this work, we study the application of well-known HPC static load balancing techniques to the distributed training of deep models. Our approach is assigning a different batch size to each replica, proportional to its relative computing capacity, hence minimizing the staleness problem. Our experimental results (obtained in the context of a remotely sensed hyperspectral image processing application) show that, while the classification accuracy is kept constant, the training time substantially decreases with respect to unbalanced training. This is illustrated using heterogeneous computing platforms, made up of CPUs and GPUs with different performance.en
dc.description.versionversión final
dc.identifier.citationMoreno-Álvarez, S., Haut, J.M., Paoletti, M.E. et al. Training deep neural networks: a static load balancing approach. J Supercomput 76, 9739–9754 (2020). https://doi.org/10.1007/s11227-020-03200-6
dc.identifier.doihttps://doi.org/10.1007/s11227-020-03200-6
dc.identifier.issn0920-8542
dc.identifier.urihttps://hdl.handle.net/20.500.14468/24388
dc.journal.titleJournal of Supercomputing
dc.journal.volume76
dc.language.isoen
dc.page.final9754
dc.page.initial9739
dc.publisherSpringer
dc.relation.centerFacultades y escuelas::E.T.S. de Ingeniería Informática
dc.relation.departmentLenguajes y Sistemas Informáticos
dc.rightsinfo:eu-repo/semantics/openAccess
dc.rights.urihttp://creativecommons.org/licenses/by-nc-nd/4.0/deed.es
dc.subject12 Matemáticas::1203 Ciencia de los ordenadores ::1203.17 Informática
dc.subject.keywordsDeep learningen
dc.subject.keywordsHigh-performance computingen
dc.subject.keywordsDistributed trainingen
dc.subject.keywordsHeterogeneous platformsen
dc.titleTraining deep neural networks: a static load balancing approachen
dc.typeartículoes
dc.typejournal articleen
dspace.entity.typePublication
relation.isAuthorOfPublication3482d7bc-e120-48a3-812e-cc4b25a6d2fe
relation.isAuthorOfPublication.latestForDiscovery3482d7bc-e120-48a3-812e-cc4b25a6d2fe
Archivos
Bloque original
Mostrando 1 - 1 de 1
Miniatura
Nombre:
MorenoAlvarez_Sergio_2020TrainingDeepNeuralNe.pdf
Tamaño:
1.5 MB
Formato:
Adobe Portable Document Format
Descargar
Bloque de licencias
Mostrando 1 - 1 de 1
No hay miniatura disponible
Nombre:
license.txt
Tamaño:
3.62 KB
Formato:
Item-specific license agreed to upon submission
Descripción:
Descargar
Colecciones
Artículos y papers