Prediction and Interpretability of HPC I/O Resources Usage with Machine Learning - PEPR NumPEx
Pré-Publication, Document De Travail Année : 2024

Prediction and Interpretability of HPC I/O Resources Usage with Machine Learning

Résumé

I/O management tools proposed for high-performance computing (HPC) environments usually rely on accurate I/O bandwidth estimations to make their decisions. However, finding the correct I/O subsystem configuration that provides this maximal I/O bandwidth is particularly hard. In this work, we focus on finding a good estimate of the number of I/O resources (e.g., OSTs and I/O nodes) that provides the maximal bandwidth while minimizing the system occupation and taking into account the natural I/O variability. We use machine learning techniques to do so, focusing on intrinsic application features and system configurations. We show I/O resource usage is predictable and further study the impact of different features. We also validate our models with four I/O kernels from real applications. Finally, we show that our model, when used for resource allocation, can improve application performance.
Fichier principal
Vignette du fichier
main.pdf (681.74 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04698511 , version 1 (16-09-2024)

Licence

Identifiants

  • HAL Id : hal-04698511 , version 1

Citer

Alexis Bandet, Francieli Boito, Guillaume Pallez. Prediction and Interpretability of HPC I/O Resources Usage with Machine Learning. 2024. ⟨hal-04698511⟩
107 Consultations
72 Téléchargements

Partager

More