Authors:

Austin Riley Lovell

Date Created:

Not specified

Course Title:

Professor:

Not specified

About Paper:

Estimating when the scheduler will run a job submitted to a high-performance computing system is an important task that can improve the user experience and help optimize resource allocation within the system. Due to the complex nature of jobs submitted to HPC systems and their variable run times, queue time prediction is challenging. Models attempting to solve this problem have been developed in the past; however, they often wildly mispredict queue times and could be more reliable. In this work, there is an attempt to create queue time estimation models based on modern deep learning architecture. The approach incorporates feature engineering using interval trees based on job submit, start, and end time to find connections between queued jobs and the parameters of currently running and previously queued jobs to improve prediction accuracy. Synthetic data creation using the SMOTE method and feature normalization through log and min-max scaling are utilized before feeding historic jobs from the Slurm Workload Manager on Anvil into a feed-forward neural network for training. The results are benchmarked against KARNAK 2.0, a pre-existing decision tree-based prediction method, and a traditional XGBoost regression model by comparing prediction validity within various minute thresholds. It is expected that the deep learning-based model will outperform these existing methods. The model will be deployed through a command line tool on Anvil. Given a specific queued job or a hypothetical job with certain requested resources, users can receive a queue time estimate and optimize their job submissions based on this information. Keywords: Machine Learning; Neural Network; AI; HPC; High Performance Computing

Source:

Purdue University / 2024

Topics:

No topics listed

Co-authors:

Austin Riley Lovell