Authors:

Abigale Tucker

Date Created:

Not specified

Course Title:

Professor:

Not specified

About Paper:

Analyzing data produced by supercomputers is essential for performance monitoring, anomaly detection, and ensuring the long-term usability of computational resources. This project presents a data pipeline and warehouse for collecting, transforming, storing, and querying supercomputing data using data engineering tools. Services are configured and deployed in a persistent, container-based environment with a database on physical hardware. Initially, simulated data, such as system logs and monitoring data, is ingested and processed into structured, queryable formats. This testing validates pipeline functionality before integrating production data. The resulting data store will serve as a foundation for future analysis, visualization, and operational insights for use by data scientists, system engineers, and end users of the computing resources. This work advances the development of scalable infrastructure that support system-wide monitoring, performance optimization, and resource efficiency within Purdue University's supercomputing environment. Keywords: Data Engineering; ETL; HPC; Data Warehouse; Data Pipeline

Source:

Purdue University / 2025

Topics:

No topics listed

Co-authors:

Abigale Tucker