Authors:

R Author

Date Created:

Not specified

Course Title:

Professor:

Not specified

About Paper:

High-performance computing (HPC) clusters produce vast amounts of data, but inconsistency in accessing that data limits its utility. This poses a challenge many operators of these systems face: How does one create a centralized system that can monitor and utilize this data? A common solution posed is to adopt a Software as a Service (SaaS) solution. These solutions provide an easy way to store and process data, but can be prohibitively expensive to deploy and operate. As an alternative, we developed our own data warehousing solution, utilizing the processing power of Purdue's most powerful supercomputer; Anvil. To emulate the data handling capabilities of SaaS systems, our warehouse implements an Extract Transform Load (ETL) pipeline using open-source services from the Apache ecosystem like Kafka and Iceberg. My primary contribution to the project was configuring software deployments in Kubernetes, the orchestration platform responsible for deploying these tools. With built-in support for load balancing, and rolling updates, Kubernetes provides a robust framework for running services at scale. The ETL pipeline is operating on Kubernetes, running on Anvil hardware. This system streamlines data access, providing users with a centralized and user-friendly interface for querying cluster data. It also allows admins to quickly respond to ongoing issues using grafana alerting. The result of this solution is a vastly simplified data querying experience. Users no longer need to juggle a diverse set of tools and interfaces, they can instead interact with one standardized sql-based interface and web dashboard. Keywords: HPC; Kubernetes; Apache; Data Warehouse; SQL

Source:

Purdue University / 2025

Topics:

No topics listed

Co-authors:

R Author