Authors:

Joshua Kamphuis

Date Created:

Not specified

Course Title:

Professor:

Not specified

About Paper:

Recent years have seen rapid development in artificial intelligence, and an increasingly broad scope of AI as a tool in industries outside of technology. With tools like Stable Diffusion, a text-to-image generative model, AI has entered cultural, artistic spaces. We investigate the use of this diffusion model to visualize music in real- time. We record raw music data and convert it into musical instrument digital interface (MIDI) and Mel- Spectrogram audio formats to extract features that vary between pieces and develop as a piece progresses. These features are used to predict the genre of a piece and the emotions it conveys using a neural network and support vector machine, deep learning models trained on large datasets of samples from classical music songs. The outputs from these models are used to generate prompts that Stable Diffusion translates into images, which are generated multiple times per second and strung together into video. The accuracies of the deep learning algorithms will be evaluated on training data. For real-time implementation, we evaluate Stable Diffusion's various available pipelines and modifiable parameters to balance quality and speed and investigate frame interpolation to make video smoother. These metrics will provide conclusions and recommendations on impactful audio features for music emotion recognition (MER) and efficient use of Stable Diffusion.

Source:

Purdue University / 2023

Topics:

No topics listed

Co-authors:

Joshua Kamphuis