Authors:

Pranav Sanghi

Date Created:

Not specified

Course Title:

Professor:

Not specified

About Paper:

Reinforcement learning from human feedback (RLHF) is a key technique for aligning large language models (LLMs) with human values, but its training dynamics are often unstable and poorly understood. This instability limits our ability to robustly apply LLMs in real-world, safety- critical applications. In this research, we aim to investigate how the geometry of the training loss landscape contributes to such instability, focusing on two prominent alignment methods: Proximal Policy Optimization (PPO) and Direct Preference Optimization (DPO). Our approach involves visualizing the structure of these loss landscapes by interpolating between reference and fine-tuned policy parameters, as well as sampling filter-normalized directions in weight space. We apply these techniques using a fine-tuned LLM on the Anthropic Helpful- Harmless dataset, and evaluate model behavior using a learned reward function. While results are still preliminary, this study is expected to yield insight into how optimization geometry correlates with training stability. By characterizing sharpness, curvature, and smoothness in the loss surface of each method, we aim to understand why PPO often suffers from instability, whereas DPO appears more robust. Ultimately, our work seeks to inform the development of more stable and interpretable RL- based alignment algorithms for future LLM systems. Keywords: Reinforcement Learning; Large Language Models; Machine Learning

Source:

Purdue University / 2025

Topics:

No topics listed

Co-authors:

Pranav Sanghi