Arxiv paper - Self-Improving Robust Preference Optimization
Update: 2025-04-23
Description
In this episode, we discuss Self-Improving Robust Preference Optimization by Eugene Choi, Arash Ahmadian, Matthieu Geist, Oilvier Pietquin, Mohammad Gheshlaghi Azar. The paper introduces Self-Improving Robust Preference Optimization (SRPO), an offline RLHF framework that enables models to self-improve and generalize across tasks by jointly optimizing a self-improvement and generative policy through a min-max objective. SRPO reformulates this objective into a non-adversarial offline loss that can be efficiently optimized using supervised learning. Experiments show SRPO significantly outperforms existing methods like DPO and IPO on benchmarks such as XSum and Arena-Hard, achieving higher win rates against human and AI baselines.
Comments
In Channel
Download from Google Play
Download from App Store
United States00:00
00:00
1.0x
