Jan 29, 2024 · The initial phase of RLHF involves learning human values using a reward model from ranking data. It is observed that the performance of the ...

Jan 29, 2024 · This paper delves into these issues, leveraging the theoretical insights to design improved reward learning algorithm termed 'Iterative Data ...

Jan 29, 2024 · This paper introduces Iterative Data Smoothing (IDS) as a solution to reward overfitting and overoptimization in Reinforcement Learning from ...

This paper demonstrates that optimizing for response length is a significant factor behind RLHF's reported improvements in these settings. First, we study the ...

Apr 17, 2024 · Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF. Paper • 2401.16335 • Published Jan 29 • 1 ...

Jan 30, 2024 · Reinforcement Learning from Human Feedback (RLHF) is a pivotal technique that aligns language models closely with human-centric values.

Jan 29, 2024 · Reinforcement Learning from Human Feedback (RLHF) is a pivotal technique that aligns language models closely with human-centric values.

Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF ... Reinforcement Learning from Human Feedback (RLHF) is a pivotal technique ...

3 days ago · To mitigate overoptimization, we first propose a theoretical algorithm that chooses the best policy for an adversarially chosen reward model; ...

Jan 30, 2024 · The approach is validated on both a multi-armed bandit test problem, and RLHF with Pythia using a larger reward model as ground truth.