Expanding the Capabilities of Reinforcement Learning via Text Feedback

Expanding Reinforcement Learning Capabilities with Text Feedback

Recent advancements in reinforcement learning (RL) for large language models (LLMs) have revealed a novel approach utilizing text feedback as an intermediary signal to enhance model training efficiency. Current RL strategies often rely on minimal feedback, typically limited to binary rewards. The introduction of text feedback seeks to bridge this gap, offering a more informative, yet cost-effective, alternative.

Introducing RL from Text Feedback (RLTF)

The proposed framework, RL from Text Feedback (RLTF), formalizes a multi-turn RL training setup, where text feedback is utilized during training but is unavailable at inference. Models must internalize the feedback to enhance their performance in single-turn scenarios during testing.

Two innovative methods have been introduced:

• Self Distillation (RLTF-SD): This approach trains the policy to align with its own feedback-conditioned second-turn outputs, encouraging consistency in model responses.
• Feedback Modeling (RLTF-FM): This method aims to predict the feedback as an auxiliary objective, further guiding the model in its learning process.

Theoretical and Empirical Evaluations

The researchers conducted a thorough theoretical analysis of both RLTF methods, followed by empirical evaluations across various tasks, including reasoning puzzles and creative writing exercises. Results consistently demonstrated that both RLTF methods significantly surpassed strong baseline models across these benchmarks.

The findings underscore the potential of integrating text feedback as a rich source of supervision in RL, suggesting this approach could lead to more robust and adaptable LLMs.