3

Preference optimization techniques have become a standard final stage for training state-of-art large language models (LLMs). However, …