Question about the coefficient of v_loss in PPO

[ZaberKo]

In brax's PPO implementation, why there are duplicate 0.5 in v_loss? In my experience, the coefficient should be 0.5 or 1.0, instead of 0.25.

brax/brax/training/agents/ppo/losses.py

Line 182 in 19bbd9b

v_loss = jnp.mean(v_error * v_error) * 0.5 * 0.5

[btaba]

Hi @ZaberKo , this question came up here #593

We've considered adding a v_loss_coef as a hyper but it never made too much of a diff. If you find that it helps your policies significantly, please send us a PR or issue

[ZaberKo]

After some tests, I agree the value of v_loss_coef with 0.5 or 0.25 makes no diff. Nevertheless, it is strange to put this extra 0.5, which may lead to some confusion.