About the QAT code

[JingdiZhou]

Hello , I noticed that your contribution has contained QAT in qat.py , but when I read the code of train.py ,I found there is no real quantization of each model. For example, when I using command

python qat.py --algo a2c --env BreakoutNoFrameskip-v4 -q 7 --quant-delay 5000000 -n 10000000

I found that in a2c.py w_bits and act_bits actually haven't been used in other places ,which means the training process is actually without quantization . Therefore, I want to know if I misunderstood or confirmed that there was no QAT？

    def __init__(self, policy, env, gamma=0.99, n_steps=5, vf_coef=0.25, ent_coef=0.01, max_grad_norm=0.5,
                 learning_rate=7e-4, alpha=0.99, epsilon=1e-5, lr_schedule='constant', verbose=0,
                 tensorboard_log=None, _init_setup_model=True, w_bits=None, act_bits=None, quant_train=None, quant_delay=None, policy_kwargs=None,
                 full_tensorboard_log=False, seed=None, n_cpu_tf_sess=None):

[EdwinRou]

If I understood their work correctly, they indeed do not use Quantization-Aware Training (QAT). Instead, they use ActorQ, which is their own custom method of introducing quantization in Reinforcement Learning (RL) training.

This approach relies on applying quantization to the actor policy neural network, which interacts with the environment, while leaving the learning neural network of the RL framework unchanged. Therefore, quantization is applied during the training process but not to the neural network responsible for learning. This is why it is distinct from QAT and also different from simple Post-Training Quantization (PTQ).

(I haven't read the code yet, so I may have misunderstood your comment)