4 classes model; targeted attacks are different

[jS5t3r]

Follow up from #76

Since I have a model trained in only 4 classes I adapted the number of targets t.

I changed this block:

auto-attack/autoattack/autoattack.py

Line 265 in 6482e4d

if version == 'standard':

to

        elif version == 'standard_4':
            self.attacks_to_run = ['apgd-ce', 'apgd-t', 'fab-t', 'square']
            if self.norm in ['Linf', 'L2']:
                self.apgd.n_restarts = 1
                self.apgd_targeted.n_target_classes = 3 # 9
            elif self.norm in ['L1']:
                self.apgd.use_largereps = True
                self.apgd_targeted.use_largereps = True
                self.apgd.n_restarts = 5
                self.apgd_targeted.n_target_classes = 3 # 5
            self.fab.n_restarts = 1
            self.apgd_targeted.n_restarts = 1
            self.fab.n_target_classes = 3 # 9
            #self.apgd_targeted.n_target_classes = 9
            self.square.n_queries = 5000

where t = classes - 1 = 4 - 1 = 3, so that I dont get any warning.

It seems that APGD-CE is creating most perturbations.

using standard_4 version including apgd-ce, apgd-t, fab-t, square
initial accuracy: 93.00%
apgd-ce - 1/1 - 458 out of 465 successfully perturbed
robust accuracy after APGD-CE: 1.40% (total time 112.7 s)
apgd-t - 1/1 - 0 out of 7 successfully perturbed
robust accuracy after APGD-T: 1.40% (total time 122.2 s)
fab-t - 1/1 - 0 out of 7 successfully perturbed
robust accuracy after FAB-T: 1.40% (total time 133.2 s)
square - 1/1 - 0 out of 7 successfully perturbed
robust accuracy after SQUARE: 1.40% (total time 169.0 s)
max Linf perturbation: 0.00784, nan in tensor: 0, max: 1.00000, min: 0.00000
robust accuracy: 1.40%

Well, I think that I am not doing something wrong, right?

[fra31]

Looks fine to me. As sanity check you can run apgd-ce as last attack by modifying the order in the self.attacks_to_run list.

[jS5t3r]

On page 4 of this publication: https://arxiv.org/abs/2112.01601
Figure 3 shows an examination where the classifier is trained on 1000 classes. (standard configuration: https://github.com/adverML/auto-attack/blob/aec817234bc565004fe3e9bee9afe41e931ca9ad/autoattack/autoattack.py#L277)
Figure 4 shows an examination where the classifier is trained on 4 classes. I used this configuration: https://github.com/adverML/auto-attack/blob/aec817234bc565004fe3e9bee9afe41e931ca9ad/autoattack/autoattack.py#L293

I am just wondering, that the plots should look more similar.