Going Further: Flatness at the Rescue of Early Stopping for Adversarial Example Transferability

The transferability of adversarial examples is a prolific research topic [Benz et al., 2021, Dong et al., 2018, Gubri et al., 2022a, b, Li et al., 2018, Lin et al., 2019, Springer et al., 2021, Wu et al., 2020, Xie et al., 2019, Zhao et al., 2022]. Zhao et al. [2022] suggest comparing transferability techniques within specific categories, a recommendation adhered to in Section 6. They classify gradient-based transferability techniques into four categories: model augmentation, data augmentation, attack optimizers, and feature-based attacks. Section 6 shows that our method improves these techniques when combined. Model augmentation adds randomness to the weights or the architecture to avoid specific adversarial examples: GN [Li et al., 2018] uses dropout or skip erosion, SGM [Wu et al., 2020] favors gradients from skip connections during the backward pass, LGV [Gubri et al., 2022b] collects models along the SGD trajectory during a few additional epochs with a high learning rate. Data augmentation techniques transform the inputs during the attack: DI [Xie et al., 2019] randomly resizes the input, SI [Lin et al., 2019] rescales the input, and VT [Wang and He, 2021] smooths the gradients locally. Attack optimizers smooth updates during gradient ascent with momentum (MI, Dong et al. [2018]) or Nesterov accelerated gradient (NI, Lin et al. [2019]), or minimize sharpness (RAP, [Qin et al., 2022]). RAP minimizes sharpness through a min-max bi-level optimization problem, similar to SAM but in the input space. Section 6 shows that SAM and RAP are best combined, indicating their complementary effects on two distinct factors.

Despite the important amount of work on transferability, the way to train an effective single surrogate base model has received little attention in the literature [Zhao et al., 2022]. Benz et al. [2021], Nitin [2021], Zhang et al. [2021] point that early stopping SGD improves transferability. Springer et al. [2021] propose SAT, slight adversarial training that uses tiny perturbations to filter out some non-robust features. Section 5 evaluates SAT. Our approach sheds new light on the relation between flatness and transferability. Springer et al. [2021] implicitly flatten the surrogate model, since adversarial trained models are flatter than their naturally trained counterparts [Stutz et al., 2021]. We observe a similar implicit link with early stopping in Section 4. Gubri et al. [2022b] propose the surrogate-target misalignment hypothesis to explain why flat minima in the weight space are better surrogate models. Section 6 shows that LGV, their model augmentation technique, is best with ours, indicating complementary effects on two distinct factors, respectively ensembling diverse representations and training a single generic representation.

Benz et al. [2021], Zhang et al. [2021], Nitin [2021] point out that fully trained surrogate models are not optimal for transferability and propose a hypothesis based on the perspective of robust and non-robust features [Ilyas et al., 2019]. Ilyas et al. [2019] disentangles features that are highly predictive and robust to adversarial perturbations (RFs), and features that are also highly predictive but non-robust to adversarial perturbations (NRFs). A feature $f_{r}$ is $\gamma$-robust if it remains predictive under a specified set of adversarial perturbations $\Delta$, $\mathbb{E}_{(x,y)\sim D}\left[\inf_{\delta\in\Delta(x)}y\cdot f_{r}(x+\delta)\right]\geq\gamma$. A NRF $f_{nr}$ is $\beta$-predictive, i.e, $\mathbb{E}_{(x,y)\sim D}\left[y\cdot f_{nr}(x)\right]\geq\beta$ but is not $\gamma$-robust feature for any $\gamma\geq 0$. According to Benz et al. [2021], Nitin [2021], the training of DNNs mainly learns RFs first and then learns NRFs. We term this the RFs/NRFs evolution hypothesis (Table 1). NRFs are transferable [Ilyas et al., 2019], but also brittle. RFs are more stable and can improve transferability [Springer et al., 2021, Zhang et al., 2021]. RFs can be attacked: RFs against $L_{p}$ norm $\varepsilon$ perturbations are vulnerable against perturbations with higher $\varepsilon^{\prime}$ or another $L_{p^{\prime}}$ norm [Springer et al., 2021, Zhang et al., 2021]. Section 3 provides some observations that tend to refute the RFs/NRFs evolution hypothesis (summary in Table 2).

Several training techniques increase natural generalization and reduce loss sharpness in the weight space. Keskar et al. [2017] links batch size to sharpness, defined in the weight space as $\max_{\|\epsilon\|_{2}\leq\rho}\mathcal{L}(w+\epsilon)-\mathcal{L}(w)$. SWA [Izmailov et al., 2018] averages the weights at the last epochs for flatness. SAM [Foret et al., 2020] minimizes the maximum loss around a neighborhood by performing a gradient ascent step, defined by $\epsilon_{t}=\rho\frac{\nabla\mathcal{L}(w_{t})}{\|\nabla\mathcal{L}(w_{t})\|_{2}}$, followed by a gradient descent step, $w_{t+1}=w_{t}-\alpha_{t}\left(\nabla\mathcal{L}(w_{t}+\epsilon_{t})+\lambda w_{t}\right)$. At the cost of one additional forward-backward pass per iteration, SAM avoids deep, sharp holes on the surface of the loss landscape [Kaddour et al., 2022]. Its $\rho$ hyperparameter controls the size of flat neighborhoods. Several variants exist that improve natural generalization [Kwon et al., 2021, Zhuang et al., 2022] or efficiency [Liu et al., 2022, Du et al., 2021] (description in Appendix E). Nevertheless, the relationship between sharpness and natural generalization is subject to scientific controversy [Andriushchenko et al., 2023, Wen et al., 2023, Bisla et al., 2022]. In Sections 5 and 6, we explore the use of SWA, SAM and six variants to train better surrogate models.

First, we check that a fully trained surrogate model is not optimal for transferability. We train two ResNet-50 surrogate models on CIFAR-10 and ImageNet using standard settings. Appendix B reports the success rates of the BIM attack applied at every epoch and evaluated on 10 fully trained target models per dataset. For both datasets and diverse targeted architectures, the optimal epoch for transferability occurs around one or two thirds of training²²2Transferability decreases along epochs, except for the two vision transformers targets on ImageNet where the transferability is stable at the end of training..

We show that early stopping works similarly well on surrogate models trained on robust and non-robust datasets. We retrieve the robust and non-robust datasets from Ilyas et al. [2019], that are altered from CIFAR-10 to mostly contain RFs and, respectively, NRFs. We train two ResNet-50 models on both datasets with SGD (hyperparameters reported in Appendices B and C). Figure 2 shows the transferability across training epochs, averaged over the ten regularly trained targets. The success rates of both robust and non-robust surrogate models evolve similarly to the model trained on the original dataset: transferability peaks around the epochs 50 and 100 and decreases during the following epochs. This observation is valid for all ten targets (Appendix C). According to the RFs/NRFs evolution hypothesis, we expect “X-shaped” transferability curves: if DNNs first mainly learn RFs and then NRFs, the transferability from NRFs would increase and the transferability from RFs would strictly decrease (from early stopping to full training). The RFs/NRFs hypothesis does not describe why early learned NRFs are better to target a regular DNN than fully learned NRFs.

We observe that an early stopped surrogate model trained on the original dataset is best to target both RFs and NRFs targets. Here, we keep the original CIFAR-10 dataset to train the surrogate model. We target four ResNet-50 models trained on the robust and non-robust datasets of Ilyas et al. [2019]³³3In this experiment, we include two additional non-robust datasets $D_{\text{rand}}$ and $D_{\text{det}}$ from Ilyas et al. [2019]. By construction, their only useful features for classification are NRFs. They were excluded from the previous experiment due to training instability.. Figure 3 shows that the same epoch of standard training is optimal for attacking all four models, i.e., composed of either RFs or NRFs. The RFs/NRFs evolution hypothesis fails to explain why early stopping is best to target NRFs.

Overall, we provide new evidence that early stopping for transferability acts similarly on robust and non-robust features. We do not observe an inherent trade-off between RFs and NRFs. Since the higher the transferability, the more similar the representations are, we conclude that the early trained representations are more similar to both RFs and NRFs than their fully trained counterparts. Therefore, the hypothesis that early stopping favors RFs over NRFs does not hold. We conjecture that a phenomenon orthogonal to RFs/NRFs explains why fully trained surrogates are not optimal.

The optimal number of surrogate training epochs for transferability occurs just after the decay of the LR. We train a ResNet-50 surrogate model for 150 epochs on CIFAR-10, using the standard LR schedule of Engstrom et al. [2019] which divides the LR by 10 at epochs 50 and 100. For the ten targets considered individually, the highest transferability is between epochs 51 and 55 (Appendix B). Figure 4 shows that transferability suddenly peaks after both LR decays (red line). We observe the same phenomenon on ImageNet⁴⁴4On ImageNet, we train a ResNet-50 surrogate for 90 epochs with LR decays at epochs 30 and 60. The highest transferability per target occurs either after the first decay (epochs 31 or 35) or after the second one (epochs 62 or 67), except for both vision transformer targets, where transferability plateaus at a low success rate after the second decay.. Overall, the success of early stopping appears to be related to the exploration of the loss landscape, which is governed by the learning rate.

This peak of transferability can be consistently observed at any point of training. Here, we modify the standard double decay LR schedule to perform a single decay at a specified epoch. The learning rate is constant (0.1) until the specified epoch, where it is ten times lower for the rest of the training. We evaluate the transferability of five surrogates with a decay at, respectively, epoch 25, 50, 75, 100 and 125. Figure 4 reports a similar transferability peak for all these surrogates, except for the smaller peak at epoch 25 where the decay occurs before the end of the initial convergence. The consistency of the peak of transferability across training epochs is valid for all individual targets (Appendix D). As a baseline, we use a constant learning rate, under which transferability plateaus without any observed peak. Therefore, we conclude that the step decay of the LR enables early stopping to improve transferability.

When the LR decays, the sharpness in the weight space drops. Figure 5 reports two sharpness metrics per the training epoch of our standard CIFAR-10 surrogate: the largest Hessian eigenvalue measures the sharpness of the sharpest direction in the weight space (red, worst-case sharpness) and the Hessian trace measures the total sharpness of all weight space directions (blue, average sharpness). Both types of sharpness decrease abruptly, significantly and immediately after both LR decays. Simultaneously, transferability peaks (orange).

We conclude that the effect of early stopping on transferability is tightly related to the dynamics of the exploration of the loss surface, governed by the learning rate. Overall, Figure 1 illustrates our observations:

1. 1.

   Before the LR decays, the training bounces back and forth crossing the valley from above (top gray arrows). For an extended discussion, see Appendix D.

2. 2.

   After the LR decays, training goes down the valley. Soon after, SGD has its best transferability (“early stopped SGD” gray star). Sharpness is reduced.

3. 3.

   As training continues, the loss decreases while sharpness slowly increases. SGD settles into a “deep hole” in the loss landscape, with specific representations of low transferability (“fully trained SGD” red star).

The training techniques known to decrease the sharpness of the models train better surrogate representations. We evaluate the transferability of seven training techniques belonging to two families, SWA and SAM (see Section 2 and Appendix A for a more detailed presentation). SWA [Izmailov et al., 2018] decreases sharpness implicitly by averaging the weights collected by SGD. Our SWA surrogate is the average of the weights obtained by our standard SGD surrogate at the end of the last 25% epochs⁵⁵5We also update the batch-normalization statistics of the SWA model with one forward pass over the training data on CIFAR-10 (10% on ImageNet).. Figure 6 shows that SWA (yellow) improves the success rate compared to fully trained SGD (red) on both datasets. On ImageNet, SWA beats the early stopped SGD surrogate, but not on CIFAR-10. Indeed, SWA helps to find flatter solutions than those found by SGD, but SWA is confined to the same basin of attraction [Kaddour et al., 2022]. To remediate to this issue, we also train several surrogate models with SAM [Foret et al., 2020] and its variants, i.e., GSAM [Zhuang et al., 2022], ASAM [Kwon et al., 2021], AGSAM (GSAM+ASAM), WASAM (SAM+SWA, Kaddour et al. [2022]), and LookSAM [Liu et al., 2022]. SAM explicitly minimizes sharpness during training by solving a min-max optimization problem. At each iteration step, SAM first maximizes the loss in a neighborhood to compute a second gradient that is used to minimize the loss (details in Appendix E). We train one model per SAM variant using the original SAM hyperparameter ($\rho=0.05$). Figure 6 shows that SAM and its variants (dotted lines) train surrogate models that have a significantly higher transferability than fully trained SGD, early stopped SGD and SWA, on both datasets. On ImageNet, the success rate of SAM averages over the ten targets at 18.7%, compared to 13.3% for full training with SGD, 14.5% for SGD at its best (epoch 66) and 15.2% for SWA, and respectively 77.3%, 56.6%, 67.7% (epoch 54) and 60.5% on CIFAR-10. SAM finds different basins of attractions than SGD [Foret et al., 2020, Kaddour et al., 2022]. Therefore, some basins of attraction are better surrogate than others, and explicitly minimizing sharpness reaches better ones.

We uncover that the size of the flat neighborhood of SAM and its variants induces a regularization that is tightly linked to transferability. We observe that SAM and its variants with uncommonly large flat neighborhoods train significantly and consistently better surrogate models. SAM seeks neighborhoods with uniformly low loss of size controlled by its $\rho$ hyperparameter (Section 2). We tune it on CIFAR-10 on distinct validation sets of natural examples, target, and surrogate models (details in Appendix E). For all SAM variants, the optimal $\rho$ for transferability is always larger than the original $\rho$, and unusually large compared to the range of values used for natural accuracy. Indeed, we find $\rho$ of 0.3 optimal for SAM, and Foret et al. [2020] originally uses $\rho$ of 0.05. Kaddour et al. [2022] and Zhuang et al. [2022] tune $\rho$ with a maximum of, respectively, 0.2 and 0.3. Figure 6 reports the transferability of SAM and its variants with both the original $\rho$ (dotted) and with the larger $\rho$ found optimal on CIFAR-10 (plain). All SAM variants train a better surrogate model with large $\rho$ values⁶⁶6As expected, LookSAM shows a slower learning behavior over the epoch, compared to other SAM variants: LookSAM is an efficient variant that computes the additional SAM gradient only once each five optimizer iteration.. In the following, we denote l-SAM for SAM with large $\rho$ (0.3), and similarly l-AGSAM and l-LookSAM (respectively, 4⁷⁷7l-AGSAM uses $\rho$ value of 4, since as suggested by Kwon et al. [2021] adaptive variants should use $\rho$ 10 times larger. We found our observations consistent with this recommandation. and 0.3). Kaddour et al. [2022] show that changing $\rho$ ends up in different basins of attraction. Therefore, the stronger regularization induced by l-SAM avoids large sharp holes on top of the loss surface, and significantly improves transferability.

The stronger regularization of SAM with a large value of $\rho$ is specifically related to transferability. First, this strength of regularization may degrade natural accuracy. On ImageNet with ResNet-18, the top-1 accuracy of SAM with large $\rho$ is equal to 67.89%, less than SAM with the original $\rho$ (70.29%) and even less than fully trained SGD (69.84%). This observation extends to ResNet-50 and to the other variants of SAM on ImageNet (Appendix E). Therefore, the improvement in generalization of adversarial examples cannot be explained by an improvement in natural generalization (better fit to the data). Second, unlike SAM, a stronger regularization of weight decay decreases transferability, showing a specific relation between transferability and SAM. We train multiple surrogate models using SGD with different values of weight decay. The optimal weight decay value for the ResNet-50 surrogate is the same value used to train the target model (see Appendix F for details). Therefore, not all regularization schemes help to train a better surrogate model.

Overall, the sharpness of the surrogate model is tightly related to transferability:

• •

  Minimizing implicitly or explicitly the loss sharpness trains better surrogate models.

• •

  The strong SAM regularization avoids deep sharp minima in favor of unusually large flat neighborhoods that contain more generic representations.

• •

  The stronger SAM regularization is tailored for transferability: it can reduce natural accuracy, and other strong regularization schemes, such as weight decay, do not aid in transferability.

l-SAM and l-AGSAM are competitive alternatives to existing surrogate training techniques, and l-LookSAM offers good transferability for a small computational overhead. For a fair comparison, we choose the epoch of the early stopped SGD surrogate by evaluating a validation transferability at every training epoch⁹⁹9To avoid data leakage that violates our no-query threat model, we craft one thousand adversarial examples from images of a validation set and evaluate them on a distinct set of target models.. We retrieve the SAT (Slight Adversarial Training) ImageNet weights used by Springer et al. [2021], and we train SAT on CIFAR-10 using their hyperparameters. Table 3 reports the average success rate of the aforementioned techniques, alongside their computational overhead. This overhead is the ratio of forward-backward passes needed by the training technique to those required by SGD. On both datasets, l-AGSAM is the best surrogate. l-AGSAM beats the transferability of SAT, while dividing the training cost by two on ImageNet and four on CIFAR-10. Nevertheless, l-AGSAM doubles the computational number of forward-backward passes compared to SGD. By computing the additional SAM gradient only once per five iterations, l-LookSAM is a viable alternative to contain the computational overhead to 1.23, while having higher transferability than SGD. Overall, sharpness-aware minimizers with large flat neighborhoods offer a good trade-off between transferability and computation.

l-SAM is a good base model to combine with existing model augmentation, data augmentation, and attack optimization transferability techniques. These categories aim complementary objectives: model and data augmentations reduce the tendency of the attack to overfit the base model by adding randomness to gradients. Attack optimizers intend to smooth the gradient updates. Table 4 reports the success rate of nine transferability techniques combined with our l-SAM base model on ImageNet. For all perturbation norms $\varepsilon$, l-SAM provides a base model that improves every nine techniques, compared to the standard fully trained SGD surrogate, from 0.4 to 35.5 percentage points. RAP [Qin et al., 2022]) is particularly interesting, since RAP minimizes sharpness like SAM but in the input space. SAM and RAP are best combined, indicating their complementary effects on two distinct factors: SAM finds generic representations, while RAP finds adversarial examples that do not overfit a single representation.