Adversarial Example Researchers Need to Expand What is Meant by ‘Robustness’

The hypothesis in Ilyas et. al. is a special case of a more general principle that is well accepted in the distributional robustness literature — models lack robustness to distribution shift because they latch onto superficial correlations in the data. Naturally, the same principle also explains adversarial examples because they arise from a worst-case analysis of distribution shift. To obtain a more complete understanding of robustness, adversarial example researchers should connect their work to the more general problem of distributional robustness rather than remaining solely fixated on small gradient perturbations.

Detailed Response

The main hypothesis in Ilyas et al. (2019) happens to be a special case of a more general principle that is commonly accepted in the robustness to distributional shift literature : a model’s lack of robustness is largely because the model latches onto superficial statistics in the data. In the image domain, these statistics may be unused by — and unintuitive to — humans, yet they may be useful for generalization in i.i.d. settings. Separate experiments eschewing gradient perturbations and studying robustness beyond adversarial perturbations show similar results. For example, a recent work demonstrates that models can generalize to the test examples by learning from high-frequency information that is both naturally occurring and also inconspicuous. Concretely, models were trained and tested with an extreme high-pass filter applied to the data. The resulting high-frequency features appear completely grayscale to humans, yet models are able to achieve 50% top-1 accuracy on ImageNet-1K solely from these natural features that usually are “invisible.” These hard-to-notice features can be made conspicuous by normalizing the filtered image to have unit variance pixel statistics in the figure below.

Given the plethora of useful correlations that exist in natural data, we should expect that our models will learn to exploit them. However, models relying on superficial statistics can poorly generalize should these same statistics become corrupted after deployment. To obtain a more complete understanding of model robustness, measured test error after perturbing every image in the test set by a Fourier basis vector, as shown in Figure 2. The naturally trained model is robust to low-frequency perturbations, but, interestingly, lacks robustness in the mid to high frequencies. In contrast, adversarial training improves robustness to mid- and high-frequency perturbations, while sacrificing performance on low frequency perturbations. For instance adversarial training degrades performance on the low-frequency fog corruption from 85.7% to 55.3%. Adversarial training similarly degrades robustness to contrast and low-pass filtered noise. By taking a broader view of robustness beyond tiny $\ell_p$ norm perturbations, we discover that adversarially trained models are actually not “robust.” They are instead biased towards different kinds of superficial statistics. As a result, adversarial training can sacrifice robustness in real-world settings.

How, then, can the research community create models that robustly generalize in the real world, given that adversarial training can harm robustness to distributional shift? To do so, the research community must take a broader view of robustness and accept that $\ell_p$ adversarial robustness is highly limited and mostly detached from security and real-world robustness . While often thought an idiosyncratic quirk of deep neural network classifiers, adversarial examples are not a counterintuitive mystery plaguing otherwise superhuman classifiers. Instead, adversarial examples are in fact expected of models which lack robustness to noise . They should not be surprising given the brittleness observed in numerous synthetic — and even natural  — conditions. Models reliably exhibit poor performance when they are evaluated on distributions slightly different from the training distribution. For all that, current benchmarks do not expose these failure modes. The upshot is that we need to design harder and more diverse test sets, and we should not continue to be singularly fixated on studying specific gradient perturbations. As we move forward in robustness research, we should focus on the various ways in which models are fragile, and design more comprehensive benchmarks accordingly . As long as models lack robustness to distributional shift, there will always be errors to find adversarially.