ABSTRACT
Given their generalization capabilities,deep learning algorithms may represent a powerful weapon in the arsenal of antivirus developers. Nevertheless, recent works in different domains (e.g., computer vision) have shown that such algorithms are susceptible to backdooring attacks, namely training-time attacks that aim toteach a deep neural network to misclassify inputs containing a specific trigger. This work investigates the resilience of deep learning models for malware detection against backdooring attacks. In particular, we devise two classes of attacks for backdooring a malware detector that targets the update process of the underlying deep learning classifier. While the first and most straightforward approach relies onsuperficial triggers made of static byte sequences, the second attack we propose employslatent triggers, namely specific feature configurations in the latent space of the model. The latent triggers may be produced by different byte sequences in the binary inputs, rendering the triggerdynamic in the input space and thus more challenging to detect.
We evaluate the resilience of two state-of-the-art convolutional neural networks for malware detection against both strategies and under different threat models. Our results indicate that the models do not easily learn superficial triggers in aclean label setting, even when allowing a high rate (\geq 30%) of poisoning samples. Conversely, an attacker manipulating the training labels (\textitdirty label attack) can implant an effective backdoor that activates with a superficial, static trigger into both models. The results obtained from the experimental evaluation carried out on the latent trigger attack instead show that the knowledge of the adversary on the target classifier may influence the success of the attack. Assuming perfect knowledge, an attacker can implant a backdoor that perfectly activates in 100% of the cases with a poisoning rate as low as 0.1% of the whole updating dataset (namely, 32 poisoning samples in a dataset of 32000 elements).
Lastly, we experiment with two known defensive techniques that were shown effective against other backdooring attacks in the malware domain. However, none proved reliable in detecting the backdoor or triggered samples created by our latent space attack. We then discuss some modifications to those techniques that may render them effective against latent backdooring attacks.
Supplemental Material
- 2022. Avast. https://www.avast.com/technology/ai-and-machine-learning. [Online; accessed Dec 2nd, 2022].Google Scholar
- 2022. Chocolatey. https://chocolatey.org/.Google Scholar
- 2022. Eset. https://www.eset.com/uk/home/nod32-antivirus/. [Online; accessed Dec 2nd, 2022]Google Scholar
- 2022. Kaspersky. https://www.kaspersky.com/enterprise-security/wiki-section/ products/machine-learning-in-cybersecurity. [Online; accessed Dec 2nd, 2022].Google Scholar
- 2022. VirusShare. https://virusshare.com/.Google Scholar
- 2022. VirusTotal. https://virustotal.com/.Google Scholar
- T. Abou-Assaleh, N. Cercone, V. Keselj, and R. Sweidan. 2004. N-gram-based detection of new malicious code. In Proceedings of the 28th Annual International Computer Software and Applications Conference, 2004. COMPSAC 2004., Vol. 2. 41--42 vol.2. https://doi.org/10.1109/CMPSAC.2004.1342667Google ScholarCross Ref
- Hojjat Aghakhani, Fabio Gritti, Francesco Mecca, Martina Lindorfer, Stefano Ortolani, Davide Balzarotti, Giovanni Vigna, and Christopher Kruegel. 2020. When malware is Packin'Heat; limits of machine learning classifiers based on static analysis features. In Network and Distributed Systems Security (NDSS) Symposium 2020.Google Scholar
- Hyrum S Anderson, Anant Kharkar, Bobby Filar, and Phil Roth. 2017. Evading machine learning malware detection. black Hat 2017 (2017).Google Scholar
- Hyrum S Anderson and Phil Roth. 2018. Ember: an open dataset for training static pe malware machine learning models. arXiv preprint arXiv:1804.04637 (2018).Google Scholar
- Daniel Arp, Erwin Quiring, Feargus Pendlebury, Alexander Warnecke, Fabio Pierazzi, Christian Wressnegger, Lorenzo Cavallaro, and Konrad Rieck. 2022. Dos and Don'ts of Machine Learning in Computer Security. In 31st USENIX Security Symposium (USENIX Security 22). 3971--3988.Google Scholar
- Daniel Arp, Michael Spreitzenbarth, Malte Hubner, Hugo Gascon, Konrad Rieck, and CERT Siemens. 2014. Drebin: Effective and explainable detection of android malware in your pocket.. In Ndss, Vol. 14. 23--26.Google Scholar
- Niket Bhodia, Pratikkumar Prajapati, Fabio Di Troia, and Mark Stamp. 2019. Transfer learning for image-based malware classification. arXiv preprint arXiv:1903.11551 (2019).Google Scholar
- Brittain Blake. 2022. Apple lawsuit says 'stealth' startup Rivos poached engineers to steal secrets. https://www.reuters.com/legal/litigation/apple-lawsuit-saysstealth-startup-rivos-poached-engineers-steal-secrets-2022-05-02/. [Online; accessed Dec 2nd, 2022].Google Scholar
- Michele Carminati, Luca Santini, Mario Polino, and Stefano Zanero. 2020. Evasion Attacks against Banking Fraud Detection Systems. In 23rd International Symposium on Research in Attacks, Intrusions and Defenses, RAID 2020, San Sebastian, Spain, October 14--15, 2020, Manuel Egele and Leyla Bilge (Eds.). USENIX Association, 285--300. https://www.usenix.org/conference/raid2020/presentation/ carminatiGoogle Scholar
- Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. 2018. Detecting backdoor attacks on deep neural networks by activation clustering. arXiv preprint arXiv:1811.03728 (2018).Google Scholar
- Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. 2017. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526 (2017).Google Scholar
- Xiaoyi Chen, Ahmed Salem, Michael Backes, Shiqing Ma, and Yang Zhang. 2021. Badnl: Backdoor attacks against nlp models. In ICML 2021 Workshop on Adversarial Machine Learning.Google Scholar
- Mario D'Onghia, Matteo Salvadore, Benedetto Maria Nespoli, Michele Carminati, Mario Polino, and Stefano Zanero. 2022. Apícula: Static Detection of API Calls in Generic Streams of Bytes. Computers & Security (2022), 102775.Google Scholar
- Mohammadreza Ebrahimi, Ning Zhang, James Hu, Muhammad Taqi Raza, and Hsinchun Chen. 2020. Binary black-box evasion attacks against deep learningbased static malware detectors with adversarial byte-level language model. arXiv preprint arXiv:2012.07994 (2020).Google Scholar
- Alessandro Erba, Riccardo Taormina, Stefano Galelli, Marcello Pogliani, Michele Carminati, Stefano Zanero, and Nils Ole Tippenhauer. 2020. Constrained Concealment Attacks against Reconstruction-based Anomaly Detectors in Industrial Control Systems. In ACSAC '20: Annual Computer Security Applications Conference, Virtual Event / Austin, TX, USA, 7--11 December, 2020. ACM, 480--495. https://doi.org/10.1145/3427228.3427660Google ScholarDigital Library
- Mojtaba Eskandari and Sattar Hashemi. 2011. Metamorphic malware detection using control flow graph mining. Int. J. Comput. Sci. Netw. Secur 11, 12 (2011), 1--6.Google Scholar
- Zhang Fuyong and Zhao Tiezhu. 2017. Malware Detection and Classification Based on N-Grams Attribute Similarity. In 2017 IEEE International Conference on Computational Science and Engineering (CSE) and IEEE International Conference on Embedded and Ubiquitous Computing (EUC), Vol. 1. 793--796. https://doi.org/ 10.1109/CSE-EUC.2017.157Google ScholarCross Ref
- Nicola Galloro, Mario Polino, Michele Carminati, Andrea Continella, and Stefano Zanero. 2022. A Systematical and longitudinal study of evasive behaviors in windows malware. Computers & Security 113 (2022), 102550.Google ScholarDigital Library
- Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C Ranasinghe, and Surya Nepal. 2019. Strip: A defence against trojan attacks on deep neural networks. In Proceedings of the 35th Annual Computer Security Applications Conference. 113--125.Google ScholarDigital Library
- Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep learning. MIT press.Google ScholarDigital Library
- Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014).Google Scholar
- Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. 2017. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733 (2017).Google Scholar
- Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. 2019. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access 7 (2019), 47230--47244.Google ScholarCross Ref
- Sanghyun Hong, Nicholas Carlini, and Alexey Kurakin. 2021. Handcrafted Backdoors in Deep Neural Networks. CoRR abs/2106.04690 (2021). arXiv:2106.04690 https://arxiv.org/abs/2106.04690Google Scholar
- Ling Huang, Anthony D Joseph, Blaine Nelson, Benjamin IP Rubinstein, and J Doug Tygar. 2011. Adversarial machine learning. In Proceedings of the 4th ACM workshop on Security and artificial intelligence. 43--58.Google ScholarDigital Library
- Roberto Jordaney, Kumar Sharad, Santanu K Dash, Zhi Wang, Davide Papini, Ilia Nouretdinov, and Lorenzo Cavallaro. 2017. Transcend: Detecting concept drift in malware classification models. In 26th USENIX Security Symposium (USENIX Security 17). 625--642.Google ScholarDigital Library
- Kesav Kancherla and Srinivas Mukkamala. 2013. Image visualization based malware detection. In 2013 IEEE Symposium on Computational Intelligence in Cyber Security (CICS). 40--44. https://doi.org/10.1109/CICYBS.2013.6597204Google ScholarCross Ref
- Jeffrey O. Kephart, Gregory B. Sorkin, William C. Arnold, David M. Chess, Gerald J. Tesauro, and Steve R. White. 1995. Biologically Inspired Defenses against Computer Viruses. In Proceedings of the 14th International Joint Conference on Artificial Intelligence - Volume 1 (Montreal, Quebec, Canada) (IJCAI'95). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 985--996.Google Scholar
- Bojan Kolosnjaji, Ambra Demontis, Battista Biggio, Davide Maiorca, Giorgio Giacinto, Claudia Eckert, and Fabio Roli. 2018. Adversarial malware binaries: Evading deep learning for malware detection in executables. In 2018 26th European signal processing conference (EUSIPCO). IEEE, 533--537.Google Scholar
- Jeremy Z. Kolter and Marcus A. Maloof. 2004. Learning to Detect Malicious Executables in the Wild. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (Seattle, WA, USA) (KDD '04). Association for Computing Machinery, New York, NY, USA, 470--478. https: //doi.org/10.1145/1014052.1014105Google ScholarDigital Library
- Marek Krcál, Martin Bálek, and Otakar Jasek. 2018. Deep Convolutional Malware Classifiers Can Learn from Raw Executables and Labels Only. In ICLR.Google Scholar
- Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. 2018. Adversarial examples in the physical world. In Artificial intelligence safety and security. Chapman and Hall/CRC, 99--112.Google Scholar
- Junyu Lin, Lei Xu, Yingqi Liu, and Xiangyu Zhang. 2020. Composite backdoor attack for deep neural network by mixing existing benign features. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security. 113--131.Google ScholarDigital Library
- Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. 2018. Fine-pruning: Defending against backdooring attacks on deep neural networks. In International Symposium on Research in Attacks, Intrusions, and Defenses. Springer, 273--294.Google ScholarCross Ref
- Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. 2017. Trojaning attack on neural networks. (2017).Google Scholar
- Keane Lucas, Mahmood Sharif, Lujo Bauer, Michael K Reiter, and Saurabh Shintre. 2021. Malware Makeover: breaking ML-based static analysis by modifying executable bytes. In Proceedings of the 2021 ACM Asia Conference on Computer and Communications Security. 744--758.Google ScholarDigital Library
- Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2017. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 (2017).Google Scholar
- Tajuddin Manhar Mohammed, Lakshmanan Nataraj, Satish Chikkagoudar, Shivkumar Chandrasekaran, and BS Manjunath. 2021. Malware detection using frequency domain-based image visualization and deep learning. arXiv preprint arXiv:2101.10578 (2021).Google Scholar
- Robert Moskovitch, Dima Stopel, Clint Feher, Nir Nissim, and Yuval Elovici. 2008. Unknown malcode detection via text categorization and the imbalance problem. In 2008 IEEE International Conference on Intelligence and Security Informatics. 156--161. https://doi.org/10.1109/ISI.2008.4565046Google ScholarCross Ref
- United States Attorney's Office. 2020. Former Uber Executive Sentenced To 18 Months In Jail For Trade Secret Theft From Google. https://www.justice.gov/usao-ndca/pr/former-uber-executive-sentenced18-months-jail-trade-secret-theft-google. [Online; accessed Dec 2nd, 2022].Google Scholar
- United States Attorney's Office. 2021. Fraudster Sentenced to Prison for Long Running Phone Unlocking Scheme that Defrauded AT&T. https://www.justice.gov/opa/pr/fraudster-sentenced-prison-long-runningphone-unlocking-scheme-defrauded-att. [Online; accessed Dec 2nd, 2022].Google Scholar
- Feargus Pendlebury, Fabio Pierazzi, Roberto Jordaney, Johannes Kinder, and Lorenzo Cavallaro. 2019. {TESSERACT}: Eliminating experimental bias in malware classification across space and time. In 28th USENIX Security Symposium (USENIX Security 19). 729--746.Google Scholar
- Fabio Pierazzi, Feargus Pendlebury, Jacopo Cortellazzi, and Lorenzo Cavallaro. 2020. Intriguing Properties of Adversarial ML Attacks in the Problem Space. In 2020 IEEE Symposium on Security and Privacy (SP). 1332--1349. https://doi.org/10. 1109/SP40000.2020.00073Google Scholar
- Xiangyu Qi, Jifeng Zhu, Chulin Xie, and Yong Yang. 2021. Subnet Replacement: Deployment-stage backdoor attack against deep neural networks in gray-box setting. CoRR abs/2107.07240 (2021). arXiv:2107.07240 https://arxiv.org/abs/2107. 07240Google Scholar
- Edward Raff, Jon Barker, Jared Sylvester, Robert Brandon, Bryan Catanzaro, and Charles K Nicholas. 2018. Malware detection by eating a whole exe. In Workshops at the Thirty-Second AAAI Conference on Artificial Intelligence.Google Scholar
- Edward Raff, William Fleshman, Richard Zak, Hyrum S Anderson, Bobby Filar, and Mark McLean. 2021. Classifying sequences of extreme length with constant memory applied to malware detection. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 9386--9394.Google ScholarCross Ref
- Edward Raff, Richard Zak, Russell Cox, Jared Sylvester, Paul Yacci, Rebecca Ward, Anna Tracy, Mark McLean, and Charles Nicholas. 2018. An investigation of byte n-gram features for malware classification. Journal of Computer Virology and Hacking Techniques 14, 1 (2018), 1--20.Google ScholarCross Ref
- Ahmed Salem, Rui Wen, Michael Backes, Shiqing Ma, and Yang Zhang. 2022. Dynamic backdoor attacks against machine learning models. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P). IEEE, 703--718.Google ScholarCross Ref
- Ashkan Sami, Babak Yadegari, Hossein Rahimi, Naser Peiravian, Sattar Hashemi, and Ali Hamze. 2010. Malware detection based on mining API calls. In Proceedings of the 2010 ACM symposium on applied computing. 1020--1025.Google ScholarDigital Library
- Igor Santos, Felix Brezo, Javier Nieves, Yoseba K Penya, Borja Sanz, Carlos Laorden, and Pablo G Bringas. 2010. Idea: Opcode-sequence-based malware detection. In International Symposium on Engineering Secure Software and Systems. Springer, 35--43.Google ScholarDigital Library
- Giorgio Severi, Jim Meyer, Scott Coull, and Alina Oprea. 2021. {ExplanationGuided} Backdoor Poisoning Attacks Against Malware Classifiers. In 30th USENIX Security Symposium (USENIX Security 21). 1487--1504.Google Scholar
- Asaf Shabtai, Robert Moskovitch, Clint Feher, Shlomi Dolev, and Yuval Elovici. 2012. Detecting unknown malicious code by applying classification techniques on opcode patterns. Security Informatics 1, 1 (2012), 1--22.Google ScholarCross Ref
- Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K. Reiter. 2016. Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition. In Proceedings of the 23rd ACM SIGSAC Conference on Computer and Communications Security. https://doi.org/10.1145/2976749.2978392Google ScholarDigital Library
- Pramila P Shinde and Seema Shah. 2018. A review of machine learning and deep learning applications. In 2018 Fourth international conference on computing communication control and automation (ICCUBEA). IEEE, 1--6.Google ScholarCross Ref
- Charles Smutz and Angelos Stavrou. 2012. Malicious PDF Detection Using Metadata and Structural Features. In Proceedings of the 28th Annual Computer Security Applications Conference (Orlando, Florida, USA) (ACSAC '12). Association for Computing Machinery, New York, NY, USA, 239--248. https://doi.org/10. 1145/2420950.2420987Google ScholarDigital Library
- P Vinod, R Jaipur, V Laxmi, and M Gaur. 2009. Survey on malware detection methods. In Proceedings of the 3rd Hackers' Workshop on computer and internet security (IITKHACK'09). 74--79.Google Scholar
- Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y. Zhao. 2019. Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks. In 2019 IEEE Symposium on Security and Privacy (SP). 707--723. https://doi.org/10.1109/SP.2019.00031Google ScholarCross Ref
- Xiaojun Xu, Qi Wang, Huichen Li, Nikita Borisov, Carl A Gunter, and Bo Li. 2021. Detecting ai trojans using meta neural analysis. In 2021 IEEE Symposium on Security and Privacy (SP). IEEE, 103--120.Google ScholarCross Ref
- Limin Yang, Zhi Chen, Jacopo Cortellazzi, Feargus Pendlebury, Kevin Tu, Fabio Pierazzi, Lorenzo Cavallaro, and Gang Wang. 2022. Jigsaw Puzzle: Selective Backdoor Attack to Subvert Malware Classifiers. arXiv preprint arXiv:2202.05470 (2022).Google Scholar
- Yuanshun Yao, Huiying Li, Haitao Zheng, and Ben Y Zhao. 2019. Latent backdoor attacks on deep neural networks. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security. 2041--2055.Google ScholarDigital Library
- Richard Zak, Edward Raff, and Charles Nicholas. 2017. What can N-grams learn for malware detection?. In 2017 12th International Conference on Malicious and Unwanted Software (MALWARE). IEEE, 109--118.Google ScholarCross Ref
Index Terms
- Lookin' Out My Backdoor! Investigating Backdooring Attacks Against DL-driven Malware Detectors
Recommendations
Countermeasures Against Backdoor Attacks Towards Malware Detectors
Cryptology and Network SecurityAbstractAttacks on machine learning systems have been systematized as adversarial machine learning, and a variety of attack algorithms have been studied until today. In the malware classification problem, several papers have suggested the possibility of ...
A novel method for improving the robustness of deep learning-based malware detectors against adversarial attacks
AbstractMalware is constantly evolving with rising concern for cyberspace. Deep learning-based malware detectors are being used as a potential solution. However, these detectors are vulnerable to adversarial attacks. The adversarial attacks manipulate ...
Graphical abstractDisplay Omitted
Highlights- An approach to combining adversarial attacks is proposed to analyse the robustness of malware detectors against attacks.
- Ten adversarial attacks are created to generate binary-encoded malicious samples, including the proposed combined ...
Adversarial attacks against Windows PE malware detection: A survey of the state-of-the-art
AbstractMalware has been one of the most damaging threats to computers that span across multiple operating systems and various file formats. To defend against ever-increasing and ever-evolving malware, tremendous efforts have been made to propose a ...
Comments