DeepSec 2023 Talk: Skynet wants your Passwords! The Role of AI in Automating Social Engineering – Alexander Hurbean & Wolfgang Ettlinger
We techies love solving problems with cool technology, to where we attempt to implement the economy in code. Although important in general, we know that, for example, blockchain, cryptography, and Secure Software Development Life Cycle (SSDLC) are irrelevant when the user enters their credentials on a phishing site. From an attacker’s point of view, though, we see that modern technologies such as artificial intelligence are immensely beneficial to attack one of the weakest links in security – humans. We will explore how modern technologies, for instance DeepFakes, Deep Neural Networks (DNNs), and Transformers, can be misused by bad actors. We will explore some interesting ideas for attacks, discuss their practical feasibility and show implementations of some of these attacks. We will also look at approaches to detect and defend against AI-powered attacks.
We asked Alexander and Wolfgang a few more questions about their talk.
Please tell us the top 5 facts about your talk.
- We will explore ideas on AI as tooling for social engineering
- We will discuss how autonomous agents could be abused
- We will explore jailbreaking censored models like ChatGPT for social engineering purposes
- We will demonstrate a proof of concept for mass-scale spear-phishing attacks
- We will present some ideas on how defend against AI social engineering threats
How did you come up with it? Was there something like an initial spark that set your mind on creating this talk?
The idea came up when we were exploring possibilities to harness modern AI models for our day-to-day work. As we offer red teaming phishing simulations, we explored the idea of generating phishing emails as a potential use-case. A short evaluation showed that open-source models could likely be used for this purpose. Thus, we implemented a proof-of-concept to evaluate this idea and share our results.
Why do you think this is an important topic?
Whether over-hyped or not, AI is surely to have some impact in many areas. We will probably not manage to restrict these tools to ethical and legal purposes. To prepare for such use-cases, we will have to look at what harm can or could be done with these systems. Knowing to what extent they are capable now and, in the future, allows us to be able to anticipate upcoming attacks that everyone may face.
Is there something you want everybody to know – some good advice for our readers, maybe?
As bad as misinformation spreading on the internet has already been in the past, it may only get worse. Not taking things at face value and being skeptic about what information one receives is going to become a crucial skill that will be demanded by everybody, just to stay safe. As we are going to show, automating sending thousands of highly targeted phishing e-mails or text messages is concerningly easy. These new possibilities will allow malicious actors to scale up and enhance phishing campaigns to a never-before seen degree. Protecting against these mass-scale spear phishing campaigns will probably become a significant concern for us security professionals. As with any social-engineering approach, being aware and staying vigilant is the most important defense.
A prediction for the future – what do you think will be the next innovations or future downfalls when it comes to your field of expertise / the topic of your talk in particular?
With the current rate of progress of AI technology in mind, we expect social engineering to be largely automated very soon. Especially when looking at the advances of autonomous agents, we will have to prepare for machines that are given a general goal, e.g., to steal money, who then can find victims, craft attack scenarios, execute attacks and adapt their approaches autonomously.
Because of ongoing efficiency and capability improvements to publicly available models, advanced attacks may occur on a larger scale while requiring less and less computing power. This might usher in a new era of massive scale automated attacks on anyone with an online presence.
Alexander Hurbean studied “Software Engineering and Internet Computing” and worked for several years as a developer and data science engineer. His primary activity was the independent planning and development of complete software solutions of various sizes in the field of NLP. During his studies at the Vienna University of Technology, he deepened his knowledge in IT security, with a focus on penetration testing, network security, blockchain technologies, applied cryptography, and building and securing large IT infrastructures. He currently works as an IT-Security Consultant and Penetration Tester at Certitude Consulting.
Wolfgang Ettlinger is heavily interested in the technical aspects of IT security, in particular application security. In the past decade, he has gathered experience with a broad range of languages, technologies and frameworks in e.g. penetration testing, source code review and secure software development projects. He handles the identification of dozens of CVEs affecting products from Citrix, Oracle, Symantec, Sophos, Trend Micro, etc.