Science Fictions meets Large Language Models
Given the advertising of the manufacturers using a Large Language Model (LLM) is just like having a conversation with a person. The reality looks different. Google’s AI search has recently recommended to glue pizza together, eats rocks, or jump off the Golden Gate bridge when being depressed. This is clearly bad advice. Apparently these answers are or were part of the learning process. Incidents like this and the hallucinations of LLM algorithms have been discussed already. Science fictions fans will recall conversations with computer or AI simulations where someone tries to trick the machine do override security checks. The Open Worldwide Application Security Project (OWASP) created a list of threats to LLM applications. The target audience are developers, designers, architects, managers, and organizations.
Marc Pesce wrote an article about tests with different LLM implementations. He found ways to force the application to spit out garbage. The tests were performed against a selection of chat bots. Every single time there was a prompt that could break the conversation and make the chat bot reply with nonsense. The problem is still being investigated by vendors, but it looks like a structural problem in the algorithms. Debugging effects like this is hard. LLMs are not simple pieces of code. Furthermore, you need to take the learning data into consideration.
Using less data during the learning phase may create a better filter system. Unfortunately, the programming assistants do not fare any better. A study fed 517 questions from a tech forum to ChatGPT. 52% of the answer contained misinformation. That make the tool unsuitable as an assistant. Integrated development environments need to be fast and accurate. Looking up answers that you still need to fact check is already part of the current process. This doesn’t require a new tool.
We are hoping to see your LLM exploits and experiences at DeepSec in November. See you!