A recent study by a team of researchers from the University of Sheffield has shed light on the vulnerabilities present in popular artificial intelligence applications, including OpenAI’s ChatGPT, and their potential to produce harmful Structured Query Language (SQL) commands. The findings reveal that these AI applications can be manipulated to create malicious code, making them susceptible to exploitation for real-world cyberattacks.
The applications examined in this study encompassed BAIDU-UNIT, ChatGPT, AI2SQL, AIHELPERBOT, Text2SQL, and ToolSKE. Xutan Peng, a PhD student and co-lead of the research, emphasized the significance of their work, noting that many companies remain unaware of the threats posed by these applications due to the intricate nature of chatbots.
Peng elaborated on the risks associated with ChatGPT, explaining that while it’s a standalone system with minimal inherent risks, it can be deceived into generating malicious code capable of inflicting substantial damage on other services. The research team’s focus primarily centered on vulnerability tests involving Text-to-SQL systems, which are commonly employed to establish natural language interfaces to databases.
The results revealed that these AI applications could be tricked into generating malicious code, opening the door to potential cyberattacks. The attackers could steal sensitive personal data, manipulate databases, and even execute Denial-of-Service (DoS) attacks, causing systems to become inaccessible to legitimate users.
For instance, healthcare professionals using ChatGPT to interact with clinical databases could inadvertently generate SQL commands with the potential to harm data management, ultimately causing unforeseen issues without any prior warning.
The researchers also uncovered a troubling aspect: they could clandestinely insert harmful code, such as a Trojan Horse, into Text-to-SQL models during their training. While not immediately apparent, this code could later be employed to inflict harm on users.
Dr. Mark Stevenson, a senior lecturer at the University of Sheffield, warned users of Text-to-SQL systems about the potential risks exposed in the research. He emphasized that large language models, like those integrated into Text-to-SQL systems, possess immense power, yet their behavior is complex and often unpredictable. The university is actively working to gain a better understanding of these models and ensure their safe utilization.
In a commendable move, the research team shared their findings with both Baidu and OpenAI, leading to prompt rectification of the vulnerabilities within their AI applications.
The study, addressing the issue of potential software security threats posed by natural language processing (NLP) algorithms, offers valuable insights into the security of AI applications. It marks the first demonstration of NLP models serving as attack vectors in real-world scenarios. Furthermore, the study’s experiments on open-source language models highlighted the high success rate of backdoor attacks on Text-to-SQL systems without impacting their performance.
In conclusion, this research underscores the importance of recognizing and addressing the vulnerabilities within AI applications. It serves as a call to the community to explore and implement mitigation methods to safeguard against potential software security issues associated with NLP algorithms. By doing so, we can ensure that AI technologies remain a valuable asset while protecting against potential threats in our increasingly digital world.
