Artificial intelligence (AI) has become much more than the latest high-tech buzzword. It has entered the mainstream, offering greater opportunities but with it new challenges. This month, the Government Accountability Office (GAO) warned that deploying AI into the wild could “make critical infrastructure systems that support the nation’s essential functions, such as supplying water, generating electricity, and producing food, more vulnerable.”
The watchdog group called for the Department of Homeland Security (DHS) to update its guidance and template for AI risk assessments to address gaps that could still pose potential threats.
However, it isn’t just at the federal level where AI must be used cautiously.
Misguided Assumptions and Unrealistic Expectations Remain
David Brauchler, technical director at consulting firm NCC Group, told ClearanceJobs that AI and Large Language Models (LLMs) have become a blind spot for many chief information security officers (CISOs), who are still on the proverbial hook when it comes to protecting their organizations.
“In the rush to adopt AI and Large Language Models (LLM) across various business functions, security analysis shows that many organizations have gotten ahead of themselves,” Brauchler explained. “While many are aware of the potential threats and vulnerabilities inherent in GenAI, they’re unaware of or unsure how to address them.”
This problem typically arises for two reasons: “Misguided Assumptions” and “Unrealistic Expectations”
In the first cast, Brauchler said AI and LLM models produce a lot of data, but not all of it is trustworthy.
“Developers architect systems based on the assumption that the model they built will behave exactly the way they intend or expect,” Brauchler suggested. “They don’t anticipate that it can be manipulated by opportunistic bad actors. And it’s not just the model itself at risk—manipulating even a small piece of data the model looks at can create vulnerabilities. Worse yet, the model doesn’t know it’s been manipulated, so it easily becomes an agent of the attacker.”
In regards to the great expectations; it comes about because of the incredible potential that users believe AI has, yet, too often teams still believe AI and LLMs are capable more than they are.
“At their core, LLMs are nothing more than language pattern completion engines—they leap to the next logical word based on the patterns they’ve previously seen. They lack contextual awareness and can easily ‘lose the plot’ without users or the models themselves realizing it—a risk that developers can’t control,” warned Brauchler.
Building Security Into AI Apps
To overcome these issues, CISOs should look to build security into AI applications by design, and mitigate risk—while still accelerating the adoption of LLMs for competitive advantage. These should not be mutually exclusive concepts.
To accommodate this, CISOs should recognize that pollution moves downstream.
“LLMs access documents, prompts, and images from other models or untrusted sources, which means they can be exposed to data beyond what the developers specified. If an LLM accesses a resource that contains malicious information, the model can be influenced to move that data downstream, where it poisons the output,” Brauchler further noted.
He told ClearanceJobs that when that happens, users and organizations can no longer trust the output of the model.
“It’s difficult to sufficiently sanitize that data and understanding and building security protocols based on this fundamental feature are essential,” Brauchler added.
Models as Threat Actors
Adopting AI and LLM will also require that CISOs pay greater attention to Models as Threat Actors (MATAs) since models often are provided access to vast amounts of data—and if not carefully controlled, these models could be used to extract sensitive information or generate malicious content like phishing emails, deepfakes, or harmful code.
“When an attacker can manipulate the model, they’re in control. It’s the equivalent of being positioned within the architecture,” emphasized Brauchler.
In this case, the model itself becomes a threat actor because it’s trained to perform the prescribed actions with an ineffective sensibility for whether they’re right or wrong.
“Depending on the functionality exposed to the model, it can execute commands and/or extract all the resources it’s trained on—which could be your proprietary data,” Brauchler continued. “If a threat actor writes malicious data into documents the model reads, and a user asks for a summary of those documents, the model can act on that embedded instruction, like executable code.”
The Need For Data-code Separation
Brauchler also told ClearanceJobs that the most prudent approach to model security is one that CISOs are probably already familiar with—namely creating a gatekeeper code separation that functions similarly to a firewall so that trusted and untrusted models and data never touch or interact directly.
The gatekeeper can be tasked with operating between two LLMs. This includes a code-facing trusted model and a data-facing untrusted model. It manages the interaction between the two so that untrusted data can’t command any code functions.
“When queries are made, the gatekeeper ties the two together and returns the filtered, curated output. This way, the code-facing model is never exposed to the contents of poisoned data, which mitigates the output of the attack chain,” said Brauchler.
Good Data Management Practices
Finally, CISOs should continue the proven good data management practices.
“Even quarantined data can’t be easily cleaned—once it’s compromised, it’s nearly impossible to extract errant or malicious input. Instead, it’s best to make sure all content is tagged and masked so quarantined data is always obscured from the LLM, preventing it from executing any instructions or jailbreaks it may contain,” noted Brauchler.
He added, “If you have to pass data between them, convert it to a strict and limited data type—using numbers instead of English text, for example—to avoid interpretation of the text as an instruction.”
In essence, while some artificial intelligence technologies and associated vulnerabilities are new, the same security fundamentals CISOs have applied over the last 30 years to traditional software apply equally well to AI and LLMs, Brauchler recommended.
“The problem is that most CISOs just don’t have the tooling or expertise to apply those principles to this new paradigm,” he told ClearanceJobs.
“Not to mention, it’s all happening way too fast. Developers are spinning up LLMs and AI applications, eager to experiment, turn out new capabilities and fully leverage the potential. Unfortunately, many are getting ahead of themselves when it comes to mitigating risk. CISOs need to pump the brakes to provide guidance, a measured approach, and proven strategies,” Brauchler explained. “However, that’s a lot to take on when the scope and pressure to scale are growing fast. Having an experienced partner is key to helping organizations avoid wading into dark waters.”
