TypeSafe’s Jev ‘even more susceptible to attack’ than LLMs, says security vendor

20/10 16:15 - TypeSafe’s Jev ‘even more susceptible to attack’ than LLMs, says security vendor
Although the Jev model rolled out last month by TypeSafe claims to have significant advantages over traditional LLMs, a security report published Wednesday argued that its streamlined approach makes it more vulnerable to some kinds of attack, compared to traditional LLMs.   “In some cases, [Jev] is even more vulnerable to attacks, with both adversarial text attacks and prompt injection being able to disrupt the model’s outputs, resulting in incorrect and potentially devastating actions,” said the blog post from researchers at HiddenLayer.  TypeSafe did not respond to multiple requests for comment.  LLM, or something else? There seems to be debate about whether Jev is technically a large language model (LLM), with TypeSafe suggesting it is not, but HiddenLayer saying that it appears to be one. TypeSafe has even coined a name for the category: System One, which TypeSafe described as “a new class of frontier models built to make fast, structured decisions that software can use directly.” Jev is the first—and thus far, only—named product categorized as System One. Jev is a streamlined model designed to function machine-to-machine. Its approach allows it to perform tasks faster and to use fewer tokens. But Kasimir Schulz, the director of security research at HiddenLayer, the blog’s co-author, said that the accelerated approach robs Jev of some of the checks and balances that exist in traditional LLMs. This means that Jev can fall for some attacker tricks that current LLMs typically detect and refuse, he said in an interview.  Despite this, “Jev is great. You should use it. It has a lot faster response time,” he said, with the caveat “it doesn’t have all of the protections” seen in traditional LLMs, so some attacks on it succeed that should have failed.  “TokenBreak, for example, normally does not work with larger LLMs, but Jev is vulnerable to it out of the box,” Schulz said.  Although simply inserting characters in front of important words, as in the original TokenBreak attack, did not yield much, HiddenLayer’s blog noted, “we did notice that increasing the number of perturbations and improving their targeting led to much higher attack success rates.” But although traditional LLMs are typically able to see through such attacks, it pointed out, “System One models appear to be vulnerable. As a result, this class of decoder should be treated with caution, given that they are susceptible to both autoregressive LLM attacks and text classifier attacks, providing potential adversaries with greater optionality.” The post cited some examples:  “On a simulated checkout page, an attacker-controlled advertisement caused the browser agent to click away from the user’s purchase [and respond to the ad] despite instructions to treat page text as untrusted,” it said. “In a separate classification test, targeted TokenBreak-style changes lowered toxicity scores for prompts both models had initially flagged.” The post argued that multiple risks are therefore associated with Jev.  “A browser agent may immediately click a selected element. A moderation system may allow an input whose toxicity score falls below a threshold. A routing system may send a request to a different tool or workflow. Each of these turns a model judgment into an operational decision, and therefore, risk,” the post said. “When evaluating System One models, test those decisions in the application where they will be used. Include untrusted webpages, misleading task-related text, and modified inputs that preserve their meaning while changing their form. Monitor the selected actions and scores and put independent checks around outcomes that matter. Keep in mind that, under the hood, these are still language models at heart.” Analysts agreed with both the report’s methodology and its technical conclusions, while also pointing out that a security vendor concluding that a new approach is not secure is somewhat self-serving. Philip Harris, an IDC research director, observed that the methodology described in the blog post “is sound, and consistent with established AI red-teaming practice: building a controlled test environment and probing for adversarial manipulation and injection is exactly the right approach.” But Gartner VP Analyst Nader Henein added that Jev, which was introduced on Sept. 15, is so new that he thinks some of the flaw reports were premature. “Jev will go through the same maturity process as do all other models, including susceptibility to seemingly rudimentary manipulation attempts,” Henein said. “This model was only released last month, so being surprised that it has issues is similar to being surprised when your baby drools.” However, Frank Dickson, principal analyst at Dickson Research, noted that the same attributes that make Jev so attractive also expose it to riskier behavior. “Jev can only choose from options the developer puts on the menu,” he said. “It limits attack paths. However, if ‘click any element on the page’ is on the menu, the attacker’s ad is on the menu. That is an architecture decision, and the buyer makes it.” He pointed out, “Jev makes AI decisions faster and cheaper, which also makes the wrong decisions faster and cheaper. Speed is a feature. It is not a security control.” Independent technology consultant Steven Eric Fisher agreed. “Constraining the format of a decision does not necessarily constrain the correctness, trustworthiness, or consequences of that decision,” he said, noting, “the simulated checkout scenario [in the post] illustrates how an agent can be redirected from an authorized task by information introduced through a less-trusted component of its operating environment. The fact that the agent was explicitly instructed not to trust page content, yet still responded to the malicious advertisement, is particularly relevant.” This article originally appeared on InfoWorld. ...