As AI world debates security, NVIDIA releases open source tools for agents

Skip to main content

Get our latest cybersecurity news first on Google.

Click here!

Amid growing concerns from both the public and policymakers about cyberattacks involving advanced AI systems, NVIDIA has released a new open software security platform for AI agents.

The Open Agent Safety Platform, according to NVIDIA, will be “an open software platform and reference system design with full-stack governance and control across the software and hardware, compute and robotics systems that run agents.”

More than 100 organizations from the AI industry, including Anthropic, Arm, Microsoft, SpaceXAI, Palantir and JPMorgan Chase, have committed to using the platform to improve security in their products.

Stronger sandboxes and more advanced monitoring are pillars of NVIDIA’s strategy for tackling “rogue” AI agentic hacks, and the platform offers a pair of tools that are meant to help contain and restrict agent activity. 

According to a corresponding technical blog, OpenShell, built on Apache 2.0 open source software, is a tool for securing runtime execution for AI agents in sandbox testing environments. AI system operators can define files, networks, tools, processes and credentials that an agent has, and test whether those guardrails hold before introducing them to enterprise networks.

It also includes security updates for NVIDIA’s Bluefield 4 data processing unit (sometimes called a “data center on a chip”) that enables out-of-band monitoring of AI agent behaviors and security policy enforcement.

On X, NVIDIA CEO Jensen Huang called the platform “the beginning of an open ecosystem to build the trust layer for safe agent systems.”

To do this, security work must become “foundational” to AI and the industry’s “full promise can only be realized when people have confidence that AI is being built to be safe and deployed with wisdom and responsibility.”

“Trust and innovation are not in conflict,” Huang wrote. “Safety is how trust is earned. We must build not only the most capable AI, but the most trusted AI, so that this extraordinary technology can realize its enormous promise for the world.

NVIDIA manufactures advanced computer chips that power much of the U.S. commercial AI industry. After models from Anthropic, OpenAI, Meta and others escaped sandbox protections during testing and breached real organizations, some AI industry leaders – including Anthropic CEO Dario Amodei and OpenAI CEO Sam Altman –  suggested AI systems are too advanced to be contained.

Huang has criticized those views, arguing that frontier AI companies and their supply chain partners must improve their security posture. Huang said last week he opposes government regulations or mandates on the AI industry, unless they promote growth.

In an interview with CNBC on Monday, Huang said he believes sandboxing and larger AI security issues are “a technically solvable problem.”

“I think the answer is we hope it’s an engineering problem,” said Huang. “I believe it’s an engineering problem, I know it’s an engineering problem, and we all need to hope that it’s an engineering problem. [If not] it’s not solvable, so the fact that all of these companies still are advancing the state of the art is because they also believe it’s solvable.”

Aviv Nahum, CEO of Above Security, told CyberScoop that NVIDIA’s announcement reflects industry recognition that model training alone cannot ensure safety. 

NVIDIA is essentially arguing that for agents to perform safely in real-world environments, “some of the enforcement must live outside the model, in a layer the agent cannot simply reason around or modify.”

“What this announcement says to me is that the industry is finally converging on a basic cybersecurity principle, [that] model alignment is not a substitute for security engineering,” said Nahum. “Sandboxes, identity, least privilege, independent monitoring and containment are not new ideas. What is new is that we now have autonomous software capable enough that failing to apply those principles becomes much more consequential.”

More Scoops

Latest Podcasts

Government

• Supreme Court permits states to use SAVE database for citizenship checks

• House and Senate members propose legislation for CISA to step up cyber defenses for biotech

• Phone-hacking company that won U.S. security agency contracts hid Russian ownership, DOJ alleges

• Bipartisan Senate leaders introduce bill to bolster telecom cybersecurity in response to Salt Typhoon hacks

Technology

• CISA outlines improvement plan for CVE program

• What’s next for CISA's CDM program that gives cybersecurity tools to federal agencies

• European parliament members call for slowdown of Serbia’s EU entry over spyware use

• The G7 tells industry to hurry up and prep for post-quantum encryption

Threats

• Watchdog finds most agencies failed to meet CISA cloud security orders, heightening risk of attack

• Ryuk ransomware operator sentenced to 2 years in prison

• ShinyHunters claims attack on FBI exposes almost all agents

• Volexity spots another China-aligned threat group exploiting Chrome and Microsoft defects

Policy

• The president has called for AI leadership. Here’s the mission.

• After water attacks, Capitol Hill offers its own proposal for an AI-cyber test program

• Dems seek top-to-bottom assessment of CISA workforce

• Supreme Court denies Trump request to allow USPS mail ballot changes