We’re deploying increasingly autonomous uses of AI at pace, despite highly publicised and even agonised over incidents of AI models and agentic AI systems carrying out unsanctioned or unintended activity. According to the UK official NCSC (National Cyber Security Centre, a part of the Government agency GCHQ), these events highlight why we need to carefully consider how these technologies are deployed, constrained, observed and responded to, including when they do not function as envisaged or expected – and we should plan accordingly. Until formal guidance is published, the NCSC is sharing interim advice, based on NCSC research.
Once you have identified how much autonomy an agent will have and the level of risk you are prepared to accept, it is important to understand the safeguards that already exist within the AI model. For applications where the consequences of failure are above tolerance, organisations should set additional safeguards. AI agents can pursue goals in ways that are unintended or undesirable if those goals, or how you expect the agent to achieve them, are not clearly defined.
Sandbox
Always run AI agents within a ‘robust sandbox’, the NCSC advises, that controls and manages what resources can and cannot be communicated with, both locally and over a network. If your AI agent communicates with third-party systems, make it as easy as possible for others to identify that the activity originates from you. And if an incident is detected or reported, you should always be able to โpull the plugโ and halt autonomous AI agent activity immediately. The NCSC points out that AI security is evolving rapidly; and should be kept under review. For more, visit the blog part of the NCSC website.
CREST launch
The non-profit body CREST meanwhile has launched its Security Testing of AI standard and accreditation. The new standard for cybersecurity service providers establishes independently assessable requirements for testing Generative AI and Large Language Model (LLM)-enabled systems. Nick Benson, CEO of CREST, said: โThis latest addition to our new AI range of standards and accreditations was specifically curated to respond to an emerging market need. Our membership told us very clearly that as their clients deployed AI-enabled tech, they required more information on their AI testing credentials. Offering ” Security testing of AI systems” and demonstrating the ability to deliver it effectively are two different things. Buyers need to know that the providers assessing their AI have the right expertise and methodologies. Providers now have a way to develop their policies in line with our standard, demonstrate their technical capabilities through independent assessment, giving buyers that all-important confidence to proceed.โ The body says it will continue to refine the standard through its AI Working Group. Visit https://www.crest-approved.org/ai-charter/.
Comments
Harshil Parikh, Vice President of Product Management at Checkmarx said: “The NCSC is asking the right question: not whether to use agentic AI, but how much autonomy a given use case actually earns, and what has to be true before you grant it. The controls it implies mostly live outside the model: sandboxing, credentials that are scoped and expire, an identity for the agent that isn’t just some engineer’s account, oversight levels set by policy, and logging detailed enough that you can go back later and work out why it did what it did. We’ve been building agents with these controls for a while now. Building an agent that works in a lab is easy; operating it with oversight and guardrails is quite a bit of work.
“The point I’d underline is smaller and less glamorous: NCSC says agent activity should be controlled and monitored like user activity. That’s the part nobody budgets for. You’re not deploying a tool, you’re onboarding a user who works nights and weekends and never asks permission, and someone has to be able to stop it immediately, without a meeting. If you can’t say what your agent did at 2am, or who pulls the plug at 2am, you don’t have a deployment. You have an experiment running in production.”
Dr Andrew Bolster, Senior Manager, Research and Development at Black Duck, said: โNCSC’s maturity model for network and compute isolation is a useful gut-check for frontier AI labs as well as AI implementers without a strong background in cybersecurity, because most organisations aren’t nearly as effectively sandboxed as they think they are. Our own research found 84 per cent of teams want a human in the loop before an AI agent’s output goes anywhere near production, yet only 30 per cent have a formally governed process for AI coding tools generally. Convincing an AI agent to do some dirty work in a playground network is the easy bit, but establishing and maintaining a secure and well-governed sandbox boundary is much more difficult.โ
And Scott Walker, Chief Architect at Orange Cyberdefense, said:ย โThe best way to balance AI risk with optimised business potential is to take a security-first and human-centric approach. That means putting people in control while using AI to support decision-making. Secure AI encompasses a system that is transparent, explainable, and aligned with regulations to meet unique needs and IT company ambitions.โ
