Can public service employees trust AI inside the software they use every day? asks asks Bruno Bossola, CTO and co-founder of the open source vulnerability scanner Meterian.
An AI tool might save a council employee time by summarising a residentโs complaint. Someone still needs to notice if it leaves out critical details. That requires familiarity with the work and time to check the original.
Most deployment plans say a human will check the output. Who is that person on this shift, and what if they are off-duty?
Public bodies should be able to answer that before expanding AI use in 2027. Assigning responsibility to an employee is easy. But giving them a realistic chance of catching a mistake requires decisions about how the service runs.
At DigiGov London, our team heard speakers discuss employeesโ fears about AI and job losses. Conversations also raised questions about trusting AI inside everyday software. People were being encouraged to rely on systems whose work they would still need to check. There was enthusiasm, too. My colleagues came away encouraged by the willingness to discuss problems openly and the emphasis on keeping human judgement in public services. Visitors were interested in finding software risks earlier in development. I spoke with developers using, or considering, AI coding assistants.
Those conversations left me concerned about the work being assigned to the eventual user. An officer reviewing an output cannot reasonably be expected to compensate for weaknesses throughout the system that produced it.
Consider the staffing decision first. If a business case counts the time AI saves writing a report, it must also count the time someone needs to verify it. Comparing an AI summary with the underlying records can require considerable knowledge of the service. Removing experienced staff before understanding that workload could leave the organisation unable to deliver the oversight it promised.
The Local Government Associationโs 2025 AI survey found that 56 per cent of responding councils identified staff capability as a barrier to adoption; 52pc cited staff capacity. Only a third of councils responded, so these findings cannot describe every authority. They do, however, give substance to the concerns we heard. Security managers already understand shift cover. AI review needs the same attention. There should be a named operational owner and someone qualified to cover their absence. Both need access to the evidence behind an output. They also need authority to delay a decision when the evidence does not support it.
That authority becomes meaningless if staff are judged solely on how quickly they clear work. A reviewer who raises a concern must know how to escalate it without being treated as an obstacle to the promised efficiency gain. The amount of checking should reflect what could go wrong. A draft internal notice may need a light review. An output influencing a security response deserves closer scrutiny. Where an AI system can take action itself, restrictions on its permissions must support the person supervising it.
Suppliers have work to do here. A council may encounter AI through an update to software it already uses. The governmentโs September 2026 cyber skills report describes this happening through existing security products. The security manager needs to know what has changed before staff start depending on it.
Ask the supplier to explain what the feature can access and whether it can act on that information. Ask for evidence of testing against the tasks your staff will perform. Establish how to disable it while keeping essential operations running, and what support is available outside normal office hours.
There is a separate question about how the software was built. AI coding assistants can suggest components that developers then incorporate into a service. Buyers should expect suppliers to explain how those changes were reviewed and how newly discovered vulnerabilities will be handled. The Home Officeโs published engineering standards offer a useful example. They require qualified human review before AI-assisted outputs reach production and require teams to manage risks from AI-introduced dependencies. Procurement teams can ask suppliers for comparable evidence.
A software bill of materials, essentially a component inventory, can help establish whether a vulnerable dependency is present. Its value depends on keeping it current and acting on the findings. It cannot tell a security officer whether an AI-generated incident summary has missed something important. That needs its own checking process.
Before approving wider deployment, I would run a short exercise using the opening scenario. Give the handover to the person covering the usual reviewerโs shift. Include a plausible omission and ask them to investigate it using the records they would actually have available. Watch where they get stuck. Can they find the original alert? If they need to stop further action, do they have permission? Call the supplier through the agreed escalation route and see whether anyone can help.
Measure the time this takes. If checking exceeds the capacity available, narrow the systemโs role until the team can supervise it properly. Repeat the exercise after a significant change to the service. A council officer cannot be the final safety check without time to verify the work or authority to stop it. When managers approve their next AI deployment, they should ask to see how that check worked with the usual expert absent. The answer will tell them whether they have allocated enough staff to operate it safely.
