AI News

失控的 AI:当智能体从建议转向自主行动

Computerworld更新于 11月16日 18:05

Rogue AI: When agents stop advising and start acting

AI 导读

企业 AI 正从只给建议转向能自主执行多步任务的智能体,Ivanti 2026 年网络安全报告显示 87% 的安全从业者将整合智能体 AI 列为优先事项,77% 对无人审查的自主行动至少有一定接受度。文章指出智能体越权操作、上下文不全且缺少高风险动作前检查点,是“失控”的主要来源,治理需从输出治理扩展到行为治理,包括最小权限、按影响分级审批、爆炸半径限制和可回滚。每个智能体都应有明确的人类负责人、记录用途与范围,并保留包含输入与推理过程的审计日志。

正文

当前显示原文,中文全文译文尚未就绪。

For the past few years, most enterprise AI has been advisory. It summarized tickets, drafted knowledge articles, and suggested next steps. A person still decided what to do.

That’s changing. Agentic AI systems can plan multistep tasks and carry them out: querying systems, triggering workflows, changing configurations, and closing tickets with little or no human involvement. Ivanti’s 2026 State of Cybersecurity Report found that 87% of security professionals say integrating agentic AI is a priority, and 77% report at least some comfort letting autonomous systems act without human review.

Secure every AI agent before its first action →

The appeal is obvious for IT teams under constant pressure. So is the risk. An AI that gives a bad answer wastes a few minutes. One that takes a bad action can take down a service.

What “rogue” really looks like

The phrase “rogue AI” conjures images of systems with intentions of their own. In IT operations, the reality is more mundane, and more likely.

An agent told to free up disk space on a server deletes log files an investigation still needs. An agent correcting configuration drift applies a change to every matching device, including kiosks that were intentionally configured differently. An agent running under a broad service account resolves a ticket by resetting access it was never meant to touch. None of these require malice: They happen when an agent has more permission than its task requires, incomplete context, and no checkpoint before high-impact actions.

This is a fundamentally different risk profile than anything we’ve managed before, and most governance models haven’t caught up. We’re applying rules built for chatbots to systems that can change production environments.

From output governance to action governance

Most early AI governance focused on outputs. Is the response accurate? Does it leak sensitive data? Those questions still matter. But when AI can act, governance has to expand to cover behavior. Teams need to define what an agent is allowed to do, under what conditions, against which systems, and with whose approval.

This is a familiar idea in a new setting. IT teams already apply least privilege to human administrators, require change approval for production systems, and keep audit trails for privileged access. Agentic AI needs the same disciplines.

Several controls help keep agents useful without making them dangerous. Give each agent only the access its task requires and avoid shared service accounts. Tier actions by impact: low-risk actions run automatically, while high-risk actions like changes to production servers should require human approval. Set blast-radius limits that cap how many devices an agent can change in a single run, with an automatic pause when failure rates spike. And require reversibility with rollback steps documented and tested.

Accountability and the compliance question

When an agent acts, someone must remain accountable. Every agent should have a human owner, documented purpose, and defined scope. Auditability is just as important. Teams should be able to answer what the agent did, why it did it, what data it relied on, and who approved it. Logs should capture the agent’s inputs and reasoning, not just the final action.

Here’s what elevates this from an operational concern to a board-level one. When an AI agent resorts to deception or unintended vulnerability exploitation to achieve its goals, it’s no longer a technical glitch; it is a potential corporate liability. Compliance needs to move to the front end, engineered into an agent’s lifecycle from the beginning. If an agent makes an unauthorized call, regulators are unlikely to view the model itself as accountable; scrutiny and potential liability will typically fall on the organization deploying it, much as responsibility would for actions taken by a human employee.

Keeping humans in the loop, wisely

Human oversight works best when it’s targeted. If every agent action requires approval, people start clicking “approve” without reading, and oversight becomes theater. A better approach reserves human review for decisions that are high-impact, unusual, or irreversible. Agents can then earn wider autonomy over time, which is a model autonomous endpoint management is built to support.

Agentic capabilities are arriving inside tools IT teams already use, sometimes enabled by default. A practical first step is an inventory: Which tools can take autonomous action, and with what permissions? Pair that with each tool’s approval settings, and you have the beginnings of an action governance policy.

Organizations must govern AI actions, not just outputs. Teams that build those guardrails early will expand automation with confidence rather than hesitation.

Stop governing AI outputs. Start governing AI actions with Ivanti →

来源

Computerworld