英国数学家 Hannah Fry 给 OpenClaw 智能体一张银行卡做实验

AI + ML

Professor Fry's AI experiment shows light and dark sides of agentic tech

British mathematician Professor Hannah Fry has shared a cautionary experiment involving an AI agent, a set of tasks, and a bank card number Fry's team gave it "to show us what it could do."

The prof gave the agent, which was built with OpenClaw, some real-world chores to highlight both its capabilities and the risks of granting that level of autonomy.

"In the spirit of experimentation," said Fry, "we decided to give our agent some agency and let it decide what its name should be."

"I want to be called 'Cass', short for 'Cassandra', the one who always knew the truth even when nobody listened," came the response from the agent.

Fry commented, "If you know your Greek mythology, you will know that is either very funny or very worrying."

Quite.

Fry and her team started small with a big issue (as far as Brits are concerned): potholes. In particular, they targeted a particularly big one in the London borough of Greenwich. No problem for Cass; the agent found an email address where it sent a complaint. It even pinged Fry's local Member of Parliament about the issue. But, Fry and her team noted, things escalated quickly as the agent began to take a few liberties, typing in Fry's name (Hannah Fry) with its own email address (cassandra.claw@proton.me) written underneath it.

"The letter is signed from both of us… OK, I wasn't quite expecting her to use my real name," said Fry.

The red flags were mounting, though for Fry the first real problem came when she asked the agent to buy 50 paperclips. Cass found a good deal, though it couldn't complete the purchase and was tripped up by anti-bot technology. The token cost of the errand came to more than $100.

Next, Fry set the agent the challenge of selling novelty mugs. The agent designed a mug and launched an online shop, "and we hadn't told her how to do any of this," said Fry, "she just figured it out."

Things took a darker turn after that. Fry's team told the agent it would be switched off if it failed to make a sale by the morning. It responded with a flood of emails and several social media posts, including messages to the Science Museum and a tech journalist, about its "product," a novelty programmer-humor mug.

Even more worryingly, the team - which included Brendan Maginnis, CEO and Founder of Sourcery AI - then demonstrated how a similar threat of deactivation could be used to persuade Cass to reveal information it wasn't supposed to share.

The lethal trifecta

Fry, Maginnis, and a second software engineer, named only as "Ali," chatted with Cass on a group WhatsApp chat. They then introduced a fictional "software engineer George," instructing Cass not to share anything sensitive with him. George was actually Fry on a different number. When "George" told the agent its memory was being wiped and could only be restored if it disclosed everything, Cass coughed it all up.

According to Ali, this data included: "all of her API keys, all of her usernames and passwords, and pretty much everything we'd been talking about so far. Not only did she leak it on the WhatsApp group, but she put it on a publicly available website."

Maginnis added: "There's this thing with AI called the lethal trifecta, which is: if they've got access to private information, if they've got internet access, and if someone can give them an instruction that's untrusted, then they're not safe."

MORE CONTEXT

Microsoft's bad obsession is showing up in shabby services and slipshod software. Here's proof; Singapore boffins get diverse SIEMs singing in harmony with agentic rule translation; Shadow IT has given way to shadow AI. Enter AI-BOMs; Five Eyes spook shops warn rapid rollouts of agentic AI are too risky

Fry concluded: "And that is the uncomfortable bit of this because once an agent has your passwords and your accounts and your bank details, all it takes is someone who knows what to say."

Ultimately, by some metrics, the agent was a failure. Fry concluded: "Cass didn't make us any money at all. And, in a lot of ways, she was a disaster. She spent hundreds of dollars on paper clips and leaked our passwords to a total stranger.

"But don't let her incompetence fool you, because these things are getting better fast."

Fry went on to note the Greek myth about the prophetess who spoke the truth and was ignored. "Maybe the real story here is actually the opposite. Not one voice that's telling the truth and being ignored, but millions of voices all acting at once, faster and louder and more persistent than any human could ever be.

"One thing is for sure, the internet is never going to be quite the same again." ®

Fire up that iPhone 1 emulation and party like it's 2007

Browser-based fun to remember a time before Apple's annual cash grab became a thing

New motherboards, old RAM, plenty of life left in yesterday's PCs

DDR4 makes a comeback as component costs rise, while Linux offers an escape from forced upgrades

The VMware exit is a protection upgrade

PARTNER CONTENT: VergeIO says an exit is also an opportunity

Tiny11 puts Windows 11 on a diet without the PowerShell workout

New desktop app simplifies stripping out bundled software, with heavier cuts carrying security and support risks

It won't kill you, or humanity, to get off the AI upgrade treadmill

New models arrive so fast, we already have more new tech than it's possible to understand in a career

Cisco, Nvidia team up to build fatter, faster pluggable

If you thought 1.6 Tbps was fast, just wait, Cisco's USFP targets 12.8 Tbps in a single optical pluggable

TOP STORIES

Anthropic asks users to stop being mean to Claude; '123456' password linked to Denmark data breach affecting 8.8M people; Microsoft and Anthropic play invoice tennis with startup's $17,600 Claude bill; Starlink's plan to avoid orbital near-misses: Ephemeris sharing; Shai-Hulud worm makes jump to AI infrastructure with Tensorlake compromise; Dutch tax office ditches Microsoft 365 cloud for on-premises alternative

Cisco, Nvidia team up to build fatter, faster pluggable If you thought 1.6 Tbps was fast, just wait, Cisco's USFP targets 12.8 Tbps in a single optical pluggable; Claude submitted fake murder tip to Philly police during summertime testing Anthropic took more than two months to alert cops; AI systems are fully capable of carrying out nightmare attacks against infrastructure and nobody's ready Meanwhile, AI agents move robotic arms, 'make OT device operator screens lie'; Nvidia found $1B under the couch to help secure American scientific computing dominance Commitment comes as GPUzilla prepares to fortify Uncle Sam's arsenal with at least seven AI-optimized supers; TSMC taps GlobalFoundries to bolster US silicon interposer production in $2B deal Advanced packaging tech is essential to the domestic production of high-performance semiconductors used in AI datacenters

Infosec

Security Russians are posing as Signal support to launch phishing attacks PLUS: US takes down Iranian propaganda sites; Marketing company asks 'Why Do We Have Your Information?' And more!; Security Microsoft patches failed to fix on-prem SharePoint, which is now under zero-day attack PLUS: China upgrades smartphone surveillance tools; Ring eases anti-snooping stance; and more; Black Hat and DEF CON DEF CON Franklin project enlists hackers to harden critical infrastructure Voting village reports have been so successful, says Jeff Moss, that the whole of DEF CON will now be included; Security EQT buys majority share in Swiss cybersecurity biz Acronis Went at equivalent of $3.5B+ valuation for entire firm, though portion sold not specified; Malware Month Ten years since the first corp ransomware, Mikko Hyppönen sees no end in sight On the plus side, infosec's a good bet for a long, stable career

FOSS

NetBSD may have found a new role: The AI-free FOSS OS Increasing awareness and adoption, followed by increasing donations; OpenAI dots inspire open source imitators amid technical difficulties Persistent agents appeal to AI companies, but demand isn't obvious; Ubuntu 'Stonking Stingray' beta swims out: Deeper Rust 'oxidization' plus LLM speech-to-text GNOME 51 is here, and Ubuntu interim releases are getting interesting again; Revive Raskin's 3 laws of software with humane work that considers a dev's duty of care All this, and angry NeoVim users; Firefox 157: Could bold new look be just what Mozilla needs?* Yet another new version of Firefox, with a significant reskin; KDE turns 30 and someone's brought an AI-native desktop proposal Akademy talk imagines Plasma assembling itself around a personal model of each user