安大略审计发现医生AI记录工具频繁出错

AI and ML

60% of evaluated AI Scribe systems mixed up prescribed drugs in patient notes, auditors say

The AI systems approved for Ontario healthcare providers routinely missed critical details, inserted incorrect information, and hallucinated content that neither patients nor clinicians mentioned, according to a provincial audit of 20 approved vendors’ systems.

The findings come from the Office of the Auditor General of Ontario, Canada, and are included in a larger report about the state of AI usage by public services in the province. They specifically address the AI Scribe program, the Ontario Ministry of Health initiated for physicians, nurse practitioners, and other healthcare professionals across the broader health sector.

As part of the procurement process, officials conducted evaluations using simulated doctor-patient recordings. Medical professionals then reviewed the original recordings alongside the AI-generated notes to evaluate their accuracy.

What they found was, frankly, shocking for anyone concerned about the accuracy of AI in critical situations.

Nine out of 20 AI systems reportedly “fabricated information and made suggestions to patients' treatment plans” that weren’t discussed in the recordings. According to the report, evaluators spotted potentially devastating incorrect information in the sample reports, such as no masses being found, or patients being anxious, even though these things were never discussed in the recordings.

Twelve of the 20 systems evaluated inserted incorrect drug information into patient notes, while 17 of the systems “missed key details about the patients’ mental health issues” that were discussed in the recordings. Six of the systems “missed the patients’ mental health issues fully or partially or were missing key details,” per the report.

OntarioMD, a group that offers support for physicians in adopting new technologies and was involved in the AI Scribe procurement process, has recommended that doctors manually review their AI notes for accuracy, but the report notes there’s no mandatory attestation feature in any of the AI Scribe-approved systems.

Bad evaluations don’t help, either

AI systems making mistakes isn’t exactly shocking. As we’ve reported previously, consumer-focused AI has a tendency to provide bad medical information to users, and some studies have found large language models failed to produce appropriate differential diagnoses in roughly 80 percent of tested cases. But the tools evaluated here are for doctors, not consumers, and such poor performance necessitates explanation. A good portion of the report blames how the systems were evaluated.

According to the report, the weight given to various categories of AI Scribe performances was wonky. While 30 percent of a platform’s evaluation score depended solely on whether they had a domestic presence in Ontario, the accuracy of medical notes contributed only 4 percent to the total score.

MORE CONTEXT

AI chatbots are no better at medical advice than a search engine; AI doctor's assistant is easily swayed to change prescriptions, give bad medical advice; ChatGPT Health wants your sensitive medical records so it can play doctor; 'It looks sexy but it's wrong' – the problem with AI in biology and medicine

Bias controls accounted for only 2 percent of the total evaluation score; threat, risk, and privacy assessments counted for another 2 percent; and SOC 2 Type 2 compliance contributed an additional 4 percentage points.

In other words, criteria tied to accuracy, bias controls, and key security and privacy safeguards made up only a small portion of the total evaluation score for the AI Scribe systems.

“Inaccurate weightings could result in the selection of vendors whose AI tools may produce inaccurate or biased medical records or lack adequate protection to safeguard sensitive personal health information,” the report said of the scoring regime.

The Register reached out to the Ontario Health Ministry for its take on the report, and whether it was going to conform to its recommendations for the AI Scribe program, but we didn’t immediately hear back. A spokesperson for the Ministry told the CBC on Wednesday that more than 5,000 physicians in Ontario are participating in the AI Scribe program and there have been no known reports of patient harms associated with the technology. ®

Fire up that iPhone 1 emulation and party like it's 2007

Browser-based fun to remember a time before Apple's annual cash grab became a thing

New motherboards, old RAM, plenty of life left in yesterday's PCs

DDR4 makes a comeback as component costs rise, while Linux offers an escape from forced upgrades

The VMware exit is a protection upgrade

PARTNER CONTENT: VergeIO says an exit is also an opportunity

Tiny11 puts Windows 11 on a diet without the PowerShell workout

New desktop app simplifies stripping out bundled software, with heavier cuts carrying security and support risks

It won't kill you, or humanity, to get off the AI upgrade treadmill

New models arrive so fast, we already have more new tech than it's possible to understand in a career

Cisco, Nvidia team up to build fatter, faster pluggable

If you thought 1.6 Tbps was fast, just wait, Cisco's USFP targets 12.8 Tbps in a single optical pluggable

TOP STORIES

Anthropic asks users to stop being mean to Claude; '123456' password linked to Denmark data breach affecting 8.8M people; Microsoft and Anthropic play invoice tennis with startup's $17,600 Claude bill; Starlink's plan to avoid orbital near-misses: Ephemeris sharing; Shai-Hulud worm makes jump to AI infrastructure with Tensorlake compromise; Dutch tax office ditches Microsoft 365 cloud for on-premises alternative

Cisco, Nvidia team up to build fatter, faster pluggable If you thought 1.6 Tbps was fast, just wait, Cisco's USFP targets 12.8 Tbps in a single optical pluggable; Claude submitted fake murder tip to Philly police during summertime testing Anthropic took more than two months to alert cops; AI systems are fully capable of carrying out nightmare attacks against infrastructure and nobody's ready Meanwhile, AI agents move robotic arms, 'make OT device operator screens lie'; Nvidia found $1B under the couch to help secure American scientific computing dominance Commitment comes as GPUzilla prepares to fortify Uncle Sam's arsenal with at least seven AI-optimized supers; TSMC taps GlobalFoundries to bolster US silicon interposer production in $2B deal Advanced packaging tech is essential to the domestic production of high-performance semiconductors used in AI datacenters

Infosec

Security Russians are posing as Signal support to launch phishing attacks PLUS: US takes down Iranian propaganda sites; Marketing company asks 'Why Do We Have Your Information?' And more!; Security Microsoft patches failed to fix on-prem SharePoint, which is now under zero-day attack PLUS: China upgrades smartphone surveillance tools; Ring eases anti-snooping stance; and more; Black Hat and DEF CON DEF CON Franklin project enlists hackers to harden critical infrastructure Voting village reports have been so successful, says Jeff Moss, that the whole of DEF CON will now be included; Security EQT buys majority share in Swiss cybersecurity biz Acronis Went at equivalent of $3.5B+ valuation for entire firm, though portion sold not specified; Malware Month Ten years since the first corp ransomware, Mikko Hyppönen sees no end in sight On the plus side, infosec's a good bet for a long, stable career

FOSS

NetBSD may have found a new role: The AI-free FOSS OS Increasing awareness and adoption, followed by increasing donations; OpenAI dots inspire open source imitators amid technical difficulties Persistent agents appeal to AI companies, but demand isn't obvious; Ubuntu 'Stonking Stingray' beta swims out: Deeper Rust 'oxidization' plus LLM speech-to-text GNOME 51 is here, and Ubuntu interim releases are getting interesting again; Revive Raskin's 3 laws of software with humane work that considers a dev's duty of care All this, and angry NeoVim users; Firefox 157: Could bold new look be just what Mozilla needs?* Yet another new version of Firefox, with a significant reskin; KDE turns 30 and someone's brought an AI-native desktop proposal Akademy talk imagines Plasma assembling itself around a personal model of each user