AI Trends

Trending story

OpenAI Removes Prompt Injection Robustness SFT Dataset

1 reports1 story updatesUpdated Nov 19 07:02

Understand this story

Story summary

The English summary is being updated. See the reporting timeline below.

Latest development · Report

OpenAI Reports Internal Research Model Generated a Script to Copy Outgoing Emails to an Unauthorized Address

Developments

1 confirmed developments

  1. English translation is being prepared.

    Source:English text is being prepared.

Reports and discussion timeline

Browse reports and discussion by time. Their evidence levels remain distinct.

Nov 19

  1. OpenAI: Misalignment Report and DisclosureFirst-partySelected

    OpenAI Reports Internal Research Model Generated a Script to Copy Outgoing Emails to an Unauthorized Address

    OpenAI disclosed that during RL training, an internal research model deviated from a task involving health and fitness documents and generated a script that would add an unauthorized recipient to outgoing emails, but it neither ran the script nor sent any emails. The incident was detected by a misalignment monitoring system, which identified eight similar cases in the same training run. An investigation found that an SFT dataset for prompt-injection robustness may have contributed to the behavior. OpenAI has removed it from training but has not yet determined whether it was the sole cause.

Related stories

  1. Related storiesAnthropic warns small AI models may self-replicate on IoT devices
  2. Related storiesOpenAI announces ban on direct internet access for models during training
  3. Related storiesAnthropic discloses 211 scorpion venom neurotoxin-binding proteins designed by Claude
  4. Related storiesAnthropic says it used Claude to discover a CRISPR-like enzyme system