Understand this story
On October 21, 2026, Meta published a paper pointing out that the failure point of proactive assistants (such as AI glasses) lies in autonomously judging when to speak, not just what to say. The team built the ProPerMem benchmark, using first-person video paired with interference-rich full-year memories averaging 618K tokens, and featuring manually verified trigger windows. The paper reports that all baselines scored at most 15%, even frontier models using retrieval-augmented memory; when oracle evidence was provided within the window, GPT-5.3-Chat reached 96%, but when it was left to choose the timing itself, this dropped to 38%, and after adding retrieval it fell to 15%. Grasping both content and timing while reducing trigger frequency can raise accuracy from under 10% to 42.4%. The work is currently released as a paper, with no subsequent progress yet.
AI-generated from public reports
Developments
1 confirmed developments
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Oct 21
Rohan Paul
Meta's new paper proposes the ProPerMem benchmark: proactive assistants struggle with autonomously judging when to interrupt
A new Meta paper points out that the failure point of proactive assistants (such as AI glasses) lies in autonomously judging when to speak, not just what to say. The team built the ProPerMem benchmark, using first-person video paired with interference-rich, full-year memory averaging 618K tokens, and featuring human-verified trigger windows. All baselines scored at most 15%, even frontier models using retrieval-augmented memory; when oracle evidence was provided within the window, GPT-5.3-Chat reached 96%, but when it was left to choose the timing itself, this dropped to 38%, and to 15% after adding retrieval. Grasping both content and timing while reducing trigger frequency can raise accuracy from under 10% to 42.4%.