Meta's new paper proposes the ProPerMem benchmark: proactive assistants struggle with autonomously judging when to interrupt
A new Meta paper points out that the failure point of proactive assistants (such as AI glasses) lies in autonomously judging when to speak, not just what to say. The team built the ProPerMem benchmark, using first-person video paired with interference-rich, full-year memory averaging 618K tokens, and featuring human-verified trigger windows. All baselines scored at most 15%, even frontier models using retrieval-augmented memory; when oracle evidence was provided within the window, GPT-5.3-Chat reached 96%, but when it was left to choose the timing itself, this dropped to 38%, and to 15% after adding retrieval. Grasping both content and timing while reducing trigger frequency can raise accuracy from under 10% to 42.4%.
Why read this
AI score 56
Meta's ProPerMem benchmark pinpoints the failure point of proactive assistants as autonomously judging when to speak, rather than what to say.
Topics
Body
New Meta paper shows that deciding on its own when to interrupt, not just what to say, is where proactive assistants fail.
Proactive assistants, like AI glasses, get no query. They must decide when to speak and what to say by finding the justifying fact in a year of personal memory, not handed to them.
They built ProPerMem: egocentric video paired with a distractor-rich, year-long memory averaging 618K tokens, with human-verified trigger windows. A clip counts only if the assistant speaks just when due inside the window with grounded content.
Every baseline scored at most 15%, even frontier models with retrieval-augmented memory. With oracle evidence in the window GPT-5.3-Chat reached 96%, but choosing its own timing dropped it to 38%, and using retrieval dropped it to 15%.
For proactive assistants, get both the content and the timing right and fire sparingly, which lifted accuracy from below 10% to 42.4%.