AI News

HCI pioneer de Souza discusses AI-mediated reading and cognitive authority at ACM MM 2026: proposes the PEEL approach

Sina TechUpdated Nov 16 21:10

AI summary

In her talk at ACM MM 2026, human-computer interaction pioneer Clarisse Sieckenius de Souza proposed the PEEL approach, using the deterministic computational tool Voyant to compare AI-generated summaries from Claude and others against the original text, thereby safeguarding researchers' cognitive authority. PEEL runs in Jupyter Notebook, preserves the original article's section structure and first-person expression, and offers different compression ratios such as 10% and 25%, using keyword frequency and semantic network visualizations to highlight new terms and connections introduced by the summary. She used a conversation in which Claude once misidentified the source of a paper's main thesis as an example, pointing out that users may not clearly know which agents behind the interface are involved in conveying information.

Body · AI translation

Add crowdsourced testing promotional copy Add crowdsourced testing promotional copy end Add App-exclusive traffic guidance

English text is being prepared.

English text is being prepared.

Have you ever had this experience: faced with dozens of pages of densely packed PDF, you really can't spare the time to read it carefully, so you toss the paper to ChatGPT or Claude and ask it to extract the core points for you.

A few seconds later, a clearly organized summary appears on the screen. You nod, feeling that you have already grasped the main content of the paper, and may even directly cite its points in your own research.

At the ongoing ACM Multimedia 2026, Clarisse Sieckenius de Souza, a pioneer in the field of human-computer interaction and professor emerita at the Pontifical Catholic University of Rio de Janeiro (PUC-Rio) in Brazil, delivered a keynote speech.

Starting from her own theory of "Semiotic Engineering," she discussed how AI is changing the relationship between people and information, and how, when we hand over the work of reading and understanding literature to AI, we can safeguard researchers' "Epistemic Authority."

In my view, human-computer interaction (HCI) can be understood as a conversation.

But this is not a simple conversation between an application or system and the user; rather, it is a computer-mediated conversation that takes place among human groups: on one side are the people who design and develop applications, and on the other are the people who use these applications.

This conversation unfolds through the interface. The interface actually plays the role of the designer's agent, conveying information to the user on the designer's behalf. It can only express itself in the ways and language pre-set by the designer—what we usually call an artificial language.

For example, when we use a website, the interface tells us which operations are allowed and which are not. On a cookie preferences page, a user may try to toggle a necessary setting, but the interface tells us that this operation cannot be completed.

This message actually comes from the designer. The feedback the user gets through the interface reflects what the designer wants the user to do, what the user can do, and what the user cannot do.

This is the basic idea of Semiotic Engineering: the interface speaks on behalf of the designer, and the user communicates with the designer by learning and using the interface language.

Of course, the interface language is not necessarily text. It can also be a fully graphical, object-centered language. We express our intentions by manipulating objects on the screen, and the interface responds to our operations through changes in those objects.

However, the interface does not always respond as the user expects.

I once encountered a rather interesting situation while preparing this talk: I tried to insert a text box into the interface, and it actually came out upside down. I had no idea why this happened.

This precisely shows that we sometimes cannot accurately predict how the interface will respond. Faced with unexpected results, we can only try to understand them and adjust our operations.

Usually, every system has a set of established interaction patterns. These patterns, to some extent, constrain what users can express and what they can do. We need to keep trying to find ways to express our intentions through the interface.

More importantly, every system has its own unique language. Even if two systems use the same words, the semantics behind those words may be completely different.

Therefore, whenever we encounter a new system, we need to go through a process of language learning.

This is somewhat similar to a child learning their native language, or an adult learning a second language. We keep trying, observe how the system responds, understand the meanings of different words and visual symbols, and then gradually master this language.

When we face a system we have never used before, we often make a great many attempts. We don't know what a certain button means, nor what result a certain visual element will produce; we can only build understanding through interaction.

So, what core elements does Semiotic Engineering actually contain? I would like to explain from three aspects.

First, human-computer interaction is a kind of mediated communication. Designers and developers convey their design intentions to users through an artificial interface language. Users communicate with the designer's agent by learning this language, and try to achieve their own goals.

Of course, users' goals are not always the goals envisioned in advance by the designer. Sometimes we creatively use an application to accomplish tasks the designer never anticipated, and these operations may even work very well.

Second, communication in the interface relies on three kinds of signs.

The first is static signs. They exist fixedly in the interface and can convey meaning simply through the way they are presented.

The second is dynamic signs. They need to convey meaning through changes over time and in state. For example, when we click a button, the change the button produces and the behavior that follows both tell us what this button means.

The third is metalinguistic signs. This is a technical term referring to signs that can explain other signs. For example, interface prompts can tell us what a certain graphical object represents, and can also guide us to take a certain action to achieve a specific goal.

Third, users have a strong capacity for language learning. Designers draw on existing interaction patterns, familiar language systems, and metalinguistic cues to help users understand new interface vocabulary and semantics.

Who exactly is talking to us? About four weeks ago, while I was using Claude to prepare this talk, something happened that surprised me greatly.

Claude's interface looks simple: it is a conversational interaction mode. We enter a question in the input box, and Claude gives an answer on the screen.

But in some cases, the interface also predicts what the user might type next based on the context of the current conversation, and proactively offers suggested replies.

That in itself is not strange. What truly surprised me was one specific conversation.

At the time, I was discussing some questions with Claude. It suddenly listed out a set of items, and the first item was actually what it believed to be the main thesis of the paper I wanted to express. The point it gave was roughly this: accountability mechanisms must be designed, and cannot simply be assumed to exist.

I was very surprised, because I had never thought of making this point the main thesis of my paper. But I found the idea interesting, so I followed its suggestion and continued the discussion. I told Claude that the argument it raised was good, and that we could discuss it further.

Unexpectedly, Claude soon corrected me. It said that one point needed to be clarified first: it had not proposed this paper thesis earlier; it had only been asking what I really wanted to express. Now, looking back over the whole conversation, it realized that this point seemed to have appeared.

So I pressed further: if it was not you who made this judgment, then who, or what, presented this point to me? Claude's answer was very interesting:

"I don't know. I cannot see your input box, so I cannot tell whether the application provided suggested replies, nor do I know what generated those replies or what context they were based on. By the time this message reached me, it looked as if you had said it. I have no way of knowing the actual situation. My earlier correction was accurate—I did not write that paper thesis in this conversation, but I also cannot know where this expression actually came from. I should have made that clear."

This is a very interesting conversation. I did not fabricate this example at all; in fact, even if I wanted to fabricate one, it would be hard to imagine such a scenario.

This experience made me realize that the AI dialogue systems we face may be far more complex than they appear on the surface. We think we are communicating with a system that can understand context, yet we may not know what other agents, mechanisms, or other sources of information behind the interface have participated in this conversation.

So, has semantic engineering lost its value? My conclusion is exactly the opposite. Semantic engineering has not lost its value; rather, it faces an even more difficult task. Because even today, the transmission of information between humans and AI can still be chaotic: we see a piece of information, but we may not be clear about where it actually came from.

We need to rethink: what information should the interface convey to users? Do users know whom they are communicating with? When a piece of information appears in the dialogue box, can we determine whether it comes from the user, the AI, or some other intermediate link?

Reading the literature is a daily part of our research work. We need to read a large number of papers, understand what one paper says, what contributions another paper makes, and what connections exist between a research question and existing work.

But now, there is simply too much for us to read.

I recall the number of submissions to this conference; it is said to exceed 8,000. Faced with such a huge number of papers, who can complete all the reading with only personal time and energy? In some cases, we do need to rely on AI. I myself also use AI.

But here there is a question worth taking seriously: when we delegate literature reading to AI, can we still retain our epistemic authority?

Epistemic authority is a philosophical concept with a specific meaning. Related discussion can be traced back to the work of scholars such as Linda Zagzebski, and other philosophers have further developed this concept since then.

Because as scientists and researchers, we must take responsibility for what we say and for the way we express our views. Our research carries social responsibility, and whether the knowledge claims we put forward hold up is not something that can be treated casually.

In order to make judgments, we rely on collective knowledge, on papers published by other researchers, on experiments they have conducted, and on the research results they have accumulated.

Perhaps these risks do not exist; perhaps they do exist. But if we do not have the corresponding mechanisms, it is hard to know just how great the risks are.

At our research institution, we developed a scheme called PEEL, short for Protocols for Epistemically Engaged AI Literacy. Its core goal is to try to establish connections among three elements: deterministic computation, artificial intelligence, and humans.

Why do we need deterministic computation?

Because one important reason we have long trusted computers is that they are deterministic: under the same conditions, they repeatedly perform the same operations.

The third, and most important, element is humans. Humans possess knowledge, intention, and the capacity for judgment. When faced with complex information, we need to weigh various factors comprehensively and ultimately decide what is credible and what holds up.

To test this approach, we chose Voyant as the deterministic computational tool and Claude as the AI tool, while humans are responsible for the final understanding and judgment.

PEEL is not a closed automated system. We hope researchers can access the relevant tools and participate in the entire process themselves. Next, I would like to show you how this method actually works.

In the PEEL workflow, researchers can first use AI to generate a text summary, and then use text analysis tools such as Voyant to compare the summary with the original text. We care not only about what the summary says, but also about examining what kind of relationship actually exists between it and the original text.

But it has a very important difference from ordinary AI summaries: we not only care about what the summary says, but also examine what kind of relationship actually exists between the summary and the original text. PEEL system introduction and experience entry

We once conducted an experiment: we gave the same text to different AI systems for summarization, including different versions of ChatGPT and Claude, and then compared the summaries they generated with the results generated by PEEL.

From a semiotic perspective, I would understand this kind of summary that preserves the features of the original text as much as possible as an icon of the original text. Here, icon does not refer to a visual image, but emphasizes that there is some perceptible similarity between the sign and its referent.

For example, if the original text is organized by chapters and sections, then the summary generated by PEEL will also preserve exactly the same chapter structure.

If the original text is written in the first person, the summary will also preserve first-person expression as much as possible.

In this way, the benefit is that when the summary preserves the structure and expression of the original text, we can more easily return to the corresponding position in the original text and check whether the summary accurately conveys the author's meaning.

We also use text analysis tools to compare semantic categories, keywords, and their frequencies of occurrence in the original text and in different summaries.

For example, we can observe how many times a certain term appears in the original text and how many times it appears in different summaries. We can also check whether a certain term appears in a summary but never appears in the original text. PEEL uses Voyant to compare the original text with multiple AI versions

(ChatGPT, Claude) in terms of differences in keyword connectivity

Do these changes necessarily mean there is a problem? Not necessarily.

If I ask you to summarize an article, you and the person sitting next to you will very likely give different summaries. You may, based on your own understanding, introduce some terms that are not directly used in the original text. This is normal, because everyone has their own knowledge background and way of interpretation. Similar things can happen with AI.

The real problem is that when we completely hand over the work of textual interpretation to AI, it may introduce related topics without constraint. These topics are connected to the original text, but they may not truly belong to the argument of the original text. Visualization generated by Voyant

You may claim in your own research that a certain author put forward a certain view, but in fact, this author never said so.

We can also use the visualization functions provided by Voyant to observe the semantic connections between the original text and different summaries. PEEL uses the visualization network graphs generated by Voyant to intuitively show changes in the semantic connections of different concepts in the summaries

In these visualization results, the original text constitutes our reference basis. Different summaries will present the new terms and new connections they each introduce, as well as changes in the original connections.

For example, some connections may appear for the first time in a summary, while other connections that originally existed in the original text may become more frequent in the summary or no longer as prominent.

These changes themselves cannot directly prove that there is a problem with the summary, but they can help us discover places worthy of further examination.

In PEEL, we also have a very important design idea: let AI become its own opponent.

We use the text visualization, traditional text analysis, and computational linguistics resources provided by Voyant to provide a review mechanism for AI-generated content. In other words, AI generates summaries, while text analysis tools provide users with resources to check the summaries.

For example, if we only want to confirm whether a certain paper contains a certain piece of information, then we only need to quickly browse the original text to confirm whether this information actually exists. We do not necessarily need to deeply study how the author reached this conclusion. If we want to use a certain view as an important basis for our own research, we need to carry out stricter verification.

This is what we call epistemic authority risk assessment.

I also want to introduce how we constrain the way Claude works. If I were to summarize the whole process, I would describe it this way:

"Use my language, but remind me that you do not know what you are talking about. Do not presume to use my language to make judgments on my behalf; instead, let me know that you do not truly understand the content you are processing."

This is a very important constraint. We hope Claude will process the original text as faithfully as possible, rather than continuously extending and supplementing it based on its own world knowledge. Its task is to compress and transform the text, not to complete all the interpretive work on behalf of the researcher.

A typical working environment for PEEL is Jupyter Notebook. Users can see the parameters, the code, and the specific analysis process, and can also modify the code according to their own needs.

If you disagree with my way of analyzing, that is completely fine. You can write your own analysis program and examine the text according to your own criteria.

We want the whole process to remain open. Because summary generation itself may also introduce bias, including bias brought by the prompts and bias brought by the way the text is analyzed. Users can not only modify the prompts, but also adjust the text analysis process to further examine where these biases come from.

In the specific presentation of summaries, we also provide different compression ratios.

For example, a 20-page paper can be compressed to 10% of the original length, or further compressed to 25% of the length. Users can compare whether the structure and main points of the original text are preserved under different degrees of compression.

Here is a very important feature: we preserve the voice of the original author as much as possible. If the original author writes "I argue," the summary will also preserve such first-person expressions as much as possible. In this way, we can quickly return to the corresponding position in the original text and check what argument the author actually made.

In addition, PEEL also classifies the different operations involved in the summarization process and assesses the risks they may bring.

For example, merging four sentences into one sentence usually carries a higher interpretive risk than directly repeating sentences from the original text. We also use direct quotations from the original text as much as possible, and through color marking and source links, allow users to trace the relevant content.

Another important feature is that users can participate in the summary generation process rather than passively accepting the results.

The initial semantic categories are proposed based on the distribution of words and lexical relations in the text, but users can modify them according to their own understanding.

In addition, we provide a large number of visualization results that allow users to compare the structure, semantics, and trends of change between the original text and the summary. Among them, I think a very important function is the Concordance in Voyant.

Suppose we want to study the discussion of "AI trust" in a paper. We can use concordance to find all the places where this term appears in the original text, and then check whether different summaries preserve this content and whether it still appears in the same context. Through these methods, we hope to increase users' confidence in the summary and also improve the quality of the summary itself.

Of course, PEEL is currently only a starting point. What we are addressing is only a very small problem that AI brings to human understanding of knowledge. Language itself is also only one kind of sign system.

For the multimedia researchers present here today, there is an even more worth-considering question:

When information is no longer conveyed through text, but expressed through sound, gestures, emotions, and other multimodal information, will we also encounter similar problems of cognitive authority?

My intuition is that we should perhaps explore both paths at the same time. In some scenarios, we do want AI to have stronger cognitive abilities than it does now; but in other scenarios, especially when research results may have major social impact, we should present and use the results of AI-assisted research more cautiously.

For example, the impact of the pandemic involves every person on Earth, and even the Earth itself. Such research has extremely major social significance. Therefore, we need to carefully assess the risks involved at this stage.

▎Question: How should we understand the meaning of the symbols in an interface? Does the meaning of these symbols depend on a single interface, or on a common reference system formed by a class of interfaces?

I would never dare to say which kind of interface is the best. The interface we are using now may be a good example: you use a language I can understand to communicate with me, but conversely, I may not be able to fully understand what you mean. You may also not be sure whether you have fully understood me.

Given that, how can we easily assert that a certain interface is the best?

What I really want to discuss is that interfaces should help us carry out better meaning negotiation.

When we cannot understand each other, we need to negotiate meaning through various strategies. This involves not only linguistic signs, but also other types of signs.

Do you remember the Claude case I mentioned earlier? At the time, I had no idea that Claude's conversation process might be mediated by other agents. Even more surprising, it did not know where this information came from either.

Therefore, we should clearly present this possibility in the interface. We need to design some kind of symbol, whether a textual prompt or a visual cue, to make users aware that the system may have such information sources and intermediate links. Because this is exactly where the risk arises.

As for what the best interface is, I still do not have a definite answer. I have always liked the design idea of letting users configure the interface themselves. Different users may need different ways of communicating, and the interface should allow them to adjust according to their own needs.

▎Question: Can PEEL detect hallucinations that appear when AI generates summaries? For example, the original text says 2+2=4, but the AI summarizes it as 2+2=5.

Some types of hallucinations can be detected.

However, I have not specifically analyzed hallucinations at present, but the materials and tools provided by PEEL already have the foundation for carrying out this kind of analysis. If you are interested, you can also make full use of these resources for research.

I would also like to add one point: many hallucinations actually come from the AI's world knowledge, rather than from language processing alone. Based on the background knowledge it possesses, the AI may believe that a certain paper is actually discussing another related topic, and therefore add to the summary some content that the original text does not cover.

This is also why we constrain Claude to process text in a way closer to machine translation, compressing and transforming the original text rather than freely improvising. The purpose of doing this is to let readers come into contact with a more condensed version of the text while still retaining their own interpretive work.

Of course, an excessively high degree of compression may also cause problems. Even if the summary itself contains no errors, readers may still misunderstand because the information is too condensed.

But I think this is a risk we need to bear when we choose to use this kind of tool. From the current experiments, when the compression ratio reaches about 25% or 30%, the text can already retain fairly good readability. Of course, this also means that the algorithm still has a lot of room for optimization, and the whole process still requires a great deal of work.

▎Question: As AI becomes able to complete more and more work, will this change reduce human creativity? In the past, we kept developing new tools to complete different tasks; now, it seems we can directly use AI to generate almost anything. Will this affect technological innovation?

I do not think this will necessarily reduce creativity.

I like to think about this question using the concept of infinity. We can have infinitely many things of the same type, but that does not mean we can only have one way of creating.

If creativity means jumping from one thing to another completely different thing, then we do indeed need to be vigilant: we may be generating infinitely many things of the same type, rather than continuously entering new domains.

In 2024, an article published in Nature by relevant researchers discussed the problem that scientific research may become homogenized. More and more people are beginning to organize their research according to the tasks that AI can complete, and researchers may also gradually adjust their work into forms suitable for AI processing.

However, I am still willing to believe that AI can be used in dozens, hundreds, or even thousands of different ways. Why must we choose the same way of using it? In my view, there is another even more important issue, namely productivity, especially productivity in scientific research.

Nowadays, people often measure researchers' performance by how many papers they have published. AI is serving this productivity cycle, and this may be more worthy of attention than the impact of AI itself on creativity. If we can temporarily step out of this productivity cycle, I think AI will show many surprising ways of being used.

I am very fortunate that I am now retired and can, to some extent, step out of this cycle. Under such circumstances, we may be able to truly explore the different possibilities of AI, rather than merely letting it help us complete existing work faster. This is the direction I would rather see.

ACM MM is currently taking place. Researchers, students, and readers who follow the frontiers of AI are welcome to join our exchange group.

Here, you can meet peers, exchange research, share admissions information, scientific research cooperation, meal invitations, submissions, team formation for attending the conference, and so on, and look for potential partners.

During the conference, we will also continue to bring on-site report updates, interpretations of guests' speeches, summaries of frontier directions, and highlights of the forums.

See you at the venue, and see you in the group.

? Group entry link: Scan the QR code to join the group or add WeChat A17648278904, with the note: ACMMM + institution/school + name + direction. Without the authorization of "AI Tech Review", it is strictly prohibited to reprint in any way on websites, forums, or communities!

For reprinting from the official account, please first leave a message on the "AI Tech Review" backend to obtain authorization. When reprinting, the source must be indicated and the official account business card must be inserted.article_adlist[

Leiphone original article, unauthorized reprinting is prohibited. For details, see the reprinting instructions. ]article_adlist Statement: Sina.com exclusive manuscript, unauthorized reprinting is prohibited. tech_web_article_v2015_block_bottom non-targeted 300*250 button 17/09 wenjing begin

non-targeted 300*250 button end

Source

Sina Tech