Three questions product managers must answer before launching an AI feature

"We need to add an AI feature too"—with one word from the boss, the product manager starts choosing models, writing prompts, and designing interactions. After launch, the click-through rate is decent, retention hasn't changed, and user feedback says "the answers aren't very accurate." The problem isn't execution; it's that three questions weren't asked clearly before starting: What step exactly are users paying a price for? What happens if the AI gets it wrong? How do we prove the AI does it better? Once you've asked these three questions, it's not too late to start.

The source may be that the boss got provoked at an industry conference, or a competitor just launched an intelligent assistant, or the technical team got a cool demo working. When the request is made, what you're given is one sentence and a deadline.

Then the product manager's path dependency begins: which model to choose, how to write the prompt, where to put the entry point, what the interaction should look like.

When the feature is built and launched, you look at the data: the click-through rate is decent, retention hasn't changed, and user feedback is full of "the answers aren't very accurate." Only in the retrospective do you realize that execution wasn't the problem—many questions simply hadn't been figured out before starting.

So before kicking off the project, at least the following three questions need to be clarified:

Question one: What step exactly are users paying a price for?

—Sounds fine, but it's the wrong way to ask.

The real question is: In this scenario, at which specific step do users get stuck? What price does this step cost them now?

It could be time, such as spending forty minutes writing a weekly report.

It could be a barrier, such as not being able to get data without knowing how to write SQL.

It could also be a psychological burden, such as not knowing where to start when facing a blank document.

A very practical test is: remove the word "AI" and describe the value of this feature in one sentence. "Help users compress a 40-minute weekly report into 5 minutes" holds up. "Use AI to improve the writing experience" does not, because it describes a means, not value.

That's not all—keep asking: Can traditional solutions address this cost?

The logic of traditional features is deterministic: same input, same output; if something's wrong, it's a bug, and you fix it.

So change the way you ask: not "can it get it right," but "if it gets it wrong, how serious are the consequences."

You can break this down from three angles:

Whether errors can be discovered in time: If the output is a piece of text that users can immediately read and judge, the risk of error is relatively controllable. If the output is a result hidden deep in a process and only exposed days later, that's more dangerous.

Whether there's a way back after an error: Can users undo with one click, can they edit manually, can they return to the original process. Good AI products all leave a dignified exit for "wrong," such as clearly labeling "AI-generated, please verify," keeping an edit entry point, and proactively saying "I'm not sure" when confidence is low.

Only after thinking these through will you know whether this feature should be built as "automatic execution" or "give a suggestion for the user to confirm."

This is the question most easily skipped.

Traditional features have ready-made acceptance methods: buttons can be clicked, processes can run through, data can be stored in the database.

If this isn't defined before starting, after launch you'll fall into two awkward situations: either everyone evaluates by feel and people talk past each other in meetings; or you only look at clicks and usage counts without knowing whether users actually got value.

So from the very beginning, we need to lock these down:

Write out the standard for "good": It doesn't need to be very refined at first—even just five judgment rules: facts must not be wrong, tone must fit the scenario, length must be within a certain range, sensitive content must not appear, and the format must be directly usable. With this, evaluation has a common language.

Prepare a batch of real samples: Find fifty to one hundred inputs from real scenarios, including typical, edge-case, and tricky ones. Every time you adjust the model or prompt, run this batch of samples and compare the differences before and after. An AI product without a sample set is basically optimizing by luck.

Design the post-launch feedback loop: Did users adopt the output, how much did they change it, did they click "unsatisfied," did they regenerate. These behavioral signals explain the problem better than mere usage counts. Adoption rate and modification rate are often much more honest than click-through rate.

If a feature can't clearly explain "how do we count it as done well," then it has already lost its direction for iteration at kickoff.

After the three questions, then talk about how to do it

Looking back at these three questions, they actually correspond to three basic product judgments: whether it should be done, whether it can be done with confidence, and whether it's done well.

They aren't new; they've just been amplified in the AI era. Because AI's capability boundaries are blurry, outputs are uncertain, and cost and effectiveness are changing rapidly, teams can easily be led along by technological possibilities and forget to return to the three origin points: users, risk, and results.

Interestingly, after thinking these three things through, many "must add AI" requirements shrink on their own—some are replaced by rule-based solutions, and some are narrowed into a narrower, more stable scenario. And those that remain have clear goals, clear boundaries, and evidence-based acceptance, so they're actually faster to build.

First ask clearly: What is the user's cost, what happens if it's wrong, and how do we prove it's good.

Once you've asked these three questions, it's not too late to start.

This article comes from the WeChat public account "Everyone Is a Product Manager" (ID: woshipm), author: Hemu, published by 36Kr with authorization.