Mike Rowe continues to make the rounds advocating for young people to consider careers in trade professions due to the way AI is shaping the economy and the job market. He’s not wrong, and product teams know it as well as anyone. Are they all jumping ship to become plumbers and electricians? No, but every product team and product leader I know currently has one question hanging over their day-to-day work: How to make this or that step of the product design and development process more efficient with AI.
It seems like every step in the process can benefit from automation: assessing the data, drafting the brief, mocking up concepts, and writing the code. AI can help you steer a course, too: it’s very good at analysis and synthesis of large sets of data. But that source data only records what users did inside your product. It says nothing about why, and nothing about the steps they take outside it to get their job done. AI collapsed the cost of making the thing. It did nothing to the cost of being wrong about the thing, because some wrong turns never appear in the data at all. They only surface one way: talking directly to customers about how they use your product.
The customer conversation AI can’t facilitate
In a recent customer interview I did for a company that runs a healthcare payment reconciliation platform, the customer showed me how she tracks reconciliation completions in a spreadsheet outside of the company’s software. That a customer felt like they had to do this manual step was news to the company founder, and something that only came to light because I sat face-to-face on a video call with the customer to talk to her about her process. Talking to customers remains a step that is best done by real people.
AI collapsed the cost of making the thing. It did nothing to the cost of being wrong about the thing.
Can’t a bot just ask them?
There may come a day when the tech gets good enough, or we get used to it enough, that people feel comfortable engaging with a bot in a genuine way. Services already exist on both sides of the table: AI facilitators that run customer interviews, and AI mock customers that software owners can query for insights. My take: Yikes. The parody “AI job interview” reels making the rounds are funny because the failure is real. Fake facilitators ask follow-up questions that march along a trained path. Humans can entertain a rabbit hole for as long as is necessary. Fake participants are worse. A synthetic customer can tell you what typically happens. It can never tell you what actually happened to a real user in your product last Tuesday, or the one-off anomaly you most need to hear about.
The customers you already hear from are not the whole story
You might be saying, “I hear from my customers plenty.” You have that one demanding customer (or two or three) that you wish you heard from less. So why pursue intentional customer input via formal User Experience Research? Because customers of B2B software quickly become power users. Many of them have adopted manual processes (and even work-arounds) that have become second nature to them. Like the manual spreadsheet check from my earlier example, what was once a pain or just a mild frustration has faded into the background as they go about their day job. Speaking directly with them, inviting them to “bring you along for the ride,” is the reliable way you’ll uncover these opportunities to wow them with your product.
How do I start?
Invite twelve customers to a one-hour conversation with you, with the goal of six to eight saying yes. If you have multiple kinds of users (called personas), treat each as its own group of six to eight, and talk to each group separately. Start with the persona you’re most curious about, or the one whose experience you most suspect needs improving. Consider incentivizing customers to say yes. A common benchmark is a $100 gift card for an hour of a professional’s time. I’ve also seen product teams offer early access to future product features instead, which some customers value more than the money.
Why only six to eight?
Because this kind of qualitative study is not aiming for statistical significance. You are trying to uncover novel pains and frustrations from your users, and research has shown that beyond about a half-dozen conversations, the returns on fresh insight diminish significantly. So why six to eight, and not exactly six? Because a couple of the conversations will go sideways, and the last one or two mostly confirm the themes you have already heard. That confirmation is worth having. It is how you know you found a pattern and not just a story.
Running the conversation
On the day of the interview, have a simple guide you can follow, so you pursue the same line of questioning with each customer (AI is great at writing research facilitation guides!). A simple formula: ask customers to walk you through a “day in the life” scenario. Before your software? During? After? What do they look at first? Have them share their screen and show you what it is like when everything goes as planned. Then ask about a specific recent time when it didn’t go as planned. What happened? Where did they get stuck? What did they do? Good facilitators ask several layers of follow-up questions, even when the story sounds simple on the surface.
Making sense of what you heard
Ensure all interviews are recorded and transcribed. The transcribed interviews become the source for data analysis, synthesis, and theme generation. AI has gotten very good at this step but you still want to thoroughly vet the AI synthesis to assess whether its conclusions jibe with what you know you heard in the interviews. What you’re ultimately looking for: Recurring pain points or frustrations that seemed to be themes across multiple users or stand-out anomaly issues that, though expressed by only one or two participants, you know are major issues you want to address in the product.
What to do with what I heard?
With your themes in hand, you and your product team will rank them by severity. If fixing item 3 represents a month’s worth of work and item 1 represents almost a complete rebuild then you simply have a judgment call about which one you go after first, second, and third.
You’ve traded the exhausting guessing game for reality. When you’ve sat across a screen and watched a real person wrestle with a workflow, you stop wondering what to build or which feature should come next. Now you know.
AI can build the product, and it will keep getting better at it. What it cannot do is sit with your customers and learn what the product should be. That job remains stubbornly, usefully human, and it belongs to whoever is willing to spend an hour asking.