Why PMs should stop conducting user interviews like a UX researcher and start conducting them like a PM

I played around with more provocative titles than the one I landed on:

• Stop conducting user interviews like a UX researcher
• Get the most out of user interviews with these 3 tips
• What every Product Manager gets wrong about user interviews

A year or so ago, I wrote on of those “how to work with me” docs. One of the things in it (that I stole from someone) is “I assume you can do your job better than I can do your job”. Which is why I ultimately landed on this title. The thing is… structured user interviews are great for a lot of things. The best UX researchers I’ve worked with (many at Twitch) have been able to furnish incredibly actionable insights that have turned into lovable features. The problem is… I’m not a UX researcher, I’m a PM; our goals are aligned but they’re not the same.

This meme is an homage to the last convention I went to before the pandemic started. I brought our newest designer to PAX East. He was going to be focused on our mobile app (back when we thought IRL streaming was going to be a big thing, thanks 2020). I told him I wanted him to hear the problems from the streamers directly. By doing so I basically broke every rule of user research. At the end of the con, he said to me… You’re basically like Denzel in Training Day. At the time, I didn’t think much of it; it was just how I did user interviews. But, I realized as I kept working in product, that many other PMs avoided interviews like the plague. Talking to the customer was the realm of UXR or PMM. PMs were supposed to be unbiased and data oriented. If you couldn’t A/B test it, it wasn’t worth building.

I’ll leave the mistakes PMs making over-relying on A/B tests for another post, we’re here to talk about listening to the customer. Let’s frame the conversation by talking about the type of interview we’re ALL familiar with, job interviews.

How job interviews are JUST like user interviews

So there are two primary types of job interviews:

• Unstructured Interviews — The type where the interviewer asks you (and any other candidate) questions they come up with on the spot. They probably know what they want to get a sense of from you, but there aren’t the same questions to everyone.
• Structured Interviews — If you’ve gone for one of the big tech firms, you’ve encountered this type of interview. They ask the same questions of each candidate. Often the panel has specific things focus areas each panelist is trying to get and there’s trust in the collective answer to be the “right” one.

I always thought this was weird. Isn’t there a “best” type of interview? The goal of all job interviews are the same right? Hire the best candidate. It turns out it’s not that straight forward.

The best result you can get out of both interviews are actually very different. The benefit of the structured interview process is that you do get the “best” candidate from the criteria you set forth to your interviewers. After all they’re asking the same questions of each candidate and grading each candidate in about the same way (we’ll pretend there’s no inherent bias for time’s sake). But if this is the case? why would someone give unstructured interviews (besides pure laziness)?

It turns out that unstructured interviews are great at pulling out red herrings. If you’ve ever prepped for a consulting interview you’re probably told there are three things they evaluate you on:

1. Can you do the job?
2. Can I put you in front of a client?
3. Would I be ok being stuck in a room with you for 24 hours?

Now, #1 can be sorted out in a structured interview, but 2 and 3 really can’t be. Who knows if you’re going to say the off color joke? Or engage in “locker room talk”? You’ll never give that response in a structured interivew. But in an unstructured interview, while you have the interviewing laughing at your last joke and your guard is down? that’s when people slip up. That’s why in consulting you NEED red herring interviews and why they aren’t so bad in the product world either.

Alright Sam, there’s two types of interviews, what does this have to do with product management?

The way most people get introduced to structured UX research is through interviews. You’re told to conduct them with a script. You don’t deviate too much from the script (as much as you can). Don’t ask leading questions. Don’t guide the interviewee. And NEVER insert yourself into the conversation.

These interviews are great for learning certain things. They’re really great when you’re testing a hypothesis or looking for usability. The observational style interview is basically the equivalent of a natural experiment to an economist. But they aren’t great for the stuff you don’t know. In fact, most of the time they’re akin to the biggest faux pas in product management, asking the customer what they want.

The fact is, your users exist beyond the the world that you’ve created in your nice structured interview script. Their world is inhabited by so many things. They’re the student who forgot to push send on gmail for the essay that’s worth half their grade, they’re the parent who’s late to work because Sophia found the candy last night and she didn’t sleep until 5, he’s the guy who’s car has a flat, and she’s the girl who forgot her BART card in her other pants. But no interview script can account for all of this.

It’s easy to forget that we call user journeys, “journeys” for a reason. The reality is that there’s a huge gap between empathy and knowledge. We need both, but probably only have time to get one.

Ok, I accept your premise… But what should I do about it?

First off, don’t stop the structured interview train. You’re still going to get some great answers from them. At the end of the day though, structured interviews are better for validation than they are for inspiration.

There are two large buckets of things you should be focused on adapting so you can get the most out of unstructured interviews (heck, by the end you might even like doing them).

Learn to ask better questions, better yet, get your customer to ask the questions

I used to travel to Seattle for work when I was at Twitch. About half my team was there so I always wanted to have more face time with them. But never one to miss an opportunity to talk to customers, I usually tried to bring streamers into the office, give my engineers exposure to streamers and give the streamers some snacks (streamers love snacks).

So I brought about 4–5 streamers in that day, and after a quick raid of our snack area, I put them into a conference room and let them at it. At the time, we had just launched community gifting and the leadership team was basically like “gimme more gifting products”. So we started talking about how it was being used in their communities. We got to the point, where the streamers started to ask each other questions. “Why did your mod do this?”“don’t you hate it when viewers do ___?”

Amidst all the conversations a particularly snack oriented streamer (she knows who she is) started talking about why one of her mods hated use gifting in channels. It was a passing mention as they went on to talk about how they were using gifting to run a contest. But we backed up. It turned out that one of her mods really disliked the attention he got when he used gifting. It attracted the wrong sort of attention to him. He’d immediately get DMs from viewers who wanted a gift sub, and streamers who wanted him to come to their channel and spend money.

It wasn’t on my radar at all, in fact, most of the questions we asked where how do we create more celebration and spectacle from a gift. But this short conversation became what is now anonymous gifting.

Or as I like to call it… “Ghost Subs”

It wasn’t a wildly successful product. The usage wasn’t as big as other features. But it was almost entirely incremental. We found a demographic of user that just never had their needs served before. There was no way to ask the question “what’s something you wish you had with gifting that only you would want?” and get this answer.

Learn to listen better, it’s not the users’ job to generalize, it’s yours

Every PM has heard “but is that generalizable?”. It’s not even a question sometimes; it’s a straight up challenge. The more you listen for specific issues your customers have, the more you’re likely to find something that’s generalizable to all your users.

When I first started exploring mobile for Twitch, we had this hypothesis that streamers could use their phones as a “second screen”. So we conducted a bunch of interviews with users to learn about their workflow and how their cell phone played a part in it.

I won’t bore you with all the details, but we had one streamer who talked about how they had 3 screens in their setup. 2 horizontal and 1 vertical. they had OBS on one, chat and messaging on another, and the third was a utility screen (usually it had Spotify or something like that). We got through the interview and they said something interested “oh, I’m gonna get a 4th screen soon”. So I asked the only question that made sense to me at the time “what are you going to put on it?”, he said “I dunno, I’m getting it to make my stream setup better.”

Now to disabuse all of you of the notion that streamers are all rich, they aren’t. You’re more likely to find a fancy setup in your friend’s basement than you are of a streamer you picked out of a hat. So why would someone drop a couple hundred bucks they probably didn’t have on something they didn’t need (they couldn’t even tell me what they were using it for). I went on with my life and I was recounted this story to another PM on my team. And it hit me, he was proud that he had made enough money streaming to buy a new monitor.

For those who don’t follow Twitch, (I suspect most of you), streamers typically have two numbers to tell them how good they’re doing (and one of them they usually ignore). They know how much money they have coming in (although most probably ignore it) and they know how many viewers they have. The two signs that they are succeeding are when those two numbers are up and to the right. But for many streamers (especially those starting out) those numbers tend to be flat. So are you doing good? 🤷‍♂. But if you made enough money streaming to afford that new mic, monitor, camera, keyboard, or graphics card… That was progress.

June of this year (2023 for those counting) Twitch launched Creator Home, easily 3–4 years in the making, I was working on this before I left (it’s probably changed a bunch since then, so this really isn’t about taking credit). But a huge factor in some of the features found in home come directly from realizing that streamers don’t have enough things to tell them whether they’re on the right track. So they had to rely on buying equipment they don’t need. This one insight didn’t birth this product, but it certainly was a key input when defining the goals.

So, I don’t work for Twitch, my customers aren’t streamers, what’s my takeaway?

In an early episode of the Brene Brown podcast, she had Sue Monk Kidd on the show (sadly I stopped following the show when it went to Spotify, sorry if I can’t use Overcast, I’m probably not listening to it). She said something I had to immediately write down:

“The deeper we go into our own experience, our own journey, the more likely we are to hit the universal.”

Now she probably meant it for something far more profound than user interviews. But the reality is that has always stuck with me. The more specific I got with user problems, the deeper I dove, the more likely I was to find this nugget of gold. Sure, it didn’t come in the form of a product. I had to polish it and chisel away at it until it was something that could be turned into a product. But it always came from a problem a user had. The best ones usually came from a problem the user thought “I’m the only one with this problem.”

Next time you have to do interviews try doing the following:

1. Let the customer go off on a tangent [Humanize your customer]
2. Share something that frustrates you about your own product [Humanize yourself]
3. Ask two (or 5) more questions than you think you need to [Dig for the nugget of wisdom]

Use your user interviews as a way to learn something unique about your customer. You’re not going to find gold in every interview. But you’re guaranteed not to find it when you’re just keeping everything surface level. Don’t expect your users to do the job for you, let them be great at their job (being themselves) and work on being great at your job (solving the problems they didn’t even know they had).