On Conducting Interviews: A guide for UX and HAI Researchers

I had an awful experience interviewing one of my dissertation research participants. Now, I have conducted more interviews during my PhD than my bank balance, and most go well. But again, it was distasteful that I had to call my advisor about the way forward :(

To give some context: we had 3 phases of my research program — (1) they learned about using LLMs for data sensemaking, (2) then we co-designed a learning experience, and (3) they implemented a lesson with their students. Immediately after each phase, we were supposed to have a reflection (semi-structured interview) session. However, due to the erratic schedule of the teacher, we could not have this reflection interview immediately after the session. And that was the genesis of the problem.

Now, I had some questions prepped, but on starting out, I was told she couldn’t recall anything from the session. Damn!

Brute-Force Interviewing

As a researcher, documentation is key to one’s scholarship, and I had that. I have an organized, time-stamped Google Sheet repository of all our conversations and data of all sorts — videos, documents, notes, slides, etc. In that moment, I quickly brought some of this data and shared it to elicit her memory.

Guess what? She said, “That wouldn’t help either.” Ok, this is bad, and I take responsibility. But I had to get something out of this planned session, else I might never regain this time.

Then I made this move next: brute-force interviewing. I had a set of questions and needed some specific answers, and I had to use what I knew from my recent conversation with her to elicit those answers.

So, assume I wanted to initially ask:

“How did you plan to manage challenges students may experience using AI during your implementation?”

She had told me she couldn’t remember planning for it. But I remembered we did plan for it. Also, I remembered that in my last conversation with her, she’d taught her students how to re-prompt the LLM to check for consistency in the output.

So, I asked instead:

“During your implementation, I noticed your students reported that Gemini hallucinated and made up data. Did you anticipate that, and how did you plan for it?”

Now she said, “Yes, I actually had to re-prompt…”

Basically, that was how I extracted as much as I could within the limits of what I remembered and the notes I’d taken of her experience. And yes, I piggy-backed her responses too. I’ll talk about this in detail below.

Lessons for a Well-Rounded Interview

But how best could a well-rounded interview have gone, even in the event of an unplanned situation like my experience? Here are some ideas and things to consider or do.

1. Document Everything

Make sure to collect all data (in whatever format), whether directly related or indirectly. Create a dedicated repository, add timestamps, add descriptions of what the data is about, and add your notes about what happened during the data collection. The addition of notes could happen after the session. And do not forget to add a unique identifier for the participant.

2. Start with Free Response

In the event you want to catch as much information as possible, start by allowing the participant to “share their experience.” Now, that sounds loose, but if you are engaged with their talk, you can steer it. As researchers, we often limit what we can learn from participants because we stick with our routine, prepared questions. However, our questions are our questions — we don’t know their mind or what they have answers to, even if we know the context of their experience. Err on the side of, say, giving them the first 15 minutes to exhaust talking about their experience. Then you can start going through your questions.

But careful here! You may just be asking questions they already talked about.

2.1 Steering the free response. I recall interviewing a teacher who talked for so long on the same question that they lost track of the actual question. To ensure they don’t talk off-course, courteously interrupt. For example, start with “hmmmmm…” along with some hand or body gestures demonstrating that you want to talk, then steer the conversation where you want. You say “hmmm” (or some pensive murmur) because that isn’t necessarily interrupting them (which could be rude if you did), but it gets their attention for a microsecond, and your body gesture adds a bit more time for you to then speak. To steer the talk, ensure you don’t completely veer off from what they’re saying, else, that would be jarring.

3. Piggy-Back on Their Words

My advisor, Dr. Michelle Jordan, would say “Yes, and…” and then add on to what you’re saying. Even when what she’s saying is an argument against what you said, the “yes” makes it benign. So try that: “Hmmm… yes, and… I was wondering what you think about…”. You interrupted courteously, then seemed to agree with them (validating them), then led them to your question.

Piggy-backing is meant to connect and flow. It also helps when you feel you’re not getting the responses you want, or you’re running out of questions to ask (yes, that happens and your prepared questions may not get you what you want eventually). You can use piggy-backing to cough up new, interesting, and insight-generating questions.

Assume the participant says, “So, I ensured they re-prompted Gemini to get multiple responses, as that helps them develop critical thinking skills and deep learning.” And assume I’d already run out of questions. My next piggy-backing move would be to pick one keyword from what they said and build on it: “Speaking of ‘critical thinking,’ can you share how re-prompting Gemini helps with that?” All you have is your attention currency, and you can buy anything from the participant with it.

4. Make Your Participant the Expert

In my experience, if you project someone as an “expert” or as someone who has something to say, they will tell you good stuff. You don’t want to shine or sound instructive. Make them shine. In my opinion, teachers are among the brightest, quick-witted, and slightly egotistical professionals you’ll interview (disclaimer: everybody has some ego). While some won’t object to feeling instructed, others will be point-blank, tacit, or unresponsive. “What do you think?”, “I think…”, “What if we…”. Use uncertainty markers like these. Yes, there are instances where you have to be the expert, such as when they ask you about your domain — go for it.

+1. We often find ourselves asking questions that are repetitive. Sometimes your participant will pre-empt or answer a future question as part of their current response, and by the time you get to that question, you may not recall they already answered it. That can leave a bad taste, as though you aren’t on top of your game or not listening.

The solution is listening. No, wait. First, read all your questions out loud to yourself at least twice before the interview. Then, during the interview, listen. Scan your questions for ones they’re already answering, and strike them out or jot a quick note. Not long notes; you don’t have that kind of time. Shorthand or even gibberish is fine, as long as it tells you that they were talking about gmnii & reprmtin.

Alright, that ends it all. Let me know your thoughts.