Since unmoderated study tools like Validately and UserTesting became widely available, they have helped many functions on a team and designers like me evaluate options and obtain early feedback on specific aspects of a product.
It has always been eye-opening to uncover issues by watching testers navigate through screens and mocks, and listening to their feedback. However, two parts of the process were particularly time-consuming for me:
-
Before the sessions: writing up an easy-to-follow script without grammar mistakes
-
After the sessions: comprehending fully and extracting verbatim quotes from the video recordings
I encountered testers who kindly corrected my grammar mistakes and struggled to make sense of my overly complex instructions, despite my spending days preparing the scripts and often having them reviewed. These issues were caught during pilot tests, but I hoped to identify them even earlier. Moreover, when it came time to document findings, taking notes from the recordings and identifying important contextual comments by the testers required me to re-watch the videos numerous times. The quality of machine-generated transcripts was not great two years ago and earlier, complicating this task further.
Now, as I prepare study scripts, I keep ChatGPT open alongside my document. For each task, I check grammar and explore better ways to phrase things. Whenever ChatGPT suggests a correction, I make sure the key points remain intact and then iterate several times to refine them into clear and simple instructions. Just like I write this post, paragraph by paragraph, with ChatGPT's help. Similar tools, such as Claude and Bard, would offer the same assistance (in case you run out of quota on one).
On to the transcript feature, I’ve only used one study tool recently (Userbrain), but this likely applies to other tools as well. With the integration of the latest language model, I've found that the transcripts of session recordings are incredibly accurate, capturing words that I might have missed with my own hearing. It’s a completely enhanced starting point for reviewing. Now, I can watch all the recordings at 2x speed to spot obvious issue areas first, and carefully read the transcripts to not miss important comments and get verbatim quotes for documentation. Then, I watch them all at 1x speed to fully understand again. It’s quite delightful.
In addition to transcription, the AI analysis provided by study tools is likely to improve over time, and the tools may eventually provide co-pilots for me to ask questions and create highlight reels. But even before we reach that level of advancement, the process improvements enabled by AI are already sufficient to allow running more studies more efficiently and enjoyably.