Please note, to comply with my non-disclosure agreement, I have omitted and obfuscated confidential information. All information in this case study is my own and does not necessarily reflect the views of Premise or its employees.
TLDR
(Too Long Didn’t Read)
How might we engage more users to map places while ensuring quality data and their safety? This research provided insight into why and how mobile app users were completing paid tasks to map points of interest. Those insights prompted improvements to the task design and growth strategy while building empathy for users across the cross-functional team.
I managed this project from end to end. That role gave me many opportunities: managing cross-continental recruitment, leveraging artifacts in user interviews, building rigorous qualitative analysis foundations through grounded coding, and storytelling through user profiles to engage cross-functional stakeholders in the findings and provide inspiration for action.
Premise is a two-sided marketplace: 1) The Premise mobile app pays users to submit sentiment (surveys), observation (photos), and location based data; 2) Customers pay Premise for data. Premise’s data collection includes mapping points of interest by tasking users to go to specific points (e.g, a business, facility, or structure), verify the location and operational status, and take some photos.
This research built an understanding of the profiles of the people likely to engage in points of interest data collection tasks—their motivations for completing tasks and completing them well and the challenges they face. It fostered empathy for users while highlighting opportunities to improve the task design and growth strategy.
Overview
Timeline
September-October 2021
Role
Sole UX researcher
1. When do users who engage in mapping tasks use Premise?
2. Why do users who engage in mapping tasks choose to use Premise or other apps, if applicable, for flexible income?
3. How do users who engage in mapping tasks select which Premise tasks to complete?
4. What challenges do users who engage in mapping tasks face in completing tasks?
Research Questions
Process
Interviews:
Leveraging user logs with SQL, I identified and recruited highly engaged users from two priority, contrasting markets to be able to understand why people chose to engage across diverse environments.
7 users participated in 30 minute semi-structured interviews. This pool was smaller than I originally intended but sufficient to provide some actionable insights based on recurring themes between them. However, there were also interesting threads in individual interviews that may have gained more robustness with additional participants if feasible — opportunities for further research.
The interviews focused on their background and experience with Premise, especially in completing mapping tasks. With participants’ consent, I shared an artifact of their experience with mapping tasks—photos they had recently submitted to the Premise app—to prompt more specific recollections. Participants shared their context for completing the task from logistics to decision points and how they felt about the experience.
Analysis & Synthesis:
I completed two cycles of grounded coding of the interview transcripts and developed an accompanying codebook using MaxQDA.
The analysis also drew on demographics of the participants collected through a survey as part of their Premise account creation.
Sharing & Socializing:
Alongside a traditional paper and deck, I developed four user profiles with the goal of generating empathy and making the users’ experience more memorable. In contrast to personas, the profiles conflated similar users’ backgrounds based on emerging themes from the interview data and anonymized real stories. The goal was to leverage the empathy-building value of personas while avoiding potential bias or stereotype-based characterizations, especially with limited data. The user profile cards included descriptions of the profiles’ gender, age, economic situation, why they use Premise, how they use Premise, and how they understand the purpose of Premise’s data collection, as well as a narrative about completing a task.
I distributed the user profile cards to team members in advance of the cross-functional research debrief and brainstorm. Attendees presented a user profile during the group session and adopted the perspective of that profile for initial brainstorming before adding ideas to address others’ experiences.
Findings
The research contrasted user profiles across four key interrelated spectrums with practical insights related to each:
Reliance on Premise
Risk-taking
Impact of uncertainty
Grasp of the purpose of the data collection
As a sufficiently opaque summary for public consumption: those with high reliance tended to be willing to endure more uncomfortable or risky situations (e.g. potential phone theft), and felt the impact of uncertainties in the experience more acutely.
Practically:
The stories of risk and reliance built understanding and empathy for users among the cross-functional team members.
The elements of the experience which created uncertainty that emerged in the interviews highlighted actionable opportunities to improve the task design and growth strategy for user profiles across the spectrums.
Users had little knowledge or concern about the use of the data collected, though the impressions contrasted between markets. While the purpose of the data had previously been a growth tactic, this emerging theme suggested that might not have reached or resonated with users — but called for quantitative research which could evaluate the generalizability.
“It’s so easy to fool ourselves into thinking we know how people are experiencing what we’re putting out there—so many unexpected insights!”
An example Contributor profile card
Impact
Sharing the stories of the on-the-ground challenges dedicated users faced helped to inspire more innovative thinking about how to optimize quality data while not penalizing users for circumstances outside of their control. For example, the team changed elements of the task which required to incentivized options with the understanding that in some situations, users truly could not meet those requirements.
The users’ experiences improved the task targeting strategy, including clustering points of interest available in the app for efficient travel and easing users’ uncertainty about task availability through additional communication.
The research motivated additional investment to expedite and clarify the task review process, so users could know more quickly and clearly if their task would not be the standard to be paid and why.
This project focused on a specific suite of Premise tasks. Another Premise research project leveraged the design and interview guide for research about a different suite of Premise tasks a year later.
Learnings
Using artifacts as a prompt in interviews was an effective tool for guiding the conversation into specific, colorful, stories that I have adapted for the design of other interview-based research since.
Two cycles of grounded coding for each transcript was more rigorous analysis than typically has been feasible in my industry experience. I completed this project as part of my research methods coursework at the UC Berkeley School of Information while consulting for Premise, so I put in that extra effort on my own student time. While that’s not always feasible or critical to achieve minimum viable rigor in a business context, it helped build my foundation for sensemaking of qualitative data. It was a perfect opportunity to practice the most rigorous approach to build the aptitude for more expedient analysis.
I was not an embedded member of the cross-functional team and should have taken more of a role in ensuring ownership of next steps. In the brainstorm I bundled with the research debrief, the cross-functional team generated actionable ideas, but they didn’t have an owner to usher them forward on the team. In similar sessions since, I have made it my practice to allocate time for deciding on action items with owners before we part ways — sometimes a portion of the discussion a product manager has preferred to facilitate and sometimes a portion I have owned myself, depending on the team.
