Leveraging a beta program to learn from users
Timeline:
Role:
Description:
July 2023 - September 2023
UX Designer, UX Researcher
Designing a beta program for SMART Technologies’ education technology web product, Lumio, in order to quickly gather user feedback.
The Problem
Lumio is a web-based learning platform for teachers to connect and engage students with interactive lessons.
Lumio wanted a way to quickly test and gather feedback on features that may not be ready for a full release. This spurred the development of a beta program. One of the most important parts of a beta program is gathering user feedback. We wanted to optimize how we collected user feedback to maximize the amount and quality of feedback we were receiving.
Requirements
I received from the Product Manager (PM) some requirements or goals for the beta program.
-
An in product beta program - not a separate client, instance, or site.
-
An opt in beta program - beta features are not available for all users
Competitive Analysis
I looked at existing products with beta programs, trying to focus on beta programs that fell within the requirements I received from the PM. I reviewed Github, Miro, and Steam, as well as a previous public beta item in Lumio.
Github allows users to opt into their beta program per feature. A user is able to choose which beta features they want access to and which they do not. Upon opt out, a user is prompted to provide feedback on the specific feature.
Miro shows beta features to all users. After using a beta feature a toast message is shown to users so they can rate the feature and an optional longer form text box shows after.
Steam has a separate client for their beta. Users have to opt in within the settings menu, and then download and re-launch the new client.
Lumio’s existing beta showed the beta item to all users and marked the beta items with a purple tag. The only place to provide feedback was in the avatar menu in the upper right hand corner of the screen. Reviewing data on this existing beta item, the amount of feedback SMART received was around .004% of the amount of usage, leading me to believe this is not the best way to collect feedback from users.
.png)
From this survey, we learned that users who join beta programs want to provide feedback to the company, therefore SMART can feel comfortable asking users for feedback. Additionally, users who do not provide feedback forget that it is an option so SMART needs to be sure to prompt users for feedback.
Survey
I was unsure of how best to collect feedback within the beta program. The competitive analysis above showed a few different ways of collecting feedback, with none being the overt standard.
​
I decided to survey teachers in order to address a couple areas of concern:
-
What type of language should we use when talking about the beta program?
-
How and when should we collect feedback from users?
-
What will collect the most feedback from users?
-
What will inconvenience users the least?
-
In order to address the last bullet point (how and when), I needed to address the three different contexts in Lumio a user could be working in. There is the library context, where a user is in My Library, Shared Libraries, or Lumio Library browsing files. There is an editing context, where a user is in a file editing, creating, or adding content. And there is a delivery context, where a user is delivering or presenting a file to others (eg: students).
​
Testing all three scenarios via a survey was difficult as I didn’t want participants to work through the same questions three different times and potentially give biased answers. I worked with the research coordinator to design a survey where a participant chooses which scenario they want to work through, shuffling those options so there wasn’t a bias for the first vs the last, and then presenting the questions specifically for the scenario that participant chose.
Along the line of forgetting to provide feedback, users overwhelmingly indicated that if the only area that asked for feedback was a link in the avatar menu, they would forget. My recommendation was that the link in the avatar menu could be used as a backup way to collect feedback but should not be used as the primary way.
Users expressed that they thought that the feedback forms might interrupt their workflows and if it did, they would be less inclined to provide feedback. This is especially important to note during delivery, as teachers may be in front of students and even less likely to provide feedback than normal.
The final recommendation was a toast message with a multiple choice or rating question with additional considerations for delaying this message when a user is in delivery mode.
User Flows
Five user flows were identified; beta opt in, beta opt out, feedback during editing and library contexts, feedback during delivery context, and sharing with non-beta participants.
Lumio beta management like opt in and opt out made the most sense to be located in the avatar menu, where a user might find account settings and sign out options. This took into consideration industry standards for opting into beta programs.
A consideration for opting out of beta and sharing with non-beta participants was making sure that files did not break without the beta feature enabled. I determined two scenarios, the first being converting a beta feature to a static element in order to share with a non beta account, and the second resulting in an error if conversion is not possible.
Taking into account the research results, the user flow for the three contexts were similar, with the main difference being timing for the delivery context. This was a complex problem because we did not want to interrupt a user's workflow if they are in the delivery context in class in front of students. However, we also do not want to wait too long after a user interacts with the beta feature to request feedback.
Wireframes
I created these wireframes to communicate with others the ideas presented in the survey learnings and user flows. At this point in the design process, I knew I was passing off this feature to another UX designer while I was away for a month. I made sure to create these wireframes, document the user flows and survey findings, and present all of the information to the UX designer who would be taking over in my absence.
The Solution
While I was unable to take this feature all of the way to implementation as I passed this feature to another designer after wireframes were created, I’d love to show what ultimately was designed and implemented.
Ultimately, the Lumio beta feature is a framework that each beta feature will utilize and adapt to their specific feature. The examples shown are for an AI quiz assist feature.
Reflection
Unfortunately the Lumio beta feature still requires follow up work to implement in context feedback toasts. There was a surprising amount of technical work needed to implement those so the Lumio beta is currently relying on feedback being submitted in the support menu. As discussed in the research, this is not ideal and will probably result in less feedback from users.
The Team
Manat MacLeod
Program Manager
