Our partner is a large telecommunication company catering to young adults. After speaking with them, they identified several concerns. Were consumers learning enough about the company's features? They worked hard to build a business model that would benefit users. But was their branding taking away from the central purpose of learning about the company's offers? The company didn't want their design to be bland. They wanted fun, but not at the expense of the user not understanding what they offer. Based on this information, we moved to the next point...
Our team quickly went to the drawing board and established our research questions and hypotheses. We decided on testing a more simple and descriptive tone, akin to its competitors, as well as the current, fun tone. We wished to test whether or not the different versions affected understanding of the offers and general satisfaction of the website. Finally, we wished to test whether the information architecture was clear enough that users were quickly able to find the information they wished to seek.
Our test featured a between-subjects design with counterbalancing. There were 13 participants.
Task 1: The first task involved the user freely browsing the webpage on one of the company's services or features and stopping once they feel they have learned enough. Did they learn about the features, or were just focused on the branding's wow factor?
Task 2: We then tested their newfound knowledge by providing a quiz assessing the information on that particular page.
Task 3: Finally, the third task entailed a tree test, in which the users tested the webpage's information architecture.
We used Qualtrics for all our questionnaires, as well as the Quiz.
To test learnability, we used quiz scores (understanding of services). To test cognitive load, questions from the NASA-TLX were used and ease of understanding was assessed via WebQual questions.
To test satisfaction, we used the customer satisfaction (CSAT) and the Visual appeal subsection of the WebQual questionnaire.
Finally, to test effectiveness, we used the Intuitive operations subsection of the WebQual questionnaire, as well as the Tree testing questionnaire from Nielsen Norman Group.
We used Optimal Workshop for our tree test in order to test the company's site menu. It consisted of 8 realistic scenarios that all of us face when switching telecom companies. Users were asked to locate where they think they would find the information that they needed.
Interestingly, users learned about the company's features and rewards just as effectively in either version of the website. However, there were some difficulties expressed by the users and the biggest pain point was the language used by the company. The participants felt the terms used were complex and confused the rewards as a result. They also were not fully able to understand the rewards, instead remembering the branding more vividly. All in all, participants expressed a high satisfaction and user testing identified specific issues which were presented to the company team. The presentation led to a rich discussion with the team and a back-and-forth with several professionals who attended the presentation.
This project was a huge undertaking in the span of 4 weeks, but the amount that we learned in such a short time is priceless. Creating a user test from the bottom up was truly enriching. I definitely learned that having a thorough guide (the detailed protocol) eases much of the stress of forgetting to do or say something, as you know you have adequately prepared yourself. It also made it easy to standardize the experience for all of our participants. All in all, this project sparked a passion for user testing and I am eager to work on more projects!