Our analytics team provided stats on adoption rates and increases in mobile and tablet traffic.
We didnâ€™t see a huge initial increase in mobile traffic, and mobile traffic was still far lower than desktop traffic, which helped us prioritize our testing post-release. We gave the dev team time to make refinements and put off mobile and tablet testing 'til a future release. We knew if we tested again too soon, we'd run into the same feedback from past cycles. We also want to make sure that we see mobile-specific issues and just not general questions about the redesign changes.
In the first few weeks of the redesign, we also saw our feedback form on the mailchimp.com contact page jump from 25-30 emails per day to about 50 or so per day. Once we saw this number return to normal, we knew we had passed the new release crazy times.
Now we can start looking deeper into issues that go beyond bugs and initial confusion.
We always want to conduct interviews with users after a release, but in the case of a redesign, itâ€™s especially important to catch people as they are adopting the new designâ€”and there arenâ€™t enough of us to catch everyone adopting in the first week! So we chose to do â€œMini Diary Studiesâ€ that used a survey format. People switching over to the redesign could fill out the provided survey on their own time, as they went through their normal MailChimp workflow. Some even filled it out multiple times as they worked in their MailChimp account throughout the first week. In total we sent the invitation to the Diary Study to 393 people on our â€œSupertestersâ€ mailing list and got 20 full responses.
We also scheduled 7 remote interviews with users to watch them work as they experienced the new MailChimp design for the first time. Responses from these interviews ranged from total delight, to barely noticing that things changed, to disappointment. Results (videos, mindmaps, bug reports) were taken to the QA and dev teams immediately after this round of interviews was complete.
The most fun we had with testing the redesign was bringing in a long-time vocal user to our office to jump into the new design for the very first time. We invited everyone who directly worked on the redesign to witness someone using the system firsthand. Our team didnâ€™t have to worry about getting distracted (or bored!) reading a report or watching a video. And they didnâ€™t need to replay the video to really understand an action our user tookâ€”they could ask questions live and get answers immediately.
We love our fast-paced and remote testing opportunities, and we love getting to know all our users around the world. But inviting someone into the office to show us how they use MailChimp is really an incredible experience. Weâ€™ll be doing more live meetings with the whole team in the future.
Always Be Testing
A little over a month after our initial launch, weâ€™re still testing our new design. Our research allowed the development teams to move swiftly and attack the most important problems first. Upcoming tests will lean toward evaluating MailChimp on mobile and tablet devices, mixing moderated and unmoderated testing. Weâ€™ll also conduct sessions with people who have never used MailChimp before to see how their experience compares with new user tests conducted before the redesign.
If you want even more info about our process, Jenn will be giving an in-depth talk on Redesigning with Research at Web Design Day in Pittsburgh
October 25, 2013.