A recent discussion in the client project touched upon an often debated topic – What is the value of user testing?
On the surface, the value of user testing is obvious. It invites representatives of people who will use a design to provide unbiased feedback to a proposed solution. It reduces the possibility of unrealistic expectations on users (e.g. User would put this around his neck every morning) and weeds out usability problems that designer may overlook. On the flip side, it’s often argued that companies over-rely on usability testing and defer design decisions to a limited number of randomly chosen people. Lab settings and paper prototypes do not provide enough context, or even the right context, for the user to realistically judge the experience of use. A lot of designers operate under the mentality that ‘some testing is better than no testing’, while others believe in this age of constant drive for disruptive innovation and the common practice of having a digital service in a prolonged beta phase, user testing is a waste of time. Users simply don’t know what they want until you show the design to them in its full glory as a working solution. All these arguments lead me to believe that user research, which user testing is part of, is ill-presentated in companies. Companies trapped in such kind of debate miss the value of usability testing because they mis-understand the broader role of user research in the design. This needs to be fixed.
The typical design process is divided into 4 phases – Discover, Define, Design and Deliver. These step repeat multiple times during a project, sometimes in this logical sequence, sometimes not. In a user-centered design process, understanding of the user’s needs is included in every step. User testing is most appropriate during the Design phases.
During the Design phase, user testing provides grounded evidence to improve details of the design. However, there is merit to the claim that users don’t know what they want but they know very well how to judge what is in front of them. For that reasons, designer should provide test materials that are well thought out, so the users can judge the quality of each design in their details. Sometimes it is OK to test an incomplete design, but request users to provide over-arching feedback on an incomplete design is not (e.g. Will you find this solution pleasurable to use on a daily basis?). Incomplete design in, incomplete user input out. With that said, it is important to recognize that results that stem disruptive innovations rarely come put of user testing. That is because by the time the process reaches the Design phase, designers and stakeholders have already defined the problem space of the project. Expecting users to react to a prototype and articulate an unmet need that will lead to a brand new service is just wishful thinking. In the rare cases where profound user input is captured, it will most likely documented as a footnote in the report, under ‘Further Considerations’. Most user-supported evidence for disruptive innovation result from design research during the Discover phase, when the problem space is still fuzzy, or after the Delivery phase when unexpected patterns are found in user’s operations with the solution through analytics, support calls and review comments.
Contrary to common believe, designers by themselves don’t hold the solution to most complex design problems that we are tasked to solve today. Understanding the design problem with the help of user’s input offer a much higher rate of success. User testing is not the only tool, nor it is always the right tool, to incorporate user input. To realize the value of user research, designers, especially design leaders, must education companies to use the right user research tool at the right time.
I just found this 2005 article from Bruce Nussbaum, about how alumnus from my school are leading design thinking and innovation strategy in different fields.
Sony should send its people to the Institute of Design (Business Week)