Performing Appropriate Evaluations: Importance and Difficulties
Susumu Kuriyama
Interaction Designer
My work encompasses not only the design of applications and websites but also the performing of evaluations. In this column entry, I have written about the importance, yet difficulty of, performing appropriate evaluations. I will be content if this entry, detailing the experience I have acquired from performing my daily duties, provides assistance to those undertaking design and evaluation tasks.
Importance of Evaluation
These days the notion of UX design (User Experience Design) is commonplace, and product owners are choosing to perform evaluations of their creations to ensure a positive user experience. Regarding our company, we have received many requests for usability testing and in each evaluation we take care to confirm that appropriate methods are employed and a representative sample of product target users is tested. This form of assessment is very effective in implementing both a better product and a superior user experience.
To give an example concerning the aforementioned usability test; even if the test is carried out on only a small number of participants, numerous usability related problems can be revealed (reference: Why You Only Need to Test with Five Users). Therefore if testing is performed at an appropriate stage during the development (for example after the creation of a prototype), it is possible to release a product which has had many of its usability problems eradicated. As a result, not only is the released product more acceptable to users but it is much more efficient and economical to find and address any issues before the product's release. Therefore, evaluation undertaken during product development is both important and effective.
Difficulty in Performing an Appropriate Evaluation
As stated above, performing an evaluation is both important and effective in product development; however, performing an appropriate evaluation is difficult. I often hear accounts related to substandard evaluation planning and poor implementation, so much so that I feel that they can be divided into the three categories described below.
- Cases where usability issues have been already been reported
- Cases where there was a selection of inappropriate assessment methods
- Cases where it is not possible to distinguish what or which aspect of evaluation is best
Regarding the first case, the product issues have already been found and reported with feedback coming from service centers in the form of user complaints. In these cases, instead of re-evaluating the problem that has been previously clarified or making immediate improvements in the product, I recommend creating an enhanced prototype which addresses the reported issues and to then perform evaluations. In order to truly improve the product the performance of a simple evaluation is not good enough, instead resolute evidence must be obtained. Therefore to truly resolve the issues and enhance the product, an in-depth study should be performed where the improved prototype is evaluated.
The second case is where what is to be evaluated is known but the selected methods are inappropriate. For example, this may include situations where developers want to assess a product's usability, however rather than undertaking a usability test, the testers elect to perform a group interview, a research technique that is more suited to collecting diverse ideas and opinions. Under these circumstances, accurate evaluation results cannot be attained. Therefore, a key precaution to prevent the misinformed selection of inappropriate techniques is to consider at the planning stage which evaluation methods will be employed. If you are unsure, it is recommended that you consult an evaluation specialist. The nature of appropriate evaluation is both wide and complex, therefore when investigating usability by oneself you should be mindful that it is often difficult for specialists to work out a solution/answer.
The final, and third, point is the most burdensome. Even while recognizing the need for evaluation, there is confusion over what exactly is to be evaluated or from which perspective evaluation should be undertaken. For example, a case may be, when a chef is developing a new dish in which he pursues a goal of “deliciousness”, however, rather than the dish's taste, the dish's “ease-to-eat” is evaluated. This sort of issue is more likely to occur when a product concept and project objective is unclear. So, evaluation is to research whether objectives can be achieved.
If it is not clear what is to be evaluated, not only will an inappropriate evaluation be conducted but it will likely lead toward the development of poorer products. To prevent this, always be definite with regard to a project's goals and concept. It is very effective to map out and clarify the project's evaluation in its entirety - for example: deciding when to perform an evaluation and how the results will be utilized.
Conclusion
In this column you've read how to deal with what are often common problems in the important yet difficult sphere of performing appropriate evaluations. I hope that the above will be useful as a point of reference and that it will lead to an improvement in products and services. In addition, our firm can provide consistent support from the initial planning and design of websites and applications through to evaluation and we'd appreciate it that you consider our support as one of your options when undertaking projects of this nature.