A comparison of heuristic evaluation and usability testing : the efficacy of a domain-specific heuristic checklist

Abstract

The need for cost-effective usability evaluation has led to the development of alternative usability methods. Reducing the number of participants in a usability test or using heuristic evaluation are the two approaches most often associated with discount methods. In this dissertation, an alternative heuristic method was developed based on the concept of a double specialist (knowledge of usability techniques and of a software product domain). Double specialists were previously shown to provide heuristic evaluation superior to that of regular usability professionals, primarily because of their knowledge of usability problems typically found within the given domain. Similar knowledge, based on competitive usability testing, was used to devise a domain-specific heuristic checklist. The product domain chosen for the research was online calendars. The calendar checklist was compared to general heuristics and to streamlined usability testing with two usability test subjects. Fifteen software developers and fifteen usability professionals were each randomly assigned to one of the three evaluation methods. They were then provided with a previously unseen software calendar and were asked to find as many usability problems as possible. Participants using the calendar checklist found more total problems, and a significantly higher percentage of user-oriented problems and severe problems than did those using the general heuristic method, suggesting that domain-specific heuristics do provide an improved heuristic approach. Streamlined usability testing, however, was found to be the most effective discount usability method. Usability professionals and software developers did not differ in their ability to find user-oriented or severe problems. It was concluded that when time and resources permit, usability testing is the method of choice. When user testing is not feasible, however, domain-specific heuristics may provide results approaching the effectiveness of the double specialist.