Usability requirements: how to specify, test and report usability

 

 Home

 

 Case study (UK):
  Prezzybox/in2netlogic: Ecommerce Web Site

 

 Summary

 Suppliers

 Purchasers

 Requirements

 CIF

 Case studies

   • Italy

   • Sweden

   • UK

   • Greece

 Conclusions

 Download

 

 

 

 

 

 

 

Page contents

Overview
Background - The evaluation of a shopping website
Purpose of the trial
Trial plan, tasks and user selection
Summary of CIF results
Experiences from using the CIF
Feedback on the CIF from the consumer and supplier organisations
Discussion
Conclusions – what other shopping sites can learn

Overview

This study was concerned with an online shopping website called Prezzybox. The site, developed by in2netlogic, offers a wide range of gift ideas for purchasers to select and have delivered to a friend or relative or to themselves. The website was evaluated by the PRUE partner - the Research School of Ergonomics and Human Factors (RSEHF) at Loughborough University (which incorporates the former HUSAT).

The main objective of the user test was to obtain user performance and satisfaction metrics for Prezzybox along with user comments to highlight any problems with the site. Testing was carried out with 12 users who were all experienced with the Internet and interested in buying from the Prezzybox site. Users were asked to make a real purchase from the site - this being their payment for taking part. Thus the evaluation was designed to test the success of users in completing an online purchase - a crucial success factor for any online shopping site.

The result of the evaluation was a CIF report, which documented user performance and satisfaction with the site. This showed that while 10 out of the 12 users were able to make a purchase, 2 failed to do so. If the consumer organisation could also capture those two users, their sales would increase by 20%!

These performance results were also supported with user satisfaction ratings. The levels of satisfaction recorded were 'satisfactory' showing that there is room to improve the 'user experience' when using the site.

Evaluator comments and recommendations for change to the site were also included, highlighting features that could be changed to help improve user success in making a purchase.

The benefits of the CIF to the consumer organisation (Prezzybox) are:

  • To find out how successful consumers will be in making a purchase from their site i.e. what percentage can actually make a purchase?
  • To provide a benchmark for user performance and attitude, which can be used for comparison with the shopping site when it is revised.
  • To obtain insights into any problems that users face when using the site (to complement the summative results) and to receive suggestions for improving the site.

The benefits of the CIF to the supplier organisation (in2netlogic) are:

  • To obtain objective feedback on the success of the design they produced.
  • To identify the most important usability issues that will enable the shopping site to support more successful purchases - and therefore improve the profitability of the site for the consumer organisation.
  • A new contract to improve the site, based on the test results and the comments and suggestions for improving the site.

Currently the Prezzybox site is being refined. If possible a second round of usability evaluation will be performed so that the consumer and supplier organisations can receive concrete evidence that the site has been improved. It is also hoped to show that this usability activity has enhanced the site from the user's point of view.

In summary, the evaluation approach of the CIF is recommended for other online shopping providers in order to test what proportion of users actually make a purchase from their site. It also sets a baseline against which new versions and upgrades to the site can be compared, while also highlighting the problems that need to be fixed if the site is to achieve a greater number of online sales.

Background - The evaluation of a shopping website

The Web is now populated with many online shopping sites but evidence seems to show that many users fail to make a purchase from them. In the United states, a study carried out in 1999 by Jared Spool who tested a series of shopping websites showed that for even the most successful site only 42% of users were able to make a purchase. For the least successful site, only 28% were able to successfully buy online. Recent data from another leading human factors researcher, Jakob Nielsen, has shown a success rate for online shopping of between 50% and 60%. These figures, which are at best moderate, clearly will have a major effect on reducing profit margins in the e-commerce sector and will also restrict the wider use of online shopping in the future.

But how can a design team know whether their online shopping site will be successful in the marketplace? An expert evaluation by Human Factors personnel, or an informal study by users may highlight problems, but how can the online shopping service provider know objectively measure how successful their site will be? The only way is to ask a sample of users to try and purchase an item from the site 'for real' and for an evaluator to observe whether they are successful. This was the approach adopted by the PRUE partner, RSEHF - Research School in Ergonomics and Human Factors (formerly HUSAT) at Loughborough University. The Institute were asked by the in2netlogic web site development company help them evaluate an online present buying website that they had produced for Prezzybox, a midlands-based mail-order company supplying novel and high quality gifts to the public via an online shopping site.

The CIF (Common Industry Format) offers a useful framework for organising and reporting on such a study. The EC supported PRUE project was set up to test the CIF in a range of different IT development situations. Thus the evaluation of the Prezzybox website was selected as RSEHF's case study within the PRUE project.

Purpose of the trial

Although the Prezzybox website had been running for some time, no formal testing of it with users had been carried out. Thus the main business objective of the user trial was to find out how easily users could find suitable items on the site to buy and whether they could successfully make a purchase.

The benefits of the CIF to the consumer organisation (Prezzybox) are:

  • To find out how successful users will be in making a purchase from their site i.e. what percentage can actually make a purchase?
  • To provide a benchmark for user performance and attitude, which can be used for comparison with the shopping site when it is revised.
  • To obtain insights into the problems that users face when using the site (to complement the summative results) and to receive suggestions for improving the site.

The benefits of the CIF to the supplier organisation (in2netlogic) are:

  • To obtain objective feedback on the success of the design they produced.
  • To identify the most important usability issues that will enable the shopping site to support more successful purchases - and therefore improve the profitability of the site for the consumer organisation.
  • A new contract to improve the site, based on the test results and the comments and suggestions for improving the site.

Trial plan, tasks and user selection

A PRUE trial plan was designed to test the success of users in completing an online purchase - a crucial success factor for any online shopping site. Users were asked to make a real purchase from the site - this being their payment for taking part.

The measurements to be carried out within the trial were essentially to record user performance in finding and buying a gift for themselves or for a friend and also satisfaction metrics with the whole process of using Prezzybox. User comments and evaluator observations were also recorded to highlight any problems with the site.

Testing was carried out with 12 users who were all experienced with the Internet and interested in buying from the Prezzybox site. The user trials were carried out in RSEHF's offices via a laptop with modem telephone connection to the Internet to simulate shopping from home. Users were asked to make a real purchase from the site - this being their payment for taking part. Thus the test was designed to test the success of users in completing an online purchase - a crucial success factor for any online shopping site.

Summary of CIF results

The result of the evaluation was a CIF report, which documented user performance and satisfaction with the site.

User performance

The main performance metric was whether users could make a purchase. While 10 out of the 12 users were successful in making a purchase, 2 (16%) were not, which indicates a potential loss of sales. The reasons for the failure to buy were that one user was concerned about revealing personal information when completing details on the site, while the other was not successful in finding anything they really wished to buy. Arguably these findings would not have transpired with a traditional formative evaluation. Thus there is scope to improve the site, e.g. by explaining to the end users that any personal details entered (name, address, email, preferences, etc.) will not be passed on to a third party. Similarly, an improved indication of the range of search facilities available could help users find something they wish to purchase more effectively.

The other metrics were:

  • Mean total task time (45 minutes - which reflected strong user interest in the site
  • Completion rate efficiency for correctly completed tasks (2.7%/min)
  • 'Mean number of total errors' per user (0.9).

There was also one minor assist each for two users. One user had looked through many items to purchase but wanted to know if there was any kind of search facility available. They were informed that they could look at the Gift Wizard. They investigated this facility but eventually selected an item identified previously. Thus the assist may have affected the task time but, arguably, not the final success in making a purchase. Another user was making a purchase and wanted confirmation that they did not need to register first. They did not expect to but wanted confirmation. Again the assist may have speeded up the process slightly. Without the confirmation the user would have spent a little longer looking at the screen instructions which informed them that registration was not necessary.

The average time of 45 minutes is not a major issue and reflects more people's interest in finding an item on the Prezzybox site rather than difficulty in overcoming problems. (Within the total task times for each user, there was very little unproductive time spent in overcoming problems in using the system.) However any improvement in the navigation facilities, screen layout and download times may show an improvement in the performance scores as people are able to view items on the site more efficiently.

User acceptance

A Software Acceptability Questionnaire (SAQ) developed at Loughborough was applied which produced mean user ratings for a range of factors on a scale of 1 to 7. The scores were all reasonably high, i.e. Usefulness: 4.2, Clarity: 4.5, Efficiency: 4.6, Supportiveness: 4.5 and Satisfaction: 3.8, while leaving scope for improvement.

The scores, which are mostly around the mid-point of the scale (4.0), show that people feel that the website is satisfactory, which is borne out by the fact that in general users enjoyed using the site. The usefulness score (4.2) reflects some peoples' comments that the site did not have all the kinds of items that they would have expected from a present or gift site. Some users wanted to be surer of the quality while others wanted the site to be more focussed. The score for clarity (4.5) shows that users had no significant problems in using the site, but could not scan the layout of the goods on screen as easily as they would have liked. Similarly, under efficiency (4.6), users could move around easily but most commented that the site could be quicker to display pages. This could have been a bigger issue if users themselves were paying the telephone bill. Again the score for support and help (4.5) is fairly high, reflecting the reassuring security statement and well guided purchase process. The satisfaction score is slightly below the midpoint (3.8).

User comments

The performance and acceptability scores were supplemented with user and evaluator comments, and recommendations for change, to the site. Users commented that Prezzybox was very interesting and enjoyable to use but they experienced problems with some aspects of the site: namely screen layout, item ordering, and needing a simpler means of specifying price range. Some users also failed to notice the 'Gift Wizard', a special feature of the site, provided to help the users select items to meet the characteristics of a particular person. Providing a clearer button for it could enhance user interaction with the site.

Future plans

Following dissemination of the report, the consumer and supplier organisations have met to discuss improvements to Prezzybox which will be implemented over time. If possible, a follow-up test will be carried out to show, objectively, if the site has improved from the user's point of view. Hopefully this approach will set the pattern of for testing e-commerce sites in the future to help them meet user needs and be easy to operate.

Experiences from using the CIF

Using the CIF has been an interesting experience. It has provided a useful framework for not only planning a user trial but on specifying the objective of the trial itself and developing a suitable plan to achieve it.

The CIF recommends some basic metrics, which must be taken but also provides sufficient flexibility to apply other metrics more suitable to the application itself. The shopping study highlighted the interesting issue about whether not using a function could be classed as a type of 'error'. For example, within the Prezzybox website, the Gift Wizard is a special feature designed to help users find gifts to match the interests and personality of the person receiving the present. However this feature was not very prominent and most users did not realise that it was present. By missing out on this feature, they may not have used the site as efficiently as possible, so may be less inclined to return to it in future.

The CIF does allow the report writer to add comments and suggestions for improving the system being tested in an Appendix, which clients are normally very interested in, even for summative testing, so perhaps there should be a specific section in the report itself to record this type of information.

Feedback on the CIF from the consumer and supplier organisations

A feedback form was distributed to both in2netlogic and Prezzybox. It is interesting to compare the ratings that each of them gave to the report as a whole and to each individual section.

Report section

Consumer rating of usefulness

(1= very useful, 7=not useful)

Supplier rating of usefulness

(1= very useful, 7=not useful)

Whole report

3

4

Executive summary

3

5

Introduction

3

4

Method

3

3

Performance results (task completion, task time, errors, assist, etc.)

5

4

Subjective ratings

2

4

User comments, evaluator observations suggestions for change.

2

2

Appendix – Evaluation materials

5

7

Interestingly the consumer organisation seemed to place slightly more value on the introductory parts of the report and the subjective ratings than the supplier. The supplier and the consumer were moderately interested in the method used to perform the study and the user comments, evaluator observations and suggestions for change.

In terms of the main benefits in receiving the CIF, and taking part on the project, both organisations felt that most useful aspect was an impartial and objective analysis of current operation, and generation of new ideas. Neither group identified any problems with the CIF, although the consumer organisation had hoped for more comments on all features of the Prezzybox site. Although some feedback on most features was provided, these depended on which features were used during the trials. The supplier stated that weighted comments would have be very useful - i.e. how important was each suggestion or observation.

Both organisations stated that the CIF would definitely lead to changes to the system. Some minor changes were made immediately, but it was also stated that all development is done through a tight specification routine to maximise quality and this takes time. However the improvements are expected to be carried out in the first quarter of 2002.

The consumer organisation felt that the CIF improved the knowledge of usability although this was less true of the supplier organisation. As the supplier and consumer organisations already work closely together, it was not expected that the CIF would have a great impact on supplier/consumer communication.

Discussion

Although the trials elicited many user comments about how to improve Prezzybox, everyone who took part seemed to thoroughly enjoyed the experience of browsing the site and choosing a gift to purchase. Nearly all stated that they would visit the site again.

The generation of the PRUE evaluation report provided a useful focus for discussion between RSEHF, the consumer and supplier organisations and laid the path for producing an improved shopping site. The idea of performing a real trial, where users actually purchased an item (as payment for taking part) enabled the group to see how well the site performed in a real situation. This was illustrated by the performance measures although these did not reflect non-use of key features such as the Gift Wizard.

The report provided a good format for reporting all features and will serve as a good record for performing the repeat trials. Although suggestions for change can be included in the report in a separate section, perhaps more thought needs to be given as to how summative results can be related more directly to design recommendations. In a developing design situation, this approach could show how design changes lead to improved user performance and satisfaction ratings.

Conclusions - what other shopping sites can learn

The summative results reported in the CIF show the most important and stringent test of an online (web) shopping site i.e. can users make a purchase from it? Without such a test, an online shopping service cannot be sure how effective their site is, and whether they may be losing many sales as a result of poor design.

Similarly, without producing a baseline for use of the site, the provider will have no basis for comparison when they produce a revised site. The CIF reports the trials in a clear and comprehensive way, allowing trials to be repeated in a faithful way and thus to produce valid results for comparison.

There is scope for the CIF to be improved i.e. by integrating formative data more closely with the summative results, so that the consumer and supplier organisations can determine which changes should be of greatest priority in order to enhance user performance i.e. to achieve a great number of e-shopping sales.

Last updated 12-Mar-02

 

Copyright © 2002 Serco Ltd. Reproduction permitted provided the source is acknowledged.