Inland Revenue used the following methods:
Hold a workshop with key stakeholders and use a context checklist, which covers the main aspects of the system. This technique was already well established in the testing area but we did it earlier in the development before design started and it was produced as a joint exercise by requirements and testing.
The user’s skills, tasks and the working environment were defined. The value of documenting this corporate knowledge should not be underestimated. Our IT supplier does not have staff with an intimate knowledge of our core processes or organisational culture. Before TRUMP there was a feeling that we had this knowledge "in our bones" and could pass this on to the IT supplier when requested. Context analysis proved there was a better way to spread that knowledge around and the document has been used time and again by all involved to act as a reminder of what and whom we were trying to design for. It was also the key input to setting the usability requirement for and for both the diagnostic and performance measurement usability evaluations.
Instructions for context analysis.
A usability analyst and seven users evaluated the existing system out in the local office network. Each user was given a short introduction and then observed using the system to do the same key tasks. Comments were captured by a usability analyst which generated a problem list and a report was produced which was fed into the development team before design of the new system began.
Reviewing the output from this work it is clear that the users should have also been asked to fill out a Software Usability Measurement Inventory (SUMI) questionnaires and we should have used the opportunity to gain efficiency and effectiveness figures for later use but apart from that the time invested was well worth it.
Instructions for evaluation of an existing system.
We had never set a detailed usability requirement early in the lifecycle before (though we had set less detailed ones later in the lifecycle) and needed a trained facilitator to guide us through the process. It was also difficult in a commercial development environment used to setting hard and fast business requirements based on legislation and/or clear operational needs to adapt to a less exact science. We needed to change that mindset and acquire an understanding that the usability requirement had to have "blurry edges" because the new system would not have the same tasks as the old one, only similarities. Our estimates, however well meaning and well researched, might (and indeed did) therefore be proved to have missed the mark somewhat when the new system was tested. The use of a trained facilitator was invaluable in helping us get to grips with all of that as well as reference to resources on the web and publications such as Usability Engineering by Nielsen.
The initial workshop proved troublesome because of the different user types and tasks involved but even when these were pared down the various estimates for effectiveness and efficiency had to be agreed and then verified out in local offices on the existing system. Once the data had been collected it then had to be processed and a document produced for the project to agree. An amount of rework was necessary before all parties were content with the measures set and then it could be published. Finally we made the mistake of not growing and refining the requirement sufficiently as our understanding of the system matured which meant that when we came to do the final performance measurement test adjustments had to be made to ensure it reflected that latest views of the business.
In hindsight it is clear however that the advantages do outweigh the time spent. All parts of the project team had a clear, common understanding of what is an acceptable standard for the usability of the system and we were able to evaluate if that benchmark was being met so helping improve and control the quality of the system. The skills necessary to set a requirement were easily transferred from the facilitator to the business and are already being applied on other projects.
Instructions for usability requirements.
We adopted a two phased approach to ensure workshops were effective and efficient:
Phase 1 - prior to workshops
Phase 2 - during the workshops
We all took this technique to our hearts. It was relatively simple to pick up as it involved the business people (the end users) documenting what they did on a daily basis back in the office. This knowledge could then be captured before every function design workshop and not only used to focus what the IT was being developed for but used in conjunction with other techniques such as task analysis and paper prototyping to verify the emerging design was meeting the needs of users and then used again to validate the final IT prototype was correct.
One of the undoubted successes of the whole project. After an initial workshop to the end users were able to produce task scenarios on demand and both formally document them in the appropriate preparation pack and produce them during the JAD sessions when presented with a proposed change to the design.
Preparation Pack
An aid for participants so they have a clear idea of what the workshop is expected to achieve and how they can contribute effectively. The packs were put together by the requirements analysts responsible for the function to be designed. The packs always included:
In our view an absolutely invaluable briefing aid for all participants in the workshops that became integral to the success of each session. The cost of 4 hours preparation was repaid by the fact that each workshop usually consisted of 8 people and once packs had been introduced business could usually be wrapped up in a single day rather than spilling over into the following which had been the norm previously. This proved to both IR and EDS that prior preparation prevents poor performance.
Paper PrototypingPaper or a whiteboard or the COOL:Gen CASE tool were used during the workshops to produce draft windows based on the business requirement and the straw-man contained in the preparation pack. These prototypes were then tested using the task scenarios, fresh designs produced and then tested again until agreement was reached and the initial straw-man design turned into a tin-man one that could be coded.
This technique was already widely used on other projects but was formalised for the trial project and (this is probably the most important part) linked to the preparation activities before the workshop and the use of task scenarios during it. As a technique it was easily picked up by the analysts and end users but really proved its worth when linked to the task scenarios. The two go hand in hand.
Style GuidesOur usual practice had been to leave Graphical User Interface standards to individual projects, which meant applications were delivered to the business with a different look and feel. An attempt had been previously to provide a corporate style guide but this had rapidly become out of date, was unwieldy to use and had no sense of ownership amongst the IT developers.
A corporate style guide and an overview of the chosen user interface style were provided to the development team and a number of major national projects that were starting at the same time and followed religiously throughout design. The benefits of this approach were immediately apparent:
We also learnt that style guides need to be at the right level to ensure they can be easily read and understood by both a technical and business audience and should not overly prescriptive to take the imagination and innovation out of design. To address the concern about too much control a waiver system was designed for when standards effected the efficiency of the compliance business.
Developers involved in previous projects commented on how much less pointless discussion spent on names and placement of controls. The one failure in this area was the third leg of the style guide set, a list of usability guidelines, designed to give business developers a few rules of thumb to complement the interface style guides. Though a set of guidelines have now been agreed by all parties it took a number of versions to get all three sets to fit together, complement rather than contradict each other and add value rather than say the same thing in a different way. This delay meant that the usability guidelines were not used during design though they will be able to be used during testing and will be used on other projects as the style guide set has already been accepted a the way forward for IR/EDS.
Instructions for task scenarios, paper prototyping and style guides.
We have been using performance measurement for a number of years but this time introduced scoring against a detailed usability requirement for all the main business tasks. Previously we had compared testers against a perfect user to see how far up the learning curve they were or constructed a cheap and cheerful requirement based around one or two tasks as part of the preparation activities. The use of a detailed requirement that formed part of the wider business requirement and that had the buy in of the whole project meant the results of the exercise carried much more credibility and empowered the usability analysts in their discussions about resolution of the problems that has been discovered.
Please be under no misunderstandings about how complex these exercises are. You need a usability analyst trained in the method and it typically takes 30 days of work over a two month period to prepare, execute, analyse and report. When you add to that the technical costs plus securing the right users the cash register is kept ringing. That said the results justify the trouble and expense. You get an end of lifecycle activity that objectively measures the system for its quality in use just as the many thousands of users will do when they switch it on from day one. You get advance notification as to the impact of the system on the efficiency and effectiveness of key business processes. And that type of data is priceless to any business not only because it gives you the opportunity to manage issues and expectations but because it's a much clearer signal to acceptance than whether the code meets a business requirement set in the dim and distant past and usually centred around functionality rather than task.
Instructions for performance measurement.
Live running
We have used the Software Usability Measurement Inventory (SUMI) tool for a number of years in development. It is an accepted industry standard way of assessing Users’ perception of the usability of software and provides six scaled scores; a Global usability figure and five sub-scales of Efficiency, Affect, Helpfulness, Control and Learnability. The Global figure is probably the most useful as it is a highly reliable and valid metric which gives a very accurate general view of the user perception of usability. It can be used for "before and after" studies in the development process or in making direct comparisons between products. The global figure can be viewed as a "temperature gauge" of usability as perceived by the users and is valuable in development to feed into your acceptance process for the system.
What we learnt from administering SUMI in live running is that it's not a good idea to wave goodbye to a system once the code has gone live. That policy not only encourages a culture of delivery to time and cost rather than delivery of a useful, usable product but means you don't gain information that helps you improve the product (and others still in development) but also can verify if your testing is rigorous enough to replicate live conditions and give you results that you can confidently rely on.
The other thing to be aware of with SUMI is the natural tendency of managers to believe you get can valid results by just sending out the questionnaire and not bothering with the bothersome aspects of completing context analysis or administering it after staff have done work using the software you want to measure the level of satisfaction with. As with most things if you cut corners you'll have an accident. SUMI needs to follow the correct steps and you can then have confidence in the results, if you don't want to follow the correct steps, forget it. You'll be better off using focus groups, straw polls or not talking to your customer.