Subjective assessment tells the evaluator how the users feel about the software being tested. This is distinct from how efficiently or effectively they perform with the software. The usual method of assessment is to used a standardised opinion questionnaire to avoid criticisms of subjectivity.
This method gives the evaluator information about how the users feel about using the software being evaluated. This should be distinguished from:
It is customary to use a close-ended questionnaire if one is available, in order to produce quantitative data, otherwise the results of the activity can be vague and open to interpretation. At worst, a critical incidents type technique may be used.
Identify the questionnaire you will use. This is not the time to start developing your own questionnaire. See the FAQ about questionnaires (see below) for more information on this topic.
There are two scenarios in which subjective assessment may be used:
A variant of (2) is that a supervisor or manager distributes questionnaires to their work-group, collects them when completed, and returns them to the evaluator. In either case, make a list of users who will fill out the questionnaire.
You will most probably also need a "screening questionnaire" to get some background data on each respondent (e.g. computer experience, job level, frequency of use of the software being evaluated). This may come out of the Context of Use study. An example of a general-purpose screening questionnaire may be found on the SUMI homepage.
It is usually not enough to discover satisfaction levels. You should also find out what features of the software give rise to unprecedentedly high or low levels of satisfaction. To do this you should ideally have access to a smaller sub-sample of your original larger sample for a second stage of testing, involving an interview. Some questionnaires will give you quite good diagnostic data and so this second stage is not as important if you use one of them.
Running
Ensure that all your targeted population has been given a questionnaire to fill out. If you are doing a mail-shot, expect approximately 20% response rate. This may be higher if you are doing it via the web. You can increase your response rate to nearly 100% by doing reminder personal calls. If you are using the questionnaire as part of a lab test, ensure that the questionnaires as filled out can be associated with all the other data (time scores, tapes, sample outputs etc.) from the respondents - so you don’t have questionnaires which you cannot match up with the rest of the data.
It is usual not to prompt the respondent how to reply. If a respondent complains that a question is inapplicable or wrong, tell them words to the effect that it is up to them to make their own judgement about each question, and that there are no right or wrong answers. Encourage them not to miss out questions.
Reporting
Subjective questionnaires usually have a scoring scheme since they are mostly based on Likert-style scaling techniques (see the FAQ on questionnaires for more information about this.) In fact, for most questionnaires, you simply sum the obtained scores for each respondent. Some questionnaires have different sub-scales which must be summed independently. The SUMI and WAMMI questionnaires have their own scoring software which put the scored data into a standard report.
In the readable body of the report, include
Include the full scored data in an appendix.
The SUMI user handbook gives detailed recommendations and methods for carrying out a user satisfaction survey which is applicable to many survey questionnaires, not just SUMI.
The following is a list of questionnaires available:
SUMI (see sumi.ucc.ie)
WAMMI (see www.wammi.com)
SUS (see www.usability.serco.com/trump/documents/Suschapt.doc)
QUIS (see www.cs.umd.edu/hcil/quis/)
USE (see http://www.stcsig.org/usability/newsletter/0110_measuring_with_use.html)
CSUQ (see www.acm.org/~perlman/question.cgi?form=CSUQ)
IsoNorm (see www.sozialnetz-hessen.de/ergo-online/Software/Isonorm-Workshop.htm - in German only)
IsoMetrics
As alternative to questionnaires, interviews or focus groups may be used, but the data resulting from such methods is generally less precise than if a standardised questionnaire is used.
If the software is still under testing, then it may be possible to do something about features which give rise to negative satisfaction ratings. Otherwise, the usability satisfaction report must be an input to planning the next release.
http://www.ucc.ie/hfrg/resources/index.html#cases
Effective Methodology for the Study of Human-Computer Interaction (with M Corbett). North Holland/Elsevier, 1990.
©UsabilityNet 2006. Reproduction permitted provided the source is acknowledged.