What is usability, and how can you test for it?

This is the writeup for the presentation: What is usability, and how can you test for it? given at Southampton University March 2009. These views are entirely my own and do not represent those of the IBM corporation.

Products are developed to be used by users.  However, if products are not designed with usability in mind, then users may not be able to get the best out of that product, or even want to buy it in the first place.  This article discusses what usability is, provides tips on designing for usability, ways that you can test for usability, and how you can get feedback from users.  This article is mostly focused on usability for software and web applications, but some of the techniques could also apply to other products and technologies.

Many people have different ideas about what usability is, and what it applies to. Before defining what usability is, it is useful to talk about what usability isn’t.

A common misconception about usability is that is applies only to simple applications, gadgets and websites. In fact usability applies to almost everything! It is actually hard when you start thinking about it to find an everyday object that is a very good example of usability.  An example of an everyday object which appears to have very good usability is a light switch.  A light switch has a simple button that can either be on or off.  But when you start to think about it a bit more, it can be hard to know which way is on or off.  And if you have multiple light switches, you have to turn them all on or off to know which switch belongs to each light!

An example of good and bad usability in an everyday object is the usability of milk cartoons. When milk cartoons were first developed they had an old fashioned fold-back opening. You had to push the cardboard back in a specific way, and these were hard to open, and when you did open them the milk usually spilt when you poured it. These were so difficult to use I remember adverts showing you how to open them, and the manufacturers appearing on news shows to demonstrate how to do it! Modern milk cartoons have mostly abandoned this design, and instead there is a screw-top on the top of the cartoon. This is much more obvious how to use, and you don’t get the problem of spilt milk. They are also better for maintaining the freshness of the milk.  These may not be perfect usability – someone with arthritis might have trouble using them for example, but the improvement in terms of usability for most users is clear.

When you apply usability to software and technology, many people believe that usability is aimed at ‘basic’ users and everyday technology. The assumption is that you only need to think about usability if you are designing a website or a gadget that is aimed at ‘home’ technology. Some people’s assumption is that complex business software does not need to be usable. After all, businesses employ technical specialists, can send their people on courses, or will read the manuals before they try and us it. These assumptions are wrong for a number of reasons that will be discussed later.

Another assumption about usability and technology is that it only applies to graphical user interfaces. This is not in fact true; usability can be applied to any interface. It doesn’t matter how complex the technology is, the usability can be improved, and in fact an effort should be made to improve it.

Another common belief is that ‘slick design’ equates to good usability. In some cases good visual design does enhance usability. Products developed by the company Apple are well known for their good design and are also regarded as being more usable. However, slick design and pretty graphics alone do not provide usability.

You can find examples of this principle quite easily on the web. Many web design companies have very cool and very slick looking websites as you may expect. A common problem with these design heavy websites is their loading time. One web design company I cam across had a very slick design based upon the Apple iPhone. This in fact was a website that had won an award for the graphic design. And it was an impressive website if you just looked at the graphics, but there were a number of problems that severely impacted the usability of the website, and meant that many users would not get the information or value out of the website that they wanted.

The first problem with the website was the loading time. When you enter the website there is a loading screen. Graphical very nice, but it takes over 50 seconds to move from this page to the actual content of the site. Research has shown that users get bored if content on the website does not load in 4 seconds! Yes it is pretty, but why would I wait for nearly a minute to see the rest?

When you do eventually get to the main content, again there are more pretty graphics with a realistic looking office with a girl sat at a reception desk. The room in the background is interactive, so you can move the mouse around and there are things to click on, and every so often the girl stretches. The problem is that there is no indication of what will happen when you click on the elements, and in fact the behaviour on the webpage is not what the user would expect! Also there are green circles on the screen, but they don’t seem to do anything. Why are they there? There are some icons on the bottom of the page that look to be navigation links, but they do not behave in the way a user would expect them. For example, clicking on these does not move you to a new page, or result in a change to the background. This is a website for a design company, and yes it is a very nice design, but the usability is not so slick. As a user I don’t know what to do with this site, and it doesn’t behave in the way I would expect. And as an average user I would never have waited long enough to try and find out more about the company!

Usability is also not just about market research, and asking people’s opinions about whether they like something or not. Usability is not just about assessing whether it works for people once it has been developed. Usability needs to be considered all the way through the development process. There is a misconception that including usability in a product is expensive and time consuming, but as discussed later in this article, having usability as part of the initial design and planning stages for a product is much more effective and in fact cheaper than finding out there is a problem once you have provided your product to your customers. With the web design companies website as an example, to change that website now to make it usable will be both difficult and costly, whereas if they had considered how people were going to use the website at the design stage it the usability could have been built in for little extra cost.

So what is usability? The following definitions are useful when describing usability:

  • Usability is the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use
  • Effectiveness is the accuracy and completeness with which users achieve specified goals
  • Efficiency is the resources expended in relation to the accuracy and completeness with which users achieve goals
  • Satisfaction is the freedom from discomfort, and positive attitudes to the use of the product
  • Context of use is the users, tasks, equipment (hardware, software and materials), and the physical and social environments in which a product is used

Usability is all about a user’s ability to use a product to achieve their specific goals. The user must be able to achieve that in an effective and efficient way. The most important measure of usability is user satisfaction. Could the user do what they wanted? Did it take a reasonable amount of time? Was it difficult? Did the user enjoy using the product?

The context of use is also important. The user must be able to use the product in the conditions and situations that they need to use it in. Does the user have the skills to perform the task or would they have needed special training? Could they use it in the environment that they needed to use it in? Will the product work on the equipment that they have available, for example if the product is aimed to be used by school-teachers, will it work on laptops, or do you have to run it on a mainframe? As another example, if the users are using a device to record measurements made in a lab, could they use it wearing gloves? What if the users want to use a device on an archaeological dig, can it be used in muddy and rainy conditions? The product has to work for the target users and in the environments that those users want to work.

So the big question usability asks for a product is can the user easily use the product for the goals they want to achieve without being shown how? Is it intuitive for the users you are aiming the product at?

Usability encompasses more than just the product, it is everything involved with the product. Every product has a lifecycle for the user, and the usability of each stage contributes to the usability of the product: from finding the product, ordering it, receiving it, setting it up, everyday use, upgrading it and replacing it. This is known as ‘The total user experience’. Often people just think about the usability of the product itself, but a failure of usability in any of these areas can prevent people buying the product in the first place, getting value from the product when they have it, and choosing not to replace it in the future.

The primary importance of usability is increased customer satisfaction. A usable product is likely to help the user to get value quickly out of a product through increased user efficiency and increased productivity. For software in particular this leads to a faster return on investment, reduced training and support costs. Customers like spending less money; this is obviously good for their business. Good usability also reduces the number of mistakes people make and can therefore improve the safety and security of the software, and the data it is used with. If users enjoy using the tool, productivity is better, and promotes a more positive work environment. Increased user satisfaction leads to happy customers! This has obvious benefits for the company selling the product.

Happy customers lead to higher revenues; you will sell more software especially if your usability is better than your competitors. If customers have high satisfaction, customer loyalty is increased, meaning that the customers will want to purchase from your company next time. These customers will also recommend your product to others.

Thinking about usability when you develop a product is very important in reducing development costs. To change a product at design time is very inexpensive. By the time development of a product is in progress it becomes more expensive. If a product has been released to customers and a usability problem is discovered, this is extremely expensive to fix. As an example, think about how much more it costs to change the colour of a car once it has been given to the customer, than if it was done at the factory in the first place. Again, think about the website design, if you discover after you have made your website public that users cannot use it, you will lose customers and it will be costly to fix.

Customer support is expensive for a developer of a product, and if a customer does not know understand a product, know how to make it do what they want, or makes a mistake because it was complex to use, they will expect the developer to solve the problem for them.

In software the usability of a product in the field can be determined by analysing the calls from service. As an example, for one of the complex software products I have worked on, approximately 90% of the calls to service did not result in defects being raised on the code. The problem was that the software did something the user didn’t expect. Improved usability would enable the user to better understand the software and make fewer mistakes or know what the correct behaviour would be. A further analysis of these calls shows that the majority of these problems relate to customer knowledge or understanding. The users didn’t know or understand how to use the software. This clearly indicates a usability failing. A much smaller percentage relates to the customer having made a mistake in their own code or configuration. Again, this could be solved by making the software easier to use.

So how does this happen and what needs to be done to improve the usability of products?

In some development organisations there is the belief in a so-called ‘Perfect User’. This is where the developer of a technology believes that the user has the same experience and knowledge as they do. Not only do they have the same skills as the developer, but also they have a clairvoyant understanding of how this technology should work, and therefore how to use it. This is often reflected in documentation associated with the product, which describes how the software works under the covers, rather than how to actually use it! I like to call this “developer-think”, and this idea of the perfect user is a common belief I come across in my work. Of course, in reality the user of a piece of software or other technology is often very different to the person who developed it, and the user has very different goals. There are certainly cases where the user is equally skilled and knowledgeable, but there is a lot of evidence to suggest that technical skills are declining within organisations. Also people with deep technical skills and expertise are very expensive, and it in the interest of organisations to employ people with less skills, or to choose products that require less training for their staff. An organisation wants to quickly get up and running with a product, and to quickly start to make a return on their investment.

You have to understand the real users, what they are trying to achieve, what their capabilities and experiences are, and the environment they are working in. Also there are different types of user with different needs. An experienced user will have different goals and requirements than a novice user. How often is someone using your product? If they only have to use it once every six months for example, a complex procedure to perform a task will force them to re-learn the product every time they use it, whereas a more user friendly procedure will enable them to remember it, and quickly achieve the task.

The following section includes tips to help you include usability in your designs.

Using familiar concepts help a user to understand the software better and how to use it. For example office software often uses concepts such as filing cabinets full of records for databases, and the trash bin for deleting files. Naming objects or components sensibly in software can help a user work with them, for example servers in systems management, tables in information management. Making up new terms and using abstract references can make software more complex and difficult to understand.

Keep it simple. Some developers want to show the complexity, almost as a way of saying ‘look how hard this was to do’. Hide the complexity from the user, allow them to do the tasks they need to do with the minimum interaction. For example, just because there are 50 properties an object could have, if it only actually needs 5 to make it work then show the user just the properties they need.

Give easy access to the features that most users will use. There are some tasks that every user of a product will perform. For example, you will have heard the saying that 90% of the people will use 10% of a product. For example in a word-processing program all users will use the basic text editing and formatting capability, but only 5% might use the mail merge functionality. More advanced features can be made less visible to users. You should optimize the design for the most frequent or important tasks. It is critically important to understand how users are going to use the software, and why they want to use it. What are the tasks that the user wants to perform? Focus on the tasks that they want to do.

Design your web site or application so that users can view and directly manipulate objects or information within the interface. Choices should be visible to users rather than hidden with cryptic key sequences. When objects and choices are immediately visible, users learn and complete work tasks efficiently.

Provide the user with sensible default values when they are carrying out complex tasks. If the user has to enter information into an interface, for example a web form or a properties dialog, they are more likely to understand what to enter and make less mistakes if you provide good examples and sensible default values.

Feedback is very important to a user, particular in software or a web site. For example, if a user clicks on a button and there will be a wait, use a message, an egg timer or progress bar to indicate something is happening. Otherwise the user might repeatedly click on a button and cause unexpected results, or give up waiting altogether.

Provide meaningful information in responses, for example if an error occurs display an error message explaining what went wrong, any consequences, and what they need to do next or to fix the problem. If they need to go elsewhere such as an error log to find information, then make sure the message tells them where to find it. There is nothing more frustrating than getting a message telling you something went wrong but you have to go somewhere else to find out what, and then you don’t know where to find it! Ideally put logs in a standard easy to find location on a file system.

Be consistent. Make sure that objects that look similar behave in similar ways, and an action should always produce the same result. Also provide a way for the user to get out of a situation, for example some way to reset the settings to default, and in software provide undo and redo actions. A user should be able to make mistakes without fearing causing permanent damage, such as permanently changed settings, or data loss. Don’t let your user be in a position to actually damage their system or data. Users will do things you don’t expect them to!

Provide help to the users where they need it. For example when the user is configuring something, for example a new camera, a small ‘get started’ type of booklet or card can help. On a software interface, providing sensible labels, and a sentence explaining what an interface element is, can make it more intuitive to use, with the user having to resort to reading the online documentation to know what to do next.

Good visual design on its own is not usability, but it can help the usability and should be included in the design process. For example, a cramped interface can be difficult to understand, and may overwhelm the user. Tiny buttons that all look the same are equally not useful, whereas the use of sensible space and well designed icons placed in a logical order can help to lead a user through a task.

You can’t talk to every customer, but there are various techniques that are used in the design process to help understand users better. These techniques described in the following section.

The first is to define and understand user roles. For any given product there is likely to be one or more user roles that will use that product. Each user role will have particular tasks they perform or goals that they want to achieve. They may also have a set of associated skills, and they may work in a particular environment. Examples of user roles might be:

  • Web developer
  • Systems administrator
  • Technical support officer
  • Lab assistant

Personas are an extension of user roles where you create a fictional character that represents a group of your users. A persona typically includes the following types of information about the fictional user:

the user’s name and information about their age, education, experience, and possibly even include a picture and more demographic details.

  • the user’s job title and major responsibilities are included, and details of relevant skills, and other related product use – for example, software, hardware, platforms?
  • what their goals and tasks are with using your product, what is their organisation, why do they want the product, what’s the problem/business opportunity that it solves?
  • the user’s physical and social environment. Where do they work, with what facilities, how many sites, how many people, how many servers, and so on?

A story is used to detail what a user wants to do with your product, or a part of your product. It describes just what the goals and tasks are, and not how a particular user might use the product.

Scenarios extend stories by using a specific user, for example based on a persona, to detail specific requirements that that user may have, or a specific way that this user uses your product based on their existing knowledge and experience, or particular environment. A story tends to focus on the so-called ‘golden path’. A user wants to achieve a particular goal, and the steps for them to do that are likely to be considered. With the scenario problems with the golden path may be highlighted, and alternate paths may need to be explored to support users with the same requirements as the persona.

There are a number of ways that you can get feedback from users at each stage of the development process. Questionnaires and market research can be used to find out about customer satisfaction for products that have already been released. Questionnaires can also be used in association with early release programs and prototypes of software, websites or other products. At this stage though, it may be too late to make major changes, but it can be useful to get ideas for improvements in a new release.

Design walkthroughs are used at the design and planning stage in development to get feed back on plans for a new product or early prototypes. This is the most effective time to get feedback from users because there is the opportunity to change the design and plan of a product. Users can tell you at this stage if a proposed function is useful or not, you can even save time by not developing something users have told you they don’t want or don’t like.

Decision support centres can be used at any stage in the process to get anonymous feedback and priorities from users. The idea is you can capture input from users anonymously and the users can then comment on other people’s ideas, and vote to determine popularity and priorities. In the design stage this can be used to capture feedback on proposed new features in a product, and find out which ones users like best, or think are the most important. In a testing or post-release evaluation stage you can find out what users like and dislike, what their priorities are, and what their ideas for improvements might be.

There are various techniques for evaluating the usability of products that are already released to customers or still under development. The effectiveness of the techniques is variable. Usability inspection is essentially comparing the software against various checklists that measure whether the product has been implemented in a way that would promote usability. For example in a checklist for assessing the usability of a website it might have items such as:

  • Is there a navigation menu?
  • Is the text readable?
  • Are links underlined?
  • Does the text all fit on the screen without needing to scroll

The problem with usability inspection is that just following usability guidelines like this doesn’t mean that something is usable. The goals that the user wants to achieve need to be considered, thought about and designed in addition to following good practice. This is the same as with graphical design on its own not being enough to make something usable.

Heuristic evaluation by a usability specialist is a better technique as the individual can make an assessment of a product based on their experience of other products and testing. A specialist is more likely to find usability problems and suggest more usable solutions.

Alpha and Beta programs, and remote user evaluations can help to get feedback from real users, at a time in the development process when there might be time to make changes. Feedback can be received individually through questionnaires, or open discussion where the users are invited to discuss their feedback together. To ensure that the feedback from the users gets into the product defects or requirements are raised.

Ideally a product will be evaluated at an earlier stage in the development cycle. The most effective way to get feedback on the usability of a product at any point in the cycle is with proper user testing. User testing can be used to get feedback on the usability of a product, even before development has begun. Ideally user testing will be performed with real users, but where this is not possible then other users can take the place of the real users. These test subjects should have similar skills or experience to user type required in the test, or a persona can be used to guide the test subjects in how to behave in the test.

Paper prototyping can be used at a very early stage. Drawings or words on paper are used to represent the interface that will be presented to the user. The developers can move paper parts of the prototype around during the test to demonstrate the interaction with the system the user will have. For example if the user presses a button, a new piece of paper may be placed in front of the first to indicate that a new window or page is displayed. If the user does an action that the developer hasn’t anticipated, then the paper prototype will quickly and inexpensively highlight the problem.

Other tools are available for more sophisticated prototyping such as smart board software, where a user can click an object on the smart-board and the image displayed on the smart-board changes to the correct image. Using tools such as Photoshop or PowerPoint interactive representations can be developed that look like a real interface. The user can interact with them in the same way they would with a real piece of software or web site. Using this more sophisticated prototyping is helpful because it is like using the real product, but is quicker to change. Feedback from one user can be used in the prototype for the next user in the test.

A proof of concept can be used to take a product and demonstrate how it can be used for a specific user. The full product does not need to be available for the proof of concept, but how it could work is being demonstrated. A product can also be taken to users in their own environment to get real feedback of how it will work for the specific goals and environment of those users.

Facilities for user testing can be as simple as a room and a facilitator with some paper for prototyping at its most basic level. User testing can however be much more elaborate depending on the needs of the product, and the time in the product life cycle where the test is being performed.

Different types of evaluation and usability testing techniques should be used at different points in the lifecycle of a product. The later in the cycle, the more expensive it is to fix any usability problems. The black section of the rectangle shows the stage when the product has been released and it is  very difficult to fix the problems.

There are a number of considerations to make when planning usability tests.

The first consideration is the users to use in your usability test. If you have real users then you will want to ensure you have a representative cross section. If you have other individuals then they will need to represent the real users you have, or your intended audience. It can be valuable to find users with a mix of experience, for example having some novice and some experienced users so that you can see how previous experience has an affect on use of the product. It has been suggested by Jacob Nielsen that the optimum number of users to run usability tests with is 5. I would agree that 5 is a good number for a usability test, and more than 10 is not useful, particularly if the product under test does not change between tests. You might think that carrying out lots of tests would be a good idea, but in fact what you find is that all users find the same problems, and after identifying these common problems what you see is individual user preferences.

Usability tests benefit from having at least one observer in addition to the specialist running the test. Having multiple observers is useful, as each person will spot different issues and can more affectively record them. You can also record the session, and play it back later to record all of the individual points. Having the product developers present as observers for the test is very valuable, because they get the opportunity to see first hand how their development decisions affect the user. The developers can also talk through alternatives with the user after the session to solve specific issues. Other individuals involved with the product can also benefit, for example testers and writers, because it is useful for these roles to understand how the user expects to use a product.

If the observers are visible to the user it is a good idea to keep the numbers down so as not to intimidate or distract them during the test. It is important that the observers do not interfere with the test, it can be very tempting to shout out the answer to the user when they are stuck – which does not help improve the usability.

For a usability test you need to have realistic tasks for the users to complete. It helps to have a scenario that includes details of the background goals of the task. Why the user wants to perform the task, and any details of their environment of significance to the task. If the test is not with a real user, then including the persona information is useful.

For the test the system or prototype should be set up in the state that the user needs it to start the task. It might be that the task involves installing and configuring a product, but you should ensure they can do this without encountering unrelated problems. For example, check that the network works correctly and the user-id has the appropriate access.

It is important to explain to the user what the test is about, and what you will do with the results. Real users in particular feel valued when their opinions are used to improve the products they have purchased.

It is also important to ensure that the user knows that they are not under test, but that the product is. Users feel stupid if they cannot work out how to use products. I have known people getting very upset and angry during usability tests when the poor usability caused them to feel stupid or frustrated. If a user from the target audience cannot use a piece of software is not their fault, it is the fault of the software, and the software’s developers. It can be a real eye opener for a developer when someone storms out of a session or ends up in tears because they can’t use their code! The user should fill out a background questionnaire on their role and experience. The user must be encouraged to talk through their session as they do it to explain what they are thinking, why they perform a particular action and what they are expecting to happen, if what they expected doesn’t happen and how they feel as they are performing the task.

In the test you need to observe and record comments and actions that the user took. You can also take measurements of efficiency, effectiveness and satisfaction. Time on task, number of clicks and numbers of errors made are examples of measures that you can make for a usability test on software. Talk the user through their experience with each task, and ask in person or using a questionnaire, what they liked, what they didn’t, what they would change, and how satisfied they feel, for example as a mark out of 10.

The results from user tests can be very effective ways of making improvements and getting changes made to a product. Observational results often have the biggest impact, especially if the product developers see the test, either as observers or on video. Quotes and emotional responses are very powerful. ‘I don’t know what to do here’, ‘This is rubbish’, ‘I expected xxx to happen when I clicked that’, ‘I’m pulling my hair out here’ and so on are examples of an emotional response which demonstrates the users frustration. Quotes and observations are good to communicate the results in a presentation or report.

Charts are also an effective way of reporting the results of a usability test. For example a chart showing a comparison of the average time users spent trying to complete a particular task can indicate which tasks are easier to complete. If you have more than one prototype you can ask a user or different users to perform the same tasks using the different prototypes and then compare them. You can also produce charts showing number of defects found, number of mistakes the user made, the number of mouse clicks to achieve tasks, and the user’s satisfaction with particular tasks.

A usability test should produce a list of usability defects, or usability requirements. You can take an iterative approach to usability testing where the defects from the first few sessions are fixed, and then more usability tests are performed with the updated prototype or system. This enables you to more quickly and effectively improve the usability, and get feedback on a new interface.

When you develop a product you should always be considering who your users are, what goals they are trying to achieve, and how you can develop your product in a way that makes it easier for those users to achieve those goals. It may be tempting to expose the complexity to the users to show how clever the product is, but your users won’t thank you for it. You can test the usability of a product at any time in the lifecycle of that product using a number of techniques, but the most effective way is to get real or representative users in a room with a realistic scenario for them to complete. Real users do not necessarily follow the golden-path when they use a product, and they will try and use your product in ways you did not expect. If you can spot the problems early in the development cycle it will make it much easier and cheaper to fix, and help you create products leading to happy, satisfied customers.


Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s