Everyone knows that it is important to test software so that you find any bugs before it goes out to customers. The majority of organisations that develop software have people who spend at least some of the time doing testing their software. Many may also do ‘user acceptance’ testing to check that the software meets the requirements of the users. And many also do code review of some sort to check that the code meets company requirements or best practices.
However it is very unusual for the user documentation associated with that software to undergo proper testing. From my own experience and talking to other technical communicators, I would say that less than 10% of organisations actually do formal testing of the documentation. Review of the documentation is more common. In my own experience review tends to come in two forms, one is peer review, or review by an editor for style and grammar, and the other is review by a ‘technical expert’ which is usually a quick scan for technical problems with concepts and a further grammar review. Sadly the majority of developers I have worked with have far more interest and knowledge of grammatical pedantry, than they do for the needs of their users..
So who cares you may say? Who cares about the documentation anyway? Well of course it is that view of the documentation as being something separate and different from the software that means they are treated differently. It is important to remember that to a user the documentation and the software are the same thing. The documentation is how they learn to use the software, and if you software is anything other than entirely intuitively usable (believe me your software won’t be), then your users rely on it. So why bother testing the documentation? Because if it is
* difficult to understandable
* difficult to find
Your users won’t be able to use your software!
I feel very passionately about the subject of testing user documentation from my background in testing, usability, and as a technical communicator. So much so that I was editor of the ISO/IEC/IEE standard on the subject. Below I’m going to give an introduction to testing and reviewing user documentation, but refer to the standard here for more detail and specific requirements for organisations to follow to ensure their documentation is tested and fit for purpose.
Formal review of the documentation is the most basic step in checking the documentation. There are three types of review, and they can pick up different kinds of problems. The first type of review is a ‘Peer review’ or ‘Buddy check’ that checks the documentation for:
*Ease of understanding
*Conformance to local style guides and consistency
*Editorial consistency and correctness
Peer reviews are more common in organisations where there are multiple members of the writing team. These reviews are usually carried out by members of the writing team or an editor. They require the reviewer to have expertise in technical communication, and the organisational conventions and requirements.
The second type of review is usually carried out by a technical expert often a developer or tester to check the following aspects of the documentation:
*Technical accuracy (concepts/reference)
*Safety and security
The technical experts should be encouraged to check the technical content of the documentation, they are not their to provide style or grammar advice. They should not simply check that the documentation matches the specification. The specification may have mistakes or be missing information or tasks that the user needs to perform.
The final type of review checks whether the documentation ‘works’. For example, for online Help this might be checking that the navigation displays correctly, that images are displayed and readable, or that links work correctly. If the documentation has been translated or localised you would want to check that the different versions display correctly and are not corrupted, and that the translations are correct. These types of reviews can be considered the equivalent for documentation of a ‘unit test’ for software.
So you might be asking yourself, isn’t a review enough?
The answer? No!!
*A peer review or edit won’t tell you if the documentation is correct or relevant
*A technical review won’t tell you whether step-by-step instructions and tasks support the user
*A review may not even use the documentation in the form it will be delivered
So what do we do next? The answer is to perform a system test of the user documentation. This is testing performed with both the software and the documentation to evaluate that the documentation is fit for purpose and supports the users sufficiently in their use of the software. The same as for software, the system, test must be carried out by a tester who is not the same person who developed the specific software or documentation component under test. Typically a tester will test a function of the software using the documentation to perform the test. In this way they can identify where the documentation and the software are different, meaning that either the software or the documentation may need to be fixed. As well as finding bugs, the tester will also identify any information that is missing from the documentation, for example any missing concepts, prerequisites, steps, examples, and so on. This is a hugely valuable activity because the tester is acting as the first user of the software. If they need information that the documentation does not provide, then there is the opportunity to add it for the benefit of the user.
If you’ve read my posts or articles on agile, you will know that a technique I strongly recommend is ‘pair programming’. Actually, its more pair testing. Have the writer and the tester sit together while the test is going on so that the writer can see for themselves what the tester is doing. Doing this they can get feedback, capture any steps they didn’t know about, and also get an insight into the concepts the user is expected to know, and the tasks they are likely to want to complete. This can also work well for the technical review if you have a technical expert that doesn’t have time to look at the docs. Set up an hour with them, show the docs and write down their comments. You can fix the problems, get new info, whilst minimising the amount of work your developer has to do.
So, is system test enough?
You guessed it! NO!!
The system tests determine whether the documentation is technically accurate (verify), but does not make a judgment on whether the documentation was usable (validate). A system test is incredibly valuable, but is performed by people who already know how to use the documentation and the software, and where to find the information, and potentially how to work around certain issues. The only way to be sure that the documentation is actually findable, understandable, and usable is to conduct a usability test.
Usability testing of user documentation is the most acceptable method of checking that the information provided in the documentation meets users’ needs, and that users can find it, understand it, and apply it. You test the documentation using real or representative users. Usability testing of user documentation ensures that the documentation:
*is sufficient to allow users to perform required functions using the system.
*is fit for its purpose and meets the user’s needs
You can do usability testing on:
*Software + user documentation together:
*installing & migrating instructions
*search and findability
*Tables of contents
To do usability testing you need to come up with realistic tasks for the test subjects. To design these tasks you can use tools such as personas, user stories, and scenarios. You may also be able to get customer facing colleagues such as sales, marketing, and support personnel to help by telling you about the business goals and tasks that your users have or have asked for assistance with.
You don’t need very sophisticated facilities to conduct user tests. But make sure that everything the user needs is set up before hand that you don’t explicitly want to test the user doing. For example, if you don’t want to test the user creating resources or installing patches, make sure you set these up for the user in advance.
You don’t have to use real users if you don’t have access to them, but you can use representative users. Ideally you want someone with the same kinds of skills that your users have, but that don’t have inside knowledge on the documentation or software that you are testing. New employees and testers or writers from other teams are usually excellent test subjects.
If you can only test with one user, do that, but four is an ideal number. Once you get above that number you see the same problems repeatedly and the other issues tend to be personal preferences. A good practice is to fix the problems identified after a session, so that you can check your fixes and move onto other problems in later sessions.
It is helpful to have other observers, in part to capture any problems you may miss. Also it is good for the developers, testers, and writers to see how the users use what they have provided. It can be a real eye opener, especially when the users are not as smart as you think, or try to do something you never imagined with your product!
To run the session, explain to the user what their role is if they are a representative user, and the task to complete. The task should not be step by step instructions, but a more high level goal. Remind the user that they are not under test – if they can’t do it, the product is wrong, not them! Resist the temptation to interfere with what they are doing if they go wrong or get stuck. Ask them why they chose a particular route or method, ask them what they expected to see or to happen.
Collect metrics: efficiency, effectiveness, & satisfaction. How long did a task take, how many clicks to find the information, how many times did they go wrong? What problems did they find? Did they get stuck, could they work around it? How happy were they? What suggestions did they have? What did they expect to find? And so on.
Take an iterative approach if you can run more than one session – run some tests, make improvements, run some more.
There are some other types of testing that you may need to do for your user documentation.
*Documentation must be accessible to the same users that are able to use the software product that the documentation supports.
*Documentation may need to comply with standards, requirements and legislation.
*If possible, include people with disabilities in the beta and usability tests of the product.
Translation and localisation testing:
If special localized versions are created, check they use:
*appropriate local language
Be wary of cultural issues, especially in examples and with images. For example, some cultures are offended by images of body parts such as bare feet.
Ideally a national of the target country should check the user documentation to ensure suitability. Testers of a localized version need to be familiar with the language and cultural conventions for the localization under test.
Refer to the standard here for information on planning and resources for testing. This is particularly useful for team leaders, managers, and project managers.
If you don’t think testing applies to books, think again. I’ve been working with a programming book, I’ll review it later so you can figure out which one, which has some serious problems that would have been found through testing. The problems are so severe, I know that they cannot have been tested by someone with out inside knowledge. I suspect that it wasn’t actually tested at all. This uses code examples. Unfortunately the approach appears to have been to start with the final code and then change it for use in earlier chapters. This means the code is wrong, often incomplete, and there are missing steps. If this had been at least system tested, the problem wouldn’t be there. And a usability test would have told them that they needed more code listings, more images, and more information on what was supposed to happen when you test the code!
Testing the user documentation is as important for your users as testing the software!! When you get frustrated with the documentation for a product – remember if you don’t test your users are having that same experience!
</end of sermon>