Assessing quality is important if you want to know whether you are achieving your quality goals, but in the case of technical documentation, what do you measure? It could be readability, usability, comprehensiveness, reduction in support calls or more product sales. It could be all of these or something else.

If you check the finished document against your specifications and standards, you can judge whether you have met certain quality criteria by conformance. But this does not mean you have a high-quality technical document, because specifications themselves can be flawed or incomplete.

Can Quality be Measured?

In manufacturing processes, there are various ways in which product quality can be measured statistically. Where technical documentation is concerned, things are not so clear. Quantifying errors is possible, but the effective use of statistics requires more than numeric analysis. Some sense must be made of the data in a way that can identify problem areas and courses of action.

In manufacturing production lines, it can be straightforward to identify a cause of non-conformance, such as an assembly or materials fault on a piece of equipment, but identifying causes of non-conformance in technical documentation is much harder.

For one thing, technical documents can be inherently complex in terms of the range and variety of content and uses. Assuming one can identity a quality fault, finding the underling cause won't be so easy. In fact, without a good feedback system, failings in quality within technical documentation may never be apparent to the organization that produced it.

Some aspects of documentation quality can be assessed both qualitatively and quantitatively and these are discussed briefly in this article.

Feedback

One method of assessing documentation in a qualitative way is from feedback. Technical documentation is predominantly used for instructions and instructional manuals. By definition its content is technical in nature, and the reader seeks to be informed usually about a product, process or system. Communication takes place only when two minds really meet. When the reader receives the same ideas and information the author wished to convey, there is understanding; if the reader receives no ideas, there is no understanding; if different ideas, there is misunderstanding. Thus the fundamental nature of technical communication offers a clue as to why feedback is a useful measure of quality, for documentation can either succeed or fail to inform. Feedback from various sources can be indicative of the level of understanding readers have of the documentation.

Feedback from Users

Most user feedback is spontaneous and it is usually obtained through one of two channels-marketing and support.

From a marketing standpoint, suggestions for improvements in documentation may be driven by unmet expectations, and these can lead to new requirements being specified that are fed back into the documentation design process. This is shown in the figure below as the upper feedback loop through sales support.

An example of this kind of feedback would be a request from a user to have the documentation produced in a different format, which, in their view, would be more convenient.

On the technical support side, users report errors or problems directly associated with the documentation's content. This is shown by the lower feedback loop. An example of this kind of feedback would be a user contacting a technical support technician to obtain help about something that the documentation has failed to explain clearly, or has explained incorrectly.

Feedback from either channel adds to the resources used for the development of documentation. It is information that can be used in the next documentation project to improve one or more aspects of the quality system.

The figure above illustrates the point that feedback is an important input to the quality system. The feedback process can be one that directly influences documentation design at all levels-it can help 'fine tune' the content by highlighting minor errors, inconsistencies and omissions, or result in a fundamental reorganization of the technical documentation.

The Problem of Getting Feedback

Sometimes, the problem for the technical communicator is how to get feedback. Even if you design your documentation to meet user expectations, it may be difficult to judge how successfully this has been achieved. Satisfied users may never provide feedback and, unfortunately, it is also true that unsatisfied users may never provide feedback.

For example, if a consumer buys a DVD player that has a faulty component preventing proper playback, he or she is unlikely to tolerate the fault. The quality issue will be reported and remedial action taken somewhere in the supply chain.

By contrast, if the manual that accompanies the DVD player is ambiguous or misleading, the consumer might be frustrated or annoyed, but provided the DVD player works satisfactorily, it is unlikely any action will be taken to report the poor quality documentation. Even if the documentation singularly fails to be useful, the consumer may choose to resolve the knowledge gap by trial and error, or seek another source of help.

In addition to techniques mentioned previously in this book, here are two ways in which post-publication feedback can be acquired:

Satisfaction Surveys. This involves the use of questionnaires, complaint report forms, customer service telephone calls, site visits, online forums, support group meetings, and so on. In some cases, incentives may be required to encourage users to respond.

When using schemes such as these to prompt for feedback, survey questions must be framed carefully to avoid bias and not confront users with questions they could either misinterpret or fail or comprehend. When analysing the feedback, it is important to have a balance of input from all different user types, so that the results give a fair representation of the readers and users of the documentation.

Support Analysis. An important source of feedback for technical documentation comes from the various lines of support available to users.

If technical documentation is doing its job, then the need for alternative sources of information, such as technical support, will be diminished. Therefore, the level and type of query that is fielded by other support mechanisms, such as telephone or email support, online forums, maintenance engineers, trainers, technicians and so on, will provide at least some measure of documentation quality.

Measuring Feedback

Surveys and support analysis can be useful sources of information about documentation quality, but there should be a mechanism by which the data can be analysed accurately and in a meaningful way.

Finding the right level of feedback is also important. All readers who raise quality issues should be listened to, but the proportion of readers they represent is also significant. If ten readers complain, there might be quality improvements to be made to the documentation, but if they represent only ten in 10,000, it is fair to assume the documentation was generally successful; after all, no documentation is perfect.

It is also important to prioritize problems, so that quality improvements are delivered where they are most needed and therefore in a cost-effective way. Perhaps only a few classes of problem are responsible for the majority of calls. For example, 50% of support calls might be due to issues relating to only 20% of the documentation.

One way to measure this is to use a technique called Pareto Analysis. Typically, this involves a histogram arranged so that the leftmost column represents the class with the highest frequency, while the rightmost column the lowest frequency. An example is shown below.

The diagram represents the ranking of the various classes and helps identify the major sources of potential quality problems and therefore those that require most attention. In the example shown above, half of all technical support calls are concerned with installation issues, so documentation covering that aspect should be the focus for quality improvement.

It takes time to record, analyse and act on the information available from the support department, but properly used, technical support information can be extremely helpful for the future design and maintenance of the documentation.

Feedback statistics are only helpful when relationships can be identified between causes and effects. If the support department of a software company receives a few hundred telephone calls complaining about a baffling configuration routine, the solution might be clearer documentation. But one cannot assume that the instructions are at fault; there could be fundamental design faults in the routine itself, which prevent it being more intuitive.

Quality Statistics for Content

Various techniques can be employed by technical communicators to measure the quality of written material using metrics and statistics that focus on the structure and language used. Some proprietary word processors provide a measure of readability based on one or more recognized tests, such as:

Flesch Reading Ease. This computes readability based on the average number of syllables per word and the average number of words per sentence. Scores range from 0 to 100. Standard writing averages approximately 60-70. The higher the score, the greater the number of people who can readily understand the document.

Flesch-Kincaid Grade Level. This computes readability based on the average number of syllables per word and the average number of words per sentence. In this case, the score indicates a U.S. grade-school level. For example, a score of 8.0 means that an eighth-grade student would understand the document. Standard writing approximates 7.0 to 8.0.

Coleman-Liau and Bormuth Grade Levels. These are similar to the Flesch-Kincaid test in that they measure readability by grade-school level, but instead use word length (in characters) and sentence length (in words) to determine the value of the level.

Use of Passive Voice. It is generally considered more effective to write in an active rather than a passive voice. Therefore, a measure of the percentage of passive sentences is often used as a quality metric. Active voice emphasizes the agent of the action and is therefore more direct. For example, 'Press the Play button' is written in active voice, while 'The Play button can be pressed' is written in passive voice.

Another method of content quality analysis is based on the evaluation of variables in a handful of sample pages for each section of the document, excluding preliminaries, table of contents, appendices, glossaries and indexes.

The process involves counting the occurrences of certain positive and negative attributes found within the sample pages, based on a predefined set of technical writing techniques and characteristics. To help clarify, here are just a few of the attributes that might be counted:

• Jargon words, which may be difficult to understand for some readers, such as 'calibrator'.
• Noun strings, which can be difficult to read, particularly if they consist of three or more successive words, such as 'database menu function'.
• Multiple action sentences, which contain instructions to perform several tasks in succession. These can be difficult to assimilate. For example, 'While holding the CTRL key, point to the Input menu, click Start and then click OK to activate the batch process, then release the CTRL key.'
• Task-oriented headings that can improve the usability of a document. There is some evidence that headings which refer to tasks users want to perform have a positive impact on usability.
• The number of relevent, integrated graphics in the document, which can aid understanding.

Counts of attributes such as those listed above are used to determine the percentage occurrence in each case. An overall quality percentage can then be calculated. Where counts are made for negative features, the results are converted to a positive percentage. For example, if 20% of the document contains jargon words, then 80% of the document does not.

Like any system that attempts to measure the written word, a good deal of the analysis is subjective. Text can be rewritten to make the statistics look good, but whether the quality of the documentation is actually improved depends on your view of quality. This is why defining quality at the outset of a technical documentation project is so important.

Measuring Quality for Projects

With sufficient data gathered over a suitable timescale, it is possible to analyse the relationships between variables at project or even company level. When technical documentation is designed to provide instructional guidance to users about products, one of the quality goals could be to reduce the costs and levels of support provided through other channels, such as telephone advice, training, maintenance visits, and so on. The only way to be sure this quality goal is achieved is to gather data and measure the results.

Having embarked on a programme of quality improvements for technical documentation, it would be useful to know whether real differences are being made. The key to measuring quality at project level is to look for correlation between the variables. Assessment should be made from different viewpoints. It might be easy to detect a reduction in the frequency of support calls, but even if the total number of support calls does not decrease, other variables may change. Perhaps the average duration of calls is less. Perhaps the nature of the calls has changed and they are easier to deal with. Perhaps the sales of products have increased, but the level of support has not.

Whatever the outcome of your quality assessments, either positive or negative, use the information as input for your next documentation project.

Awards, Prizes and Recommendations

One way to obtain recognition for high-quality technical documentation is by gaining some seal of approval from a suitable institution. There are various schemes that reward achievements in technical documentation. The process of assessment and the presentation of awards is usually organized through professional bodies, such as the Institute of Scientific and Technical Communicators (ISTC) and the Society for Technical Communication (STC).

Recommendations and approvals from recognized authorities can also help give a positive image for quality. For example, the Crystal Mark from the Plain English Campaign acknowledges clarity in communication of public information, while the World Wide Web Consortium's own quality assurance programme can validate and approve web documents that conform to their standards and recommendations.

At a higher level, there are major awards and prizes for quality available to the organization as a whole, such as the British Quality Foundation's Business Excellence Award in the UK, and the Malcolm Baldrige National Quality Program Award in the USA.

Competing for such awards and striving to get industry recognition helps keep the philosophy of quality in the foreground. Technical communicators have their part to play in raising standards and improving every aspect of technical documentation for the benefit of all involved.

Andrew J Marlow BA (Open), PGDTech (Open), MSc, DBA, MISTC

Article based on content of the book Quality Control for Technical Documentation, 2005 (ISBN 1-873407-09-2).

Copyright © 2006 Andrew J Marlow. All Rights Reserved. Reproduced for STC UK Newsletter by permission of the publisher.