Controlled readability of Seveso II company safety documents, the design of a new KPI

Abstract

Companies often use a substantial quantity of safety-related documents such as the front gate safety flyer, emergency evacuation instructions, work permits, safety procedures, work instructions and policy statements. In many cases the personnel magazines and message board notes also contain safety information. The authors and readers may not be in the same department, premises or cultural group. Both authors and readers are diverse groups when it comes to reading, writing and language skills. Previous research has found that Seveso II companies produce documents that are difficult for their workforce and visitors to understand. Authors do not write systematically to match the reader’s skill level. This may be due in part to the quality and effectiveness of layout guidance, expert linguistic advice and document appraisal systems. These do not in general provide the immediate support needed. Layout suggestions are of limited effectiveness, and availability and usability limits the value of expert advice and appraisal systems. So, without useful feedback, many authors cannot write sufficiently readable documents; to overcome this threshold, they need a quick document readability assessment tool. With such a tool, readability can become a controlled property of safety documents. Large companies may use hundreds or thousands of documents containing safety information. This paper presents a practical approach for the transition to, and monitoring of, controlled readability for all documents related to safety. A new Key Performance Indicator design on readability is proposed.

Full text

Controlled readability of Seveso II company safety documents, the design
of a new KPI
q
P. Lindhout
a,*
, J.C. Kingston-Howlett
b
, B.J.M. Ale
b
a
Ministry of Social Affairs and Employment, AI-MHC, Anna van Hannoverstraat 4, P.O. Box 90801, 2509 LV The Hague, The Netherlands
b
Delft University of Technology, TBM-Safety Science Group, Jaffalaan 5, 2628 BX Delft, The Netherlands
article info
Article history:
Received 9 July 2009
Received in revised form 14 January 2010
Accepted 11 February 2010
Available online xxxx
Keywords:
Readability
KPI
Safety document
Language issues
Literacy
Multi-language shop floor
Diversity
Industrial accident
Major hazard
SEVESO-II
BRZO 1999
abstract
Companies often use a substantial quantity of safety-related documents such as the front gate safety
flyer, emergency evacuation instructions, work permits, safety procedures, work instructions and policy
statements. In many cases the personnel magazines and message board notes also contain safety infor-
mation. The authors and readers may not be in the same department, premises or cultural group. Both
authors and readers are diverse groups when it comes to reading, writing and language skills. Previous
research has found that Seveso II companies produce documents that are difficult for their workforce
and visitors to understand. Authors do not write systematically to match the reader’s skill level. This
may be due in part to the quality and effectiveness of layout guidance, expert linguistic advice and doc-
ument appraisal systems. These do not in general provide the immediate support needed. Layout sugges-
tions are of limited effectiveness, and availability and usability limits the value of expert advice and
appraisal systems. So, without useful feedback, many authors cannot write sufficiently readable docu-
ments; to overcome this threshold, they need a quick document readability assessment tool. With such
a tool, readability can become a controlled property of safety documents. Large companies may use hun-
dreds or thousands of documents containing safety information. This paper presents a practical approach
for the transition to, and monitoring of, controlled readability for all documents related to safety. A new
Key Performance Indicator design on readability is proposed.
Ó2010 Elsevier Ltd. All rights reserved.
1. Introduction
There is an underestimated and uncontrolled risk in industry:
language issues. One manifestation of this is safety-related docu-
ments with poor readability. (Lindhout and Ale, 2009) Companies
need to address this issue and must adopt readability in their pol-
icy on safe work. Moreover, the present authors contend it is rela-
tively simple to reduce this risk. This paper provides the design of a
generic Readability KPI for companies using safety-related docu-
ments. The fairly general denominator ‘‘safety-related documents”
encompasses a vast collection; for the practical purposes of a KPI, a
narrower scope is needed. In this paper, the following definition is
used ‘‘safety-related documents belonging to the safety manage-
ment system and to operational procedures and instructions re-
quired to execute work in the presence of danger”.
It is the intention of the present authors to publish on the de-
sign of the KPI as soon as possible, and to follow that with a second
paper on the application and evaluation of the readability KPI. The
main reason for this approach is that it is likely to be some time be-
fore companies start implementing the kind of KPI we propose and
can build up experience data on which to evaluate the performance
of the KPI and its effect. The present authors see the current paper
as the first step towards industrial acceptance of this novel KPI and
wider appreciation of the risks associated with language issues.
Readability represents a limiting factor in the further improve-
ment of safety performance. The successive reductions of incident
rates in the process industry over the last decades were first
achieved with technical means. After that safety management sys-
tems brought further improvements. Current incident rate reduc-
tions revolve around safety culture (Shell, 2002). Some 5 years
ago, major accidents in companies in the Netherlands were caused
predominantly by ‘‘maintenance and non-standard operating con-
ditions” (Dees et al., 2004). Today, research shows that the main
causal factor in incidents is ‘‘procedures and instructions” (Arbe-
idsinspectie, 2009). This implicates safety-related documents and
their properties, including readability.
The category ‘Safety-related documents’ may correspond in
many companies to a substantial number of actual items. Even in
small companies, dozens of documents fall into this category. Large
companies may use hundreds or even thousands of documents
0925-7535/$ - see front matter Ó2010 Elsevier Ltd. All rights reserved.
doi:10.1016/j.ssci.2010.02.011
q
Key Performance Indicator.
*Corresponding author. Tel.: +31 70 333 4444; fax: +31 70 333 4033.
E-mail address: PLindhout@MINSZW.NL (P. Lindhout).
Safety Science xxx (2010) xxx–xxx
Contents lists available at ScienceDirect
Safety Science
journal homepage: www.elsevier.com/locate/ssci
ARTICLE IN PRESS
Please cite this article in press as: Lindhout, P., et al. Controlled readability of Seveso II company safety documents, the design of a new KPI. Safety Sci.
(2010), doi:10.1016/j.ssci.2010.02.011

containing safety information. The category is also broad with re-
spect to the format of safety-related documents. In many cases,
company personnel magazines and messageboard notes contain
safety information. Formal safety-related documents are also quite
variable in their style, varying from the front gate safety flyer, in-
tended for use by all visitors and contractors everywhere on the
premises, to policy statements to be found in the management offi-
ces. Many of these documents affect the daily jobs of everyone on
site. The variety of documents is accompanied by a wide range in
document complexity. Policy statements and company-wide safety
procedures reflect the upper complexity level of safety documents.
Highly educated personnel, well-acquainted with the company
organisation, its technical installations, the processes and their
safety risks, would be both the authors and the readers of these
documents. The readership extends to external parties like insur-
ance brokers, consultants and regulators. According to Dutch law,
all of the company’s employees and their representatives have
the right to review safety policy documents.
At the lower complexity level of safety documents are examples
such as evacuation signs, hand signals and alarm procedures. All
persons present on site must be able to comply with emergency
procedures. This is realised through a mixture of emergency plan-
ning documents, shut-down procedure documents, emergency
evacuation drills and visual signs indicating emergency exits,
fire-fighting equipment and gathering point for the head-count.
Everybody must be able to use them even in highly stressful
conditions.
As already indicated, the readership of documents is a diverse
group and not just employees of the company producing the doc-
ument. Contractors are part of the readership, often involved in
maintenance activities or modification project work, they receive
work permits, safety procedures and work instructions. Incidental
visitors such as contractors, customers and suppliers may come
from abroad. They also need to comply to company safety instruc-
tions and, depending on their task, instructions specific to their
work. Warehouse employees and truck drivers are directly in-
volved in the handling of dangerous goods. Contractor engineers
and technicians are at the forefront when installations are in
non-standard operating conditions but still contain hazardous
materials.
Ideally any individual who needs a piece of safety information
should be able to understand the document that contains it. Any
deviation from this ideal raises ethical and regulatory questions
as to whether the associated uncontrolled danger can be justified.
Hence, as well as avoiding the loss of productivity and financial
damage caused by miscommunication and other language-related
human errors, assuring high standards of readability are also con-
sistent with good corporate governance and regulatory compli-
ance. However, recent research conducted in Dutch high risk
chemical companies demonstrates that readability of safety-re-
lated documents is poorly managed. That study found about half
of the documents assessed were insufficiently readable (Lindhout
and Ale, 2009).
There are several reasons why this problem exists. Authors face
the difficulty to write a document that accommodates a diverse
readership. The readers may not be in the same team, installation,
warehouse, premises, company or not within the same cultural
group within a company (Schein, 1996). This diversity extends to
levels of literacy amongst authors and readers. Safety documents
are often written by a safety engineer or manager, work permits
by a maintenance engineer or manager. A journalist with linguistic
skills may write personnel magazine articles on safety issues. Se-
nior managers, with a financial, marketing or technical back-
ground, would write a safety policy document. A process
engineer, possibly with a degree in chemistry, might be expected
to write detailed work instructions. Only in a relatively few cases
is the author of a work instruction a member of the workforce
who will use the document in practice.
With the noted exception of personnel magazines, there is little
evidence that authors address systematically their readers’ skill
levels when writing documents. Authors wishing to tailor their
writing to the needs of a particular group will find a confusing ar-
ray of layout suggestions, expert advice and document appraisal
systems, many of which are very complicated and presume knowl-
edge beyond the writer’s expertise. Guidance on how to write
‘‘plain language” is available in many countries, but immediate
feedback about attempts to do that in the workplace is far less
available. It is noted that layout suggestions, no matter how care-
fully considered, are of limited effectiveness in improving
readability.
In summary, readability suffers when authors have little guid-
ance or feedback.
Writing better for diverse readers in a systematic way requires
measurement and managing readability of safety documents and
getting feedback on readability. Lindhout (in press) propose a
new integrated readability measurement scale, linking text and
graphic document elements. This Linked Linguistic and Layout ele-
ments document appraisal scale, further referred to as ‘‘L-scale”, al-
lows fast appraisal of both text complexity and graphical elements
within a single numerical value; it is advocated here as a suitable
tool for measuring and facilitating the management of readability.
Existing open source software offers free on-line calculation of text
indicators for the English language areas. (e.g. see: added-
bytes.com). Further development of a free on-line document ap-
praisal tool for the Dutch language areas on this basis is feasible.
With a practical assessment tool, it becomes feasible to make
readability a controlled property of safety documents. But disci-
pline is also a factor, writers would need to use the tool for each
revision of each safety document. Improvement in readability also
depends on verification that all authors are using the tool the right
way. Furthermore, to safeguard standards of readability, compa-
nies need to communicate regular verification information through
a suitable reporting structure. This is a necessary condition to
establishing an indicator of consistent performance.
Companies frequently use KPI’s (Key Performance Indicators) to
measure their performance. The readability of documents qualifies
for being monitored by a KPI since this can be done while focussing
on an agreed company-wide goal, without dictating how the re-
sults should be achieved (NIST, 2008).
Design and development of KPI’s requires a systematic and val-
idated approach. This is needed to ensure that the KPI actually
indicates the variations of the subject phenomenon as intended.
The publication of guidelines by the UKs Health and Safety Execu-
tive (HSE), entitled ‘‘Developing Process Safety Indicators: a step-
by-step guide for chemical and major hazard sites” is helpful here
(HSE, 2006). This paper reviews several other sources, including
that provided by the OECD, and integrates them through the devel-
opment of a generic readability KPI. The context of the study is Sev-
eso II companies, but the present authors believe that the KPI could
be applied elsewhere.
Within this study, the Readability KPI is considered to be part of
a risk control system. Most companies will be faced with the chal-
lenge to control readability of many documents written and re-
vised every once in a while by many different authors. The risk
to be controlled is human error provoked by over-complicated
and confusing text. The present authors found no basis for a ‘‘sam-
pling system” to reduce the number of safety documents to assess
and control. In order to control the risk, 100% of the documents
must be controlled for readability. This is one of the key require-
ments of the readability KPI. To segregate from other texts, those
safety-related documents to be readability-controlled, a registra-
tion system is needed. After registration as ‘‘safety documents”,
2P. Lindhout et al. / Safety Science xxx (2010) xxx–xxx
ARTICLE IN PRESS
Please cite this article in press as: Lindhout, P., et al. Controlled readability of Seveso II company safety documents, the design of a new KPI. Safety Sci.
(2010), doi:10.1016/j.ssci.2010.02.011

their readability must be assessed for the first time. The first com-
pany-wide readability data is likely to show a widely distribution
of readability similar to the results found by Lindhout and Ale,
(2009).
The document readability statistics must be then brought to
acceptable levels. The generic readability KPI is designed to take
into account both the initial registry progress and the readability
levels achieved in the controlled documents later on during the
monitoring phase. The KPI is designed to indicate disturbances
from the set target level and so it signals any deterioration of con-
trol. In this way, companies obtain oversight on the readability of
safety-related documents at regular intervals.
2. Readability as a problematic risk
The present authors consider insufficient readability of safety
documents to be a problematic risk. Detailed requirements on
the use of specific languages, the use of pictograms, handsigns
and safety labels were introduced in Environmental, Occupational
Safety and Transportation legislation (Council Directive, 92/58/EC).
In much the same way, literacy, readability and compliance with
safety-related documents are mentioned by, and interconnected
in, state of the art safety management standards (OHSAS, 2007;
NTA 8620).
Research among companies, designated by the European Com-
mission Seveso II directive as presenting a ‘‘high risk” to society,
has shown that these companies are not dealing with the risk asso-
ciated with insufficient readability, let alone controlling it. All the
more striking is the fact that – when specifically asked – the vast
majority of these companies acknowledge the relation between
language issues and safety risks. Various specialised consulting
companies offer on a commercial basis, linguistic advice on how
to improve readability of specific documents. However, this advice
is found not to be used by Seveso II companies (Lindhout, in press).
Language issues result from a number of factors, among them,
multi-language shopfloor environments. Other factors include
low literacy problems among both indigenous groups and foreign
workers; these can be related to a variety of medical-, social- and
environmental conditions. In the 1970s, much effort was directed
to ‘‘Analphabetics”, people who can neither read nor write. Around
the year 2000, the political focus shifted to foreign citizens settling
in and acquainting themselves with the Dutch language. In 2004,
the Dutch government moved its attention to low literacy among
the indigenous Dutch population and started a national literacy
campaign, which is still running. At least a million people belong-
ing to the indigenous population and half-a-million people from
the foreign population, have literacy problems; some 10% of the
adult population (Stichting Lezen en schrijven, 2004). Even these
numbers may underestimate the problem. The IALS (International
Adult Literacy Survey) results show that 26% of the Dutch adult
population do not reach the IALS-3 start qualification level for
the information society adopted in the Lisbon Agreement (Houtko-
op, 1999NSR, 2006–2008).
The demographic and economic developments in the European
Community have led to bigger influx of foreign workers. This leads
to a greater incidence of illiteracy and other communication prob-
lems (EU Parliament, 2001). Where safe work depends upon work-
ers understanding written and verbal instructions, on following
procedures and discussing work; these problems can manifest as
accidents and incidents (COT/DHV, 2004). All this underlines the
importance of controlling risks associated with language issues in
Seveso II companies.
To some extent, literacy can be managed through selection and
training. As discussed later, we contend that controlling readability
provides companies with a complementary and more effective
means of managing the effects of variable levels of literacy. It also
avoids reducing the pool of otherwise competent labour from
which to select. Similarly, controlling readability reduces the de-
mands created on supervision and monitoring by documents that
are unnecessarily difficult to understand and use (later in this pa-
per, a criterion for this level of difficulty is derived).
The link between well-written procedures and human activity
in nuclear technology was investigated in the 1960s and 1970s by
Swain et al. (1983) and resulted in a checklist. The US Depart-
ment of Energy (DOE) later expanded this into a guide for writing
clear technical procedures (DOE, 1992). In the Netherlands,
Wagenaar (1982) introduced the checklist and pointed out that
two factors – correct time-sequence and the availability of refer-
enced information – are of paramount importance to avoid hu-
man error. Later research on language ergonomics by Nas
(2007) identifies the two main readability elements in a docu-
ment: text, as a composite of words and sentences, and; layout,
consisting of typography,illustrations and other graphics. Bohnen
et al. (2007) developed an elaborate list of nine readability
requirements. In Bohnen’s list, five requirements relate to text
complexity and four to graphical appearance. Current linguistic
advice and practice suggests that changing the appearance or lay-
out of a document has only a limited effect on readability (Sugito,
2004).
Lardner and Fleming (1999) report the relative magnitude of
different socio-technical factors contributing to incidents caused
by human error. Their findings, shown in Table 1, indicate that
‘‘procedures” are a significant factor in many accidents.
Procedures and documentation are the single biggest factor
identified in Table 1. This may be the result of a variety of reasons,
readability among them. Literacy problems are hidden in there
also. Although more evidence is needed to quantify the exact con-
tribution to accidents, the findings of Lindhout and Ale (2009)
strongly suggests that poor readability of procedures in relation
to the literacy levels in a company environment must be deemed
a cause of accidents which, as stated earlier, can be improved
straightforwardly. This further justifies the adoption in companies
of a well-structured and safeguarded approach to ensure sufficient
readability.
Literacy and readability are interrelated. They can be measured
with the same scale. Both the scale and the relation is explored
here. The present study uses the six-point scale defined by the
Common European Framework of Reference for languages (CEFR)
(EU Commission, 2005). The CEFR framework scale treats reading
skills as a subset of language skills. The scale is language indepen-
dent and needs to be operationalized per language. Driessen et al.
(2007) introduced the CEFR in Dutch language areas. The CEFR is
built on three basic levels:
A. The starting user of a language, able to discuss personal
daily matters in the direct personal environment only, and
hence not able to fully participate in society and in a work
environment.
Table 1
Socio technical factors in human error incidents according to Lardner and Fleming
(1999).
Deficient procedures or documentation 43%
Lack of knowledge or training 18
Failure to follow procedure 16
Deficient planning or scheduling 10
Miscommunication 6
Deficient supervision 3
Policy problems 2
Other 2
P. Lindhout et al. / Safety Science xxx (2010) xxx–xxx 3
ARTICLE IN PRESS
Please cite this article in press as: Lindhout, P., et al. Controlled readability of Seveso II company safety documents, the design of a new KPI. Safety Sci.
(2010), doi:10.1016/j.ssci.2010.02.011

B. The independent user, able to take care of things as they are
in normal life, including some writing and travelling, and
able to express well during conversation.
C. The skilled user, able to read, write and understand large
and complex documents and able to use the language flu-
ently and concisely in any social or professional situation.
Each of the three basic levels is split into two sub-levels, 1 and
2, leading to the six-level scale A1 ...C2 for language skills shown
in Fig. 1. The CEFR scale is used for this study because it relates to
social context, to international adult literacy research studies and
to education system milestones.
For the Dutch population the distribution of reading skills rela-
tive to the CEFR framework was investigated by Bohnen et al.,
(2004). This distribution leads to a cumulative percentage scale
of reading ability for the population. Both distribution and cumula-
tive scale are shown in Fig. 1. The reading skill distribution is as-
sumed to be valid for the working population since no more
specific data are available.
Lindhout (in press) proposed the L-scale to measure readability.
This appraisal scale combines textual and graphical document
properties. The textual appraisal is based on a range of sample
texts, calibrated against some 20 text indicators, and use of a
numerical value of a commonly-used indicator formula. Appraisal
of graphical document properties in a document employs a score
based on a simple count of directly visible layout attributes. The
L-scale design is derived from authoritative readability research
findings on ‘‘good readability key elements in a document” at the
CINOP language research institute (Bohnen et al., 2007). The text-
related elements are appraised with an indicator formula: the
Flesch Reading Ease Score, FRES (Flesch, 1948). The numerical val-
ues of the FRES score typically range from below 0, very complex,
to above 100, easily readable. One CEFR level corresponds to
approximately 20 FRES points.
The layout related elements are covered with a 10-attribute
checklist. Each of these attributes may receive up to two points, to-
gether adding up to a maximum score of 20 points, corresponding
to one CEFR level. The L-level is calculated from the sum of FRES
and layout scores. The L-level in turn corresponds via a conversion
table to an effective CEFR readability level.
The L-scale was used to investigate the readability of 43 safety-
related documents currently in use in Seveso II companies.
The results are shown in Fig. 2. The more to the left a document
is plotted, the more easily it can be read. It would appear that
authors of personnel magazines are focussed on readability since
all their documents meet the readability criterion I, they all have
levels A2 and below. It seems common practice to write policy doc-
uments in levels B1 and up. These documents reflect – tentatively –
that both authors and readers would be highly educated staff only.
The front gate safety flyers, tank truck loading procedures and
work instructions show a wide spread of readability levels, indicat-
ing that readability is not a well-controlled document property,
regardless of the criterion used.
Fig. 2 illustrates also that readability differs for different types
of safety-related documents in the ‘‘high risk” companies investi-
gated. The policy documents are all at a high level (on the right
hand side below C1 and C2), the personnel magazines are all at a
low language complexity level (on the left hand side below A1
and A2). This shows that when companies choose to communicate
with their personnel, they are able to provide sufficiently readable
information. In the Netherlands, a safety policy must be agreed by
the works council (in which workers are represented) so these doc-
uments must be readable for the same reader community. None-
theless, companies tend to write those at the higher complexity
levels. Work instructions appear to have a huge variation in lan-
guage complexity level. This could indicate that companies are
not aware of this document property. They may also be unaware
of the risks associated with a poorly-written document. A regula-
tion compliant company cannot knowingly issue safety documents
that a significant part of their personnel cannot understand.
We found no basis for an index of safety relevance per docu-
ment. The company document classification is therefore kept sim-
ple: either safety is implicated or it is not. Evaluation on basis of
practical experience is to show later on whether this is the right
way forward.
A higher reading skill level allows an individual to read a more
complex document. A more complex document is less readable.
Readability, considered as a property of a document, can be ex-
Fig. 1. Language skill levels according to CEFR and frequency distribution of reading
skill level among the adult population in the Netherlands.
Fig. 2. Readability of 43 Seveso II company safety-related documents compared to
every day examples.
4P. Lindhout et al. / Safety Science xxx (2010) xxx–xxx
ARTICLE IN PRESS
Please cite this article in press as: Lindhout, P., et al. Controlled readability of Seveso II company safety documents, the design of a new KPI. Safety Sci.
(2010), doi:10.1016/j.ssci.2010.02.011

pressed with the same CEFR scale. Bohnen et al. (2007) identify
text and layout features important for determination of the read-
ability level on the CEFR scale.
Zakaluk et al., (1988) have validated the relation between read-
ing skill level and readability level of a text. They devise a nomo-
gram to show that with poorer reading skill more texts are
beyond comprehension, starting from the most complicated ones.
They also show that, with exception of the highest skill levels,
there is a part of the text complexity scale out of reach for a part
of the readers. At only the lowest levels of complexity is a percent-
age close to 100% of readers able to understand the text.
A simple approximation of the relationship between reader skill
and document readability is used in this study: the ‘‘readability
equation”. It describes the relation between reader skill level R
and document readability level Dfor three situations:
R>
1

2
<
3
D
In situation 1, the reader’s level of skill is higher than the readability
level of the document. This is fine when it comes to understanding
the contents of the document. There may be slight irritation in the
case of a C1 reader faced with a document on A1 level. It could be
perceived as ‘‘childish”, unnecessarily lengthy or stripped of famil-
iar buzz-words, expressions and humour. The actual challenge is
to find an acceptable language complexity level, neither too high
or too low. Any reader will accept A2 level documents, this is dem-
onstrated by daily newspapers which are generally written in A2 le-
vel language.
Situation 2 is the borderline situation. Here, the reader is being
confronted with a level very close to the limit of his/her ability to
handle mere text. The reader depends more on the help offered by
improved layout, graphics and illustrations. Krashen (1982) estab-
lished that a reader can – with considerable effort – read a docu-
ment one level above his or her skill level. In situation 2, errors
due to misinterpretation and extra time required to understand
the document become probable.
Situation 3 is problematic. The reader cannot fully understand
the document and will rely on existing knowledge and experience
if no further explanation is offered. For a worker in the presence of
danger, this may cause a hazardous situation.
If reading a document at readability level A2, 5% of the adult
population will encounter situation 2, and 95% situation 1. A2
could be considered as the best achievable level of readability for
safety documents intended for a general large company with many
safety documents not uniquely written for selected groups of
highly skilled readers. Government letters with readability level
C1 will put 15% of adults into situation 2, and 85% into situation
3. This is the reason for adoption of a readability criterion by the
government; the citizens must be sufficiently able to understand
government communications. An academic PhD thesis written on
C2 level leaves virtually everybody in despair.
In keeping with the initiative of the Dutch government, the cur-
rent linguistic advice and practice have adopted the B1 level as a
criterion. This ensures 95% of the population to be able to read a
document. This implies however that 5% of the people cannot read
it, and a further 15% will have considerable difficulty grasping it’s
contents. For another 40% of the readership, the text is at the bor-
derline, meaning that the readers are at the limit of their ability.
The remaining 40% can read it without difficulty.
Warehouse employees, truck drivers and maintenance techni-
cians, are identified as risk groups. This is because of their limited
language skills, which are often in the A1 and A2 levels (CINOP,
2005). The use of the B1 criterion for safety-related documents
used by these groups means that a sizeable proportion of individ-
uals will have difficulty comprehending them; clearly not a safe
working practice.
The criterion for readability for safety documents must there-
fore be placed on level A2. If this policy were adopted, 5% would
still have considerable difficulty when reading the documents,
but 95% could read them without difficulties. For most individuals
in that 5%, application of maximum effort in graphical layout to im-
prove the document readability will allow them to comprehend
the document. This takes full advantage of the 1 CEFR level
improvement effect of graphical layout readability improvements.
This would be an effective risk reduction method qualifying as risk
control to ALARP level: as low as reasonably practicable.
A third criterion is conceivable. A separate approach to accom-
modate highly skilled readers would lead to a mixture of readabil-
ity levels in a company, in this case well-controlled on document
level. All documents can easily be read by all respective designated
readers. The average readability level will then be somewhere be-
tween the B1 and B2 levels. Although possible, this approach has
disadvantages since there are flexibility constraints for instance
due to documents not directly suitable for a previously undesig-
nated group of readers. It is conceivable, for example, that the
workforce may change overtime; new users of the old documents
may be less able readers. Also the control of a multi-level readabil-
ity system with a set of subcriteria is more complicated. Last but
not least this would also negatively affect a small group of people
with special problems, e.g. dyslexia since they generally benefit
from low complexity level text.
Nonetheless, all this results in three levels:
Criteria for good readability:
I < A2 criterion for Safety-related documents.
II < B1 criterion for use by the general public.
III B1–B2 subcriteria average for controlled mix of readabil-
ity- and skill levels.
Since the KPI is designed for safety documents only, criterion I is
used to define the KPI performance/non-performance limit. The
three criteria levels I, II and III are shown in Fig. 2 as vertical lines.
The lower horizontal scale indicates the percentage of the Dutch
adult population able to read documents in each of the CEFR levels
A1 and A2, etc., indicated on top. For comparison, the readability
levels for every day documents A, B and C: a free newspaper, a high
school book and a government letter, serve as examples.
3. Objectives and design requirements
The purpose for a readability KPI arises from the desire to re-
duce the risks created by use of safety documents with an inappro-
priate level of readability. These dangers include: not following or
misunderstanding instructions and procedures; lack of under-
standing of equipment, installation or hazards; non-compliance
with safety regulations; wrong responses in an emergency; and
lack of feedback in case of erroneous procedures.
The readability equation R:Dhas two sides: reader’s skill level
(R) and readability level (D). The literacy skills of workers are the
more difficult to change; it takes considerable effort and time.
The results of these efforts are nonetheless important for economic
reasons such as the individual’s employability and to promote so-
cial inclusion.
However, even if all currently illiterate workers were to partic-
ipate in language courses and achieve a command of the Dutch lan-
guage on A2 level, they would still have difficulties with B1
documents and be unable to read any document written at the
B2 level or higher. If nothing were done to the document readabil-
ity levels found in Seveso II companies, improving literacy in this
way would achieve only a slight improvement in the rate of errors
provoked by poor readability. The emphasis must therefore be
P. Lindhout et al. / Safety Science xxx (2010) xxx–xxx 5
ARTICLE IN PRESS
Please cite this article in press as: Lindhout, P., et al. Controlled readability of Seveso II company safety documents, the design of a new KPI. Safety Sci.
(2010), doi:10.1016/j.ssci.2010.02.011

placed on getting sufficiently readable documents. A company pol-
icy aligned to this could be stated as:
‘‘Our company wants to promote safety by creating documents that
are easy to read and use.”
To measure implementation of this policy, a suitable KPI would
be needed. The KPI would need to be designed to fulfil a set of
requirements derived from the poor readability risk to be con-
trolled. For example, the KPI must:
– demonstrate that all safety documents are being controlled on
readability;
– demonstrate that all safety documents are sufficiently readable
against a preset criterion;
– demonstrate that all authors, generating new or changed safety
documents in a company, write at a preset readability level;
– demonstrate that the readability risk control system is safe-
guarded by external auditing;
– be designed and developed reflecting current best practice in
management reporting techniques;
– be practical, effective and easy to implement.
4. Guidelines for KPI development
The KPI design process set out in this paper is informed by ISO
9001:2000 (ISO, 2000), in particular Sections 7.3.1 and 7.3.4 which
deal with design and development. Also section of the standard
4.2.3(e) is relevant, insofar as it requires ‘‘documents to be easy
to read and to recognise”.
The HSE 254 Guide (HSE, 2006) offers a step-by-step process for
the design and development of a KPI. It is an established method-
ology suitable for use when developing any type of KPI. Although
the guide was informative, it did not constitute a complete and
substantive basis for developing a KPI on document readability.
4.1. General shortcomings in the HSE 254 guide methodology for KPI
development
As a first general comment, the completeness and consistency
of the primary causes listed on page 23 is questionable. The list
is indicative rather than exhaustive, and does not appear to be
the product of a systematic classification. For example, high or
low temperature and human error – to which readability is caus-
ally related – are missing as direct causes.
A second general comment is the limited applicability of the
guidance to workprocesses in other than standard operating
modes. Dees et al. (2004) established that shut-down, start-up,
maintenance stops and repair modes are highly correlated with
major accidents in the Netherlands.
The third general comment is related to the non-optimum use
of the ‘‘swiss cheese” model published by Reason (1997). This de-
picts a risk control system (RCS) consisting of a series of barriers
with latent weak spots or ‘‘holes” in them. Danger can evolve only
into an unwanted effect in the situation that all barriers fail ‘‘with
the holes lined up”. This ‘‘common-mode” aspect is not reflected in
the guide in spite of its considerable importance. Current practice
is to look at barrier integrity only, rather than properties such as
redundancy and common-mode failure. There is potential for fur-
ther research on this point. The development of the readability
KPI uses ‘‘control” indicators in order to identify where the barrier
weak spots are. This is a first step in the direction of detection of
common-mode weaknesses in a multiple barrier RCS.
The HSE 254 guide assumes the presence of a ‘‘risk control sys-
tem”, RCS, before the start of the KPI development. In this case the
readability KPI and the RCS that is subject to it, are inseparable. It is
likely that most firms seeking to set-up monitoring arrangements
for readability will also need to develop an RCS as part of the ini-
tiative. The guide does not recognise this contingency and given
the opportunity to connect companies to advice, it would be useful
for future editions of the guide to contain references to suitable re-
sources about RCS development.
Step 1 (arrangements) depends heavily on activities carried out
at later steps. For example, both internal parties and external par-
ties may be involved, but appointing and involving the right people
is only possible after gaining insight in the scope of work. Step 4
(critical elements) requires both an inventory of the elements of
the RCS and – confusingly – at the same time ‘‘lead indicator” parts
need to be defined. A better separation between consecutive steps
is needed.
Remark. The ‘‘worked examples” provided by the HSE guide
contain an example about communication. During the readability
KPI development, this turned out to be of limited help since the
poor readability safety aspect was not addressed.
The HSE guide introduces two types of indicators: leading indi-
cators (systematic checks if activities are done as intended) and
lagging indicators (the intended safety performance is not reached).
The definition of these indicators is found compelling but, as noted
by Hopkins (2009), not entirely consistent. The leading indicators
show activities are happening but do not necessarily measure the
quality of their results. The lagging indicators do not distinguish
between a lack of performance of a single barrier without adverse
effects and a lack of performance of all barriers leading to an un-
wanted event. The development of a KPI may require a third type
of indicator to spot failures of part of the barriers. Besides input
and output, the effectiveness of the RCS needs to be monitored,
in other words the accomplishment of control over the risk itself.
4.2. Solutions from other guidance
A literature search for other development methods resulted in
several sources of information to fill the shortfall between the
HSE guide and what was needed to develop the KPI.
The OECD Guide (2003) describes the development of Safety
Performance Indicators (SPI). This guide distinguishes between
two types of indicator: activities indicators (Is the company busy
with risk reducing activities?) and outcome indicators (Is there
any safety improvement accomplished?). An SPI must be designed
for a specific situation and needs well defined and documented
measurement principles, appraisal scales and criteria for monitor-
ing over the longterm. Numerical values that are simple to gather
are preferable, although qualitative attribute scales are possible.
The readability KPI being developed here allows direct measure-
ment on both the ‘‘activity” and the ‘‘outcome” side. The OECD
indicator solution does not detect integrity of the RCS, however.
Remark. The OECD (2003) Guide mentions on page 19 as an
example of an outcome indicator:... Many, if not most, outcome
indicators can only be measured indirectly, by use of surveys or
through an assessment by an independent observer (for example,
the ‘‘extent to which procedures are understood and applied by
employees”) ...
During the October 8, 2008 Seminar on Process Safety Perfor-
mance Indicators in Bordeaux, France, different development
methods were reviewed (EU-OECD, 2008). Shortly before this sem-
inar the OECD finalised the second issue of their SPI development
guides, now separately documented for public authorities and
industry. The second edition OECD (2008) guide for the industry
also identifies activities- and outcome indicators.
6P. Lindhout et al. / Safety Science xxx (2010) xxx–xxx
ARTICLE IN PRESS
Please cite this article in press as: Lindhout, P., et al. Controlled readability of Seveso II company safety documents, the design of a new KPI. Safety Sci.
(2010), doi:10.1016/j.ssci.2010.02.011

The CCPS Guide, (2007) is of particular relevance to the chemi-
cal industry within which many of the Seveso II companies reside.
This guide presents three types of indicator: leading metrics (status
of preventive activities), lagging metrics (incidents) and near miss/
internal lagging metrics (unsafe conditions, failure of a safety provi-
sion, near misses and small incidents). Hopkins (2009) also distin-
guishes between three types of indicators: measurement of safety
activities,safety activity detected faults and process usage faults.
Concluding, on the basis of the above findings a combination of
the CCPS ‘three indicators’ approach, and the HSE 254 Guide (HSE,
2006) step-by-step approach, were accepted as a practical basis for
developing the readability KPI. Therefore the development of the
readability KPI is based on a three types principle: leading,lagging
and control indicators as shown in Fig. 3.
5. Design of a readability KPI
Based on the requirements and guidelines discussed in the pre-
vious sections, the readability KPI was developed using the steps of
the HSE Guide and features three types of indicators.
The six steps described in the HSE 254 Guide are:
1. Establish the organisational arrangements to implement
indicators.
2. Decide on the scope of the indicators.
3. Identify the risk control systems and decide on the outcomes.
4. Identify critical elements of each risk control system.
5. Establish data collection and reporting system.
6. Review.
The literature search produced a variety of concepts around
which to develop indicators of readability performance. These con-
cepts are reviewed in this section and the suitability of each for use
as an indicator is evaluated. The evaluation focused on the extent
to which each concept supported the objective, was scientifically
well proven and practical in a company situation. Potential indica-
tors were found on both sides of the readability equation, hence
reader skills and the risk control system itself were looked at, as
well as document properties.
5.1. Information sources relating to document content
1. Readability evaluation by external linguistic experts.
High quality business to business commercial services are avail-
able in the Netherlands. There are cost and time consequences to
be considered and the linguistic advice obtained may not appeal
to the technical orientation of the document authors involved.
Measurement on individual documents against the CEFR scale is
done on a routine basis.
2. Text complexity indicator formula use by the authors.
Literature search shows that a range of automated text com-
plexity indicator formulae is available. Out of some 20 types of
indicators, the FRES indicator qualifies best since it reflects text
complexity well over the entire CEFR range. A practical advantage
is that it FRES is built into the widely-used Microsoft ‘‘Word” word
processing application, used by many technical authors’. FRES
deals only with the text component of a document; layout proper-
ties are not measured.
3. Appraisal of layout by the authors.
Many sources of layout advice intended to improve readability
are offered to the general public via internet. These take the form of
listings of do’s and dont’s in a document. No systems for auto-
mated layout appraisal were found in literature. Using the recom-
mendations from various sources, an author can improve
readability by a maximum of one CEFR level, but there is no tool
to actually measure results. Looking at only the layout element is
not sufficient in itself. Two initiatives must be mentioned here.
The first is a classification proposed by Bohnen et al. (2007) listing
nine key document features, important for readability, of which
four refer to layout. The second initiative is a detailed document
appraisal form recently proposed by Breedveld et al. (2008) and
highlighted by the Dutch Safety Association NVVK (2008). This
elaborate appraisal form directs attention beyond document layout
to aspects such as the match with reader’s skill level, words, sen-
tences and text structure. Breedveld’s scheme uses a text complex-
ity indicator formula and it evaluates the use of the document in
practice. Use of the form is explained in a six-page manual. Breed-
veld’s scheme scores document layout using subjective judgement;
in the interests of reliability, this could be done using several scor-
ers rather than just one individual. However the Breedveld scheme
is quite different from classification by Bohnen et al. (2007) and
there is little data to help evaluate the reliability, validity or usabil-
ity of either of these tools.
4. Measure the effective CEFR level of each document.
Lindhout (in press) proposes a method for integrated text + lay-
out appraisal, the L-scale. This scale uses an automated text com-
plexity indicator and a (not yet automated) ten-point score list
for layout to measure the effective CEFR level of a document.
Although this appraisal method is still experimental it requires less
than 10 min to complete per document. (see chapter 6)
5. Report poor readability.
If a ‘‘readability team” would report any documents they find to
be poorly readable following a systematic assessment; this could
indicate breaches in the risk control system before or after inci-
dents occur. This way of signalling problems is similar to existing
behaviour based safety practices. The addition of an extra category
Leading
Lagging
Danger
RCS - Risk control System
Good Readability of Safety Documents
Effect
Measurements on
activities in the RCS
Measurements on
weaknesses in the RCS
Measurements on
consequences of
failure of the RCS
Control
Fig. 3. Readability KPI design principle with leading, lagging and control indicators.
P. Lindhout et al. / Safety Science xxx (2010) xxx–xxx 7
ARTICLE IN PRESS
Please cite this article in press as: Lindhout, P., et al. Controlled readability of Seveso II company safety documents, the design of a new KPI. Safety Sci.
(2010), doi:10.1016/j.ssci.2010.02.011

in existing reporting and registration systems would suffice. The
result however is at best a report of a suspect document; verifica-
tion that poor readability was indeed the problem would be
needed to avoid false-positives being registered in the KPI.
6. Readability team.
A readability team would consist of group of people, drawn
from diverse backgrounds, that review document readability on a
regular basis. This is close to current practice in ‘‘document review
boards” and ‘‘internal process audits”. In larger companies, this
could be a permanent or recurring activity. The work of this team
would need to be organised, documented and prepared by training
to ensure lasting quality.
7. Readability audit.
Readability auditing would involve random sampling of docu-
ments for examination by internal and, even more desirably, exter-
nal auditors. The European Council Seveso II Directive 96/82 (EC,
1997) and the ISO 9001 standard chapter 4.2.3. sub E provide the
formal background for such audits. Dutch Seveso II companies
need to conform to technical standard NTA 8620 para 4.4.2. A read-
ability audit could be introduced into existing regulatory audit cy-
cles after adaptation of audit tools. Adjustments to existing audit
techniques and documentation are required. The involvement of
external audits would improve this further by providing a safe-
guard for the quality of internal readability audits and ensuring
the audit techniques remain state of the art.
5.2. Information sources relating to reader skills
8. Spotting illiteracy signals.
Although poor literacy sometimes goes undetected in the work-
force, it is possible to recognise it (Bersee et al., 2005). The human
resources department has a good opportunity during recruitment
activities. Special attention must be given to this because those cit-
izens of the mother-tongue country who have problems with read-
ing and writing tend to hide them, while foreign workers easily
admit them (Breg, 2004). However, foreign workers could be illit-
erate in their own language, a problem often ignored and clearly
not addressed by providing translations of company procedures.
Spotting this is possible and useful since it contributes to solving
illiteracy problems, but it is considered to be too unreliable as an
input for a KPI.
9. Spotting activities which improve literacy.
Workers with literacy problems, at least those who manage to
overcome the taboo on illiteracy, may take language courses and
reading and writing classes (Bohnen et al., 2004). These activities
usually originate from the human resources department and are
aimed at employability. There is no obvious link to safety manage-
ment since safety documents may not be used by some of these
people. Signalling such activities in a KPI would not be a reliable
indicator of poor readability control since it part of it may concern
neither the documents nor the readers. This renders the concept
unsuitable for use in a KPI.
10. Document testing.
Testing of the interaction between a set of documents and a
group of readers is conceivable. The present authors expect that
costs and time impact would prohibit the daily use of this
approach in companies; but this has yet to be put to the test. The
method may be useful in scientific research and validation testing.
11. Watching the language risk group.
External contractors from abroad and temporary workers are
among those that qualify as potential risk group. At the front gate,
at the issue of the work permit and at the job location there is
opportunity to recognise this group. Records would have to be kept
of this on a routine basis. Although there are incidents caused by
lack of attention on these groups, any measures or constraints
may be controversial since this touches on the free movement of
labour across Europe. No useful input for a KPI is expected here.
12. Reading skill appraisal by external experts.
This service is available on a commercial basis. It is possible to
access this via the personnel department of any company in the
Netherlands. It is not a service that is generally available to individ-
ual authors or to audit teams.
13. Deduce from existing reporting systems.
Provided that sound analysis of all incidents takes place in a
company, the existing records already contain useful data on inci-
dents and their cause. Problems revealed by investigations, relating
to issues such as communication, procedures, misunderstandings
and so on, may offer a rough indication of readability problems.
This depends on investigation methods, depth and causal factor
classification being used in a company. These would all need to
be adapted to be able to explicitly record ‘‘readability problems”
as a causal factor.
5.3. Information sources relating to the risk control system
14. Check on registered safety documents.
A decision to systematically register documents as ‘‘safety doc-
ument” opens up the possibility to ‘‘control” specific properties not
controlled by other systems. Any author writing or changing such a
clearly marked document is aware of the special requirements im-
posed upon this category. A company may even want to issue a
special procedure on this and authorise specially trained staff only.
5.4. Selection of readability concepts for use in a KPI
The above evaluation identifies concepts 1 and 4 as most sup-
portive to the objective. Concepts 2 and 3 are interesting but only
in combination with other concepts. The concepts 5, 6, 7, 13 and 14
appear worth considering further. The other concepts 8, 9, 10, 11
and 12 have clear disadvantages making them unattractive for
use in a KPI so they are not pursued further. The remaining nine
design concepts are compared in a trade off study summarised in
Table 2.
The risk control system is defined as: ‘‘Good readability of
safety-related documents”. The leading indicators are based on
performance of the RCS: measurement on readability related activ-
ities. Control indicators make the quality of the results of these
activities visible and amenable to review by management. This
can be done via readers, reporting observations about readability,
by random checks performed by internal teams or by independent
internal or external audits.
The lagging indicators are based on non-performance of the RCS,
they are a count of unwanted events caused by poor readability.
8P. Lindhout et al. / Safety Science xxx (2010) xxx–xxx
ARTICLE IN PRESS
Please cite this article in press as: Lindhout, P., et al. Controlled readability of Seveso II company safety documents, the design of a new KPI. Safety Sci.
(2010), doi:10.1016/j.ssci.2010.02.011

These can be complaints, behaviour based safety observations,
near misses or incidents.
The allocation of concepts 1, 2, 3, 4 and 14 to ‘‘leading” is logical
on this basis. In this KPI development ‘‘control” indicators are also
needed. Concepts 5, 6 and 7 must be regarded as ‘‘control” indica-
tors. Concept 13, which concerns data from incident records, is
clearly a ‘‘lagging” indicator.
To compare the concepts on the criteria ‘‘fits objective”, ‘‘is well
proven in daily practice”, ‘‘has acceptable development time” and
‘‘cost is affordable” a simple scoring system was used; the results
are shown in Table 2.
Concept 3, separate layout appraisal, has the lowest score count,
indicating it cannot easily be introduced for predominantly eco-
nomic reasons.
The use of an integrated text and layout document appraisal via
concept 4 covers layout appraisal more affordably. Furthermore
the separate appraisal of text complexity is of much more value,
worth up to six CEFR levels, than a separate layout appraisal, worth
up to one CEFR level maximum. Concept 3 is therefore abandoned.
An indicator based on concept 1 involves constant involvement
of an external advisor since every document needs to be checked
for every new version. This raises cost, logistics and confidentiality
issues for companies. Since random sampling checks of readability
by external audits, as suggested in concept 7, can ensure the qual-
ity of performance of the risk control system it is more practical to
use concept 7 and abandon concept 1.
Ideally, the KPI reflects in its indicator data that the risk control
system covers 100% of the safety documents. Not all indicator con-
cepts perform the same on this point. Concepts 5, 6, 7 and 13 can
not demonstrate 100% coverage for various reasons. In Table 2 this
is shown in the column ‘‘Document coverage”. Concepts 5 and 13
have a coverage percentage (near zero) because they depend on
incidental reports on poor readability. The readability audit envis-
aged in concept 7 would normally be based on a sampling method
that is likely to account for just a small percentage of the full file of
documents held by the company. For example in a company having
500 documents, five external readability audits on five documents
each time would result in 5% coverage.
In a company with a smaller volume of documentation of say
fifty documents the coverage will be 50%. Internal checks by a
readability team can result in higher coverage due to the choice
of the company to put in more-or-less effort. In principle the cov-
erage remains less than 100% since the indicator design is not
aimed to check every document for every version. This implies that
any of these concepts can only be used together with a concept
clearly demonstrating and ensuring 100% coverage by the risk con-
trol system. In the trade off table all leading indicator concepts
indicate 100% coverage.
The KPI is best developed with indicators that require little
development time and implementation effort. A simple ranking
was performed to show the differences between the concepts. This
is indicated in the right column ‘‘KPI” in Table 2 with an increasing
number of asterisks when the concept performs better on speed of
introduction.
5.5. Definition of a readability KPI
Having reviewed each concept for suitability, the integrated KPI
on readability can now be finalised. Leading indicator concepts 2, 3
and 4, control indicator concepts 5, 6 and 7 and lagging indicator
concept 13, are the building blocks for the KPI. The KPI needs to in-
clude at least one indicator of each kind. The functions of indicators
2 and 4 overlap to a large extent. The simplest form the KPI can
take is a combination of concepts 2, 7 and 13. This relies on the text
complexity formula only and ignores any readability improvement
by layout. The best KPI would consist of concepts 4, 5, 6, 7, 13 and
14. The most complete KPI design simply includes all indicators,
ignoring the partly overlapping appraisals in concepts 2 and 4.
Fig. 4 presents this complete KPI design. Table 3 shows the result-
ing indicators, their value range and acceptance criteria.
5.6. Data collection methods
The diversity in the group of authors in a company, the unique-
ness of each document they create and the diversity of readers
makes testing on a random sample of safety-related documents
unsuitable as a way to control the risk. For example, if one tried
to reflect each of these sources of variation in a sample of docu-
ments, it is highly likely that resulting sample size would be a size-
able proportion of the whole file of documents. Also, because these
sources of variance have more-or-less unknown properties, a great
deal of work would need to be done to inform a suitable sampling
scheme. Therefore, we advocate a 100% sample as the simplest and
most prudent strategy. Hence, the suggested company objective –
to make all documents easy to read – leads to 100% of the safety
documents having to be readability-controlled. So, every document
Table 2
Safety documents readability indicators design concept trade off.
No. Concept description Fits
objective
well
proven
Develop
time
Cost Econ.
score
Involved
parties
Complications Implementation
effort
Document
coverage
Indicator
type
KPI
1 Readability evaluation by
linguistic experts
Good Exists Zero High External Yes (A) None 100% Leading –
11103
2 Use text complexity
formula FRES
Partly Exists Zero None Authors No None 100% Leading 
½1113½
3 Appraisal of layout by
authors
Partly Pilot Long High Authors Yes (B) Medium 100% Leading –
½000½
4 Use L-scale measure CEFR
level
Good Pilot Zero None Authors No Small 100% Leading 
10113
5 Report poor readability Medium Exists Short None Readers No Small 0% Control 
01½12½
6 Readability team check Medium Partly Short Low Readers
and authors
Yes (B) Small <100% Control 
0½½½1½
7 Readability audit Medium Partly Short High External No Small <50% Control 
0½½01
13 Deduce from incident
records
Good Exists Short Low HSE dept No Medium 0% Lagging 
11½½3
14 Check doc registry and
marking
Good Exists Short Low HSE dept No None 100% Leading 
11½½3
Notes: A – legacy methods and B – coaching/training.
P. Lindhout et al. / Safety Science xxx (2010) xxx–xxx 9
ARTICLE IN PRESS
Please cite this article in press as: Lindhout, P., et al. Controlled readability of Seveso II company safety documents, the design of a new KPI. Safety Sci.
(2010), doi:10.1016/j.ssci.2010.02.011

and every revision must be appraised and corrected if found not to
comply to the criteria. Monthly reporting on this seems to imply a
lot of work. Since a company may use many safety-related docu-
ments it is necessary to try to find a way to minimise the effort.
This is possible when using a registration of readability-controlled
documents. Checking on all documents in each reporting period
is not needed when a record is kept of all documents, their checks
and their readability. This reduces the monitoring effort to a min-
imum since the output of the KPI is only affected when the record
is updated for a document that was changed or for a new
document.
A random sample of documents is useful in an external audit on
readability. It serves as a check on the presence of risk control
activities and their correct execution. The result of such an audit
has clear limitations however. A flawless result on the sampled
documents does not guarantee sufficient readability in other, not
audited, documents. An erroneous audit result on the other hand
makes all other documents outside the randomly selected sample
documents suspect.
5.7. Sampling and uncertainties related to methods of measurement
In order to accommodate the effects of variability in the apprai-
sal of individual documents, KPI measurements need to be based
on a suitable sample of safety documents. It is suggested that, in
practice, the sampling strategy needs to be set out in a suitable
procedure and supported with guidance. The scope of these proce-
dural documents should also include approaches designed to max-
imise the reliability of measurements, document appraisal using
the FRES and/or the L-scale.
Lindhout (in press) has analysed measurement uncertainties
associated with the L-scale and provides guidelines for sampling
within a document. A document may show variations in readabil-
ity between sections of the text. The text sample size needs to be at
least 75 words, and preferably more than 100, to minimise FRES er-
rors. Experience data shows that a 10 sentences sample would nor-
mally be sufficient.
For smaller documents, specifically those of one page or less, it
is suggested that the whole document needs to be assessed as a
single sample with the FRES indicator. Larger documents require
several samples and an averaged value approach. For most docu-
ments three samples taken from the beginning, middle and end
parts is accepted to be adequate in linguistic practice. Documents
containing sections written in different styles require sufficient
samples to indicate results for each style. Turning to the measure-
ment of layout, to be meaningful, the attributes on a specific sam-
ple page should be considered in conjunction with the preceding
page and the following page. Lindhout (Ibid.) found the L-scale
document appraisal to be accurate to within 1 CEFR level, there-
fore, through careful control of the measurement process, this or-
der of reliability can be assured.
5.8. Reporting metrics
Step 5 of the HSE’s guidance on performance measurement
deals with establishing the data collection and reporting system
(HSE, 2006, p. 18). There, the authors emphasise the need for
reporting data in a form that is, ‘‘as simple as possible”. Although
context is likely to be a decisive factor in deciding what format
to use and how to achieve simplicity. The ratios described below
are designed to be quite general in their applicability, but each
company needs to tailor their approach to fit the preferences of
the users of the KPI.
Readability performance can be reported, whether continuously
via intranet ‘dashboards’ or at intervals (e.g. monthly reports)
using the following:
Leading
Lagging
Danger
RCS - Risk control System
Good Readability of Safety Documents
Effect
LE1 - Marking of Safety documents
LE2 - Text complexity FRES > 50
LE3 - L-scale measurement <CEFRA2
CO1-External Readability audits
CO2-Internal Readability team checks
CO3-Poor readability reports
LA1 - Readability related
incidents, near misses and
Unsafe situations
Control
Fig. 4. Design description of a safety documents readability KPI.
Table 3
Readability KPI indicators value ranges and acceptability criteria.
Indicator Code Description Value range Non performing document count Based on concept no.
Acceptable Not Acceptable
Leading LE1 Marking of safety documents Yes No 1 per not marked document 14
LE2A Text complexity FRES check Yes No 1 per not checked document 2
LE2B Text complexity FRES >50 <50 1 per too low FRES document 2
LE3A L-scale measurement Yes No 1 per not measured document 4
LE3B L-scale effective CEFR level CEFR A1, A2 >CEFR B1 1 per too high CEFR document 4
Control CO1 External readability audits No deviation Deviation 1 per deviating document 7
CO2 Internal readability team checks No deviation Deviation 1 per deviating document 6
CO3 Poor readability reports No reports >0 Reports 1 per reported document 5
Lagging LA1 Readability related reports on incidents,
near misses and unsafe situations
No reports >0 Reports 1 per implicated document 13
10 P. Lindhout et al. / Safety Science xxx (2010) xxx–xxx
ARTICLE IN PRESS
Please cite this article in press as: Lindhout, P., et al. Controlled readability of Seveso II company safety documents, the design of a new KPI. Safety Sci.
(2010), doi:10.1016/j.ssci.2010.02.011

(a) ‘‘the percentage of sufficiently readable safety documents”.
This percentage is calculated from simple counts of non per-
forming documents found after all safety documents have
been identified and their quantity is known. The formula
to calculate this is:
KPI ½%readable safety documents
¼No of identified safety documents Non performing document count
No of identified safety documents
100
(b) The KPI can also be presented as three separate indicators –
Leading (LE), Control (CO) and Lagging (LA) – to facilitate
corrective action decision making. This requires use of sepa-
rate formulae:
ðiÞKPI LE ½%readable safety documents
¼No of identified safety documents LE count
No of identified safety documents 100
ðiiÞKPI CO ½%readable safety documents
¼No of identified safety documents CO count
No of identified safety documents 100
ðiiiÞKPI LA ½%readable safety documents
¼No of identified safety documents LA count
No of identified safety documents 100
6. Implementation-, transition- and monitoring phases
A company, once having set the goal of readability to be con-
trolled and monitored with a KPI, will need to organize the imple-
mentation. To ensure adequate planning and allocation of
resources, it is important to consider in the design phase of the
KPI project the strategy for implementation. At present, the
authors lack empirical data on which to base advice. However,
based on limited data, they have developed a mathematical model
to allow estimation of the effort and duration of the implementa-
tion phase. The implementation approach is described and then
company- and situation-dependent parameters are used to estab-
lish a formula for the effort (E) and the time duration (T).
To ensure success, the implementation phase activity needs
project status and appointing a ‘‘readability champ” is a prerequi-
site. An outline of the project steps is given here. First, an inventory
of all safety-related documents in the company is made. All of
them are marked as such and registered. Then a group of as many
as possible authors (the R-team) is trained how to conduct read-
ability appraisals on documents using the selected appraisal meth-
od. This can be the L-scale, an on-line indicator or – in principle –
any other readability appraisal method. If the L-scale is selected,
a practice training of 2–3 h will suffice.
Then all documents are appraised. The resulting readability
measurements are entered in the registration system. The first
KPI values can then be calculated. Taking the situation found in
Seveso II companies (Lindhout, in press) as an example, it may
be expected that approximately 50% of the documents assessed
will have poor readability. The next step is to improve their
readability. All authors need to be made aware of the principles
of writing plain language and effective use of graphical means. This
will require training and use of condensed guidance information
freely available from the internet or alternatively on commercial
basis. The authors involved in the first appraisal already know
how to measure the effect of their efforts to improve readability.
Both for reasons of risk reduction and to maintain momentum
in the readability project, the transition from an uncontrolled risk
situation to a functioning risk control system on safety document
readability needs to be completed in the shortest time possible,
hence with as many capable and trained authors possible. For this
reason it is advocated to deploy as many capable and trained
authors possible as part of the readability implementation strat-
egy: the impact of resources is shown in Fig. 5.
Starting when the document inventory is ready to be registered
the effort Eand duration Tof the implementation work can be esti-
mated with the formulae:
E¼N
d
½t
r
þt
a
þ
W
t
c
N
p
fEffort in working hoursg
T¼T
0
þT
1
þT
1
þ½1F
E
a
N
A
fDuration time in 8 h working daysg
Eimplementation effort in hours
Tduration of the implementation project in working days,
N
d
number of documents
N
A
number of authors in the R-team
T
0
duration of the R-team training in days (indicative: 0.5 day
for L-scale, 1 day for plain language writing)
T
1
duration of the external audit in days (indicative: 1 day, once
during the implementation project)
t
r
time to register a safety document in hours (indicative: 0.08,
signifying 5 min per document)
t
a
time to appraise the readability of a document in hours
(indicative: 0.17, signifying 10 min per document)
t
c
time to correct the readability of a page in hours (indicative:
1 h per page)
N
p
number of pages per document (indicative: 5 pages per
document)
Fsimilarity factor between documents (indicative: 0.05 is a
copy from other documents)
a
fraction of authors time allocated to the project (indicative:
0.04, signifying 1 day per month for 10 months/year)
W
fraction of documents with poor readability (indicative: 0.50,
signifying 50% as shown in Fig. 2)
The empirical data used here, is taken from appraisal of 53 doc-
uments (Lindhout, in press). A graph showing the required time T
for a range of document quantities Nd and different numbers of
authors NA, based on these formulae and the indicative values, is
presented in Fig. 5.
Example. The implementation time of the readability risk con-
trolsystem in a company with 100 safety-related documents can be
estimated as follows. As a starting ‘‘boundary” condition prior to
the introduction the readability is assumed to be distributed as
shown in Fig. 2. So about half of the 100 documents are not
sufficiently readable. The implementation is given project status
and an internal ‘‘R-team” consisting of five key document authors
is trained for one and a half day. An external audit is arranged
approximately halfway the implementation to verify whether the
project is still in its tracks. This takes a day. Each of the five authors
improve the readability of the 50 inadequate documents at a rate
of one document per month. With their combined capacity,
completion of the improvement would take them some 150
working days. During the project the result of the combined
efforts on improving readability can be shown via the KPI,
increasing from 0% to its ‘‘cruising altitude” of 100%. When
tempory set-backs by holidays, new documents and internal audits
P. Lindhout et al. / Safety Science xxx (2010) xxx–xxx 11
ARTICLE IN PRESS
Please cite this article in press as: Lindhout, P., et al. Controlled readability of Seveso II company safety documents, the design of a new KPI. Safety Sci.
(2010), doi:10.1016/j.ssci.2010.02.011

are taken into account. The estimated project duration is around 10
months.
Once companies start to measure and report readability KPI fig-
ures, the risk awareness and general continuous improvement
activities will help to ensure that safety documents will stay
readable.
External audits are recommended none the less. Special atten-
tion to the continued performance by each author should be given
since writing plain language does not come naturally to many
authors.
7. Conclusions
High risk Seveso II companies in the Netherlands use safety-re-
lated documents that are too difficult to read for some 50% of their
users. Poor readability can cause human error and so compromise
safety. This risk is not controlled in the majority of Seveso II com-
panies. A risk control system, ensuring readability of safety-related
documents, is therefore proposed.
Current linguistic advice appears to reach the authors of safety
documents only to a very limited extent, alternative solutions are
needed. It is relatively straightforward for any author to measure
readability of a document in a meaningful way. Awareness and a
brief training will do. A regular check on consistent appraisal per-
formance is possible by external auditing.
A reporting structure is found to be simple to develop since it
aligns well with the general internal reporting practice that already
exists in Seveso II companies. The KPI indicates the level of control
of safety-related documents readability. The generic KPI design set
out in this paper can be adapted for the specific company situation.
A mathematical model simulation of the introduction of such a
KPI in a typical high-risk company environment results in a transi-
tion period of less – possibly much less – than a year from an
uncontrolled situation, currently found in high risk companies, to
an up-and-running risk control system for readability of safety
documents. It is clearly possible and feasible to make readability
a controlled property of all safety documents. Hence, the feasibility
and the fitness for purpose of the presented indicator concepts for
use in a readability KPI are confirmed.
Further work is recommended in the form of a case study in a
high-risk company to validate the mathematical model simulation
of the implementation phase and to build up practical experience
with managing the KPI in the monitoring phase.
Automated document appraisal is possible when using only the
FRES text complexity indicator as a part of the KPI. In the frequently
used text processing software programme ‘‘Word” this indicator is
often built-in and can be accessed on PC’s by pressing the F7 key.
Also on-line text indicator calculation of an English sample text is
available free and easily accessible, e.g. addedbytes.com.
References
Arbeidsinspectie, 2009. Incidenten rapportage 1 januari-31 december 2007, versie 1
oktober 2008, publicatie Maart 2009.
Bersee, T., de Boer, D., Defesche, P., 2005. Analfabetisme, leidraad voor
doorverwijzers, CINOP, September, ‘s Hertogenbosch. <www.alfabetisering.nl>.
Bohnen, E., Ceulemans, C., van de Guchte, C., Kurvers, J., van Tendeloo, T., 2004.
Laaggeletterd in de Lage Landen, Hoge prioriteit voor Beleid, Nederlandse
Taalunie, 2004, Den Haag. ISBN 90-70593-05-X.
Bohnen, E. et al., 2007. Raamwerk Nederlands, Nederlands in (v)mbo-opleiding,
beroep en maatschappij, 1 oktober 2007, CINOP ‘s-Hertogenbosch, Libertas.
Breedveld, I. et al., 2008. Werken met buitenlandse/anderstalige arbeidskrachten.
Hoe communiceren ?, Universiteit Antwerpen Management School, PO
veiligheidskunde 2006/7 gr 3, pp. 49–56.
Breg, T.A., van Gorp, D.M., van Kooten, J.M., 2004. Kenniseconomie en
laaggeletterdheid, Nyenrode Institute for Competition.
CCPS, 2007. Process Safety Leading and Lagging Metrics, Centre for Chemical
Process Safety, third ed. Initial Release, December 20, 2007, pp. 2, 31–34.
<www.aiche.org/ccps>.
CINOP, 2005. Analfabetisme, leidraad voor doorverwijzers.
COT/DHV, 2004. Trend of Incident, oktober, 2004, pp. 49, 62.
Council Directive 92/58/EC, Minimumvoorschriften voor V&G signalering op het
werk, considerans.
Dees, A., Gils, W. van, Verlinde, A., 2004. Onderzoeksrapport Onderhouds – en
inspectie systemen bij BRZO bedrijven, Arbeidsinspectie MHC, january 2004.
DOE (US Department of Energy), 1992. DOE Standard ‘‘Writer’s Guide for Technical
Procedures”, DOE-STD-1029-92, December 1992. <http://tis.eh.doe.gov/
techstds/standard/std1029/s1029cn.pdf>.
Driessen, M. et al., 2007. Referentiedocument,Talen in de kwalificatieprofielen, 28
februari 2007, CINOP,’s-Hertogenbosch, herziene versie 2.0.
EC, 1997. Council Directive 96/82/EC, ‘‘Seveso II Directive”, on the Control of
Major Accident Hazards Involving Dangerous Substances, Official Journal No.
L010 of 14 January 1997, considerans, art. 5 sub 1, Annex III sub c–i, Annex
IV sub 1–f.
European Commission, 2005. The European Indicator of Language Competence, 1
August 2005, COM (2005) 356 Final. <europa.eu/scadplus/leg/en/cha/
c11083.htm>.
EP, 2001. European Parliament, workdoc. DT/439432NL, 25 June 2001,
C.Employment and Social Affairs.
Flesch, R., 1948. A new readability yardstick. Journal of Applied Psychology 32, 221–
233. <wikipedia.org/wiki/Flesch-Kincaid_Readability_Test>.
Hopkins, J., 2009. Thinking about process safety indicators. Safety Science 47 (4),
460–465.
Houtkoop, W., 1999. IALS Survey – Basisvaardigheden in Nederland.
HSE (Health and Safety Executive), 2006. Developing Process Safety Indicators: a
Step-by-Step Guide for Chemical and Major Hazard Sites, HSE 254, Crown, UK,
(ISBN 0-71766180-6).
ISO, 2000. NEN-EN-ISO 9001 (nl) Kwaliteitsmanagementsystemen – Eisen (ISO-
9001:2000), december 2000, Nederlands Normalisatie Instituut, Delft, ICS
03.120.10.
Krashen, S., 1982. Principles and Practice in Second Language Acquisition. Pergamon
Press, Oxford.
Lardner, R., Fleming, M., 1999. To err is human ...The Chemical Engineer, October 7,
Keilcentre. pp. 18–20.
Lindhout, P., in press. Taalproblemen bij BRZO – en ARIE bedrijven, een onderschat
gevaar ? PhD thesis, December 20, 2009 Delft University, The Netherlands,
(ISBN 978-90-5638-222-3).
Lindhout, P., Ale, B.J.M., 2009. Language issues, an underestimated danger in major
hazard control. Journal of Hazardous Materials. doi:10.1016/
j.hazmat.2009.07.002.
Nas, G., 2007. Taalergonomie: leesbare teksten, Instituut Nederlands, Universiteit
utrecht, 2007. <www.nasonline.xs4all.nl/taalergo>.
N
d
10 50 100 200 500
N
A
T
Documents
Authors Implementation time
2
5
10
20
50
5
10
20
50
100
200
days
Fig. 5. Time Trequired for implementation of the readability KPI in working days
for different numbers of documents Nd and number of involved authors NA.
12 P. Lindhout et al. / Safety Science xxx (2010) xxx–xxx
ARTICLE IN PRESS
Please cite this article in press as: Lindhout, P., et al. Controlled readability of Seveso II company safety documents, the design of a new KPI. Safety Sci.
(2010), doi:10.1016/j.ssci.2010.02.011

NIST (National Institute of Standards and Technology), 2008. US Department of
Commerce, 2008 Baldrige National Quality Program, pp. 53.
<www.baldrige.nist.gov>.
NSR (Nationaal Strategisch Rapport), 2006–2008. Sociale bescherming en Insluiting
Nederland 2006–2008 in het kader van de Lissabon strategie, pp. 15, 53
(Chapter 2.4.1).
NTA 8620 para 4.4.2 Requires Companies to Consider Adequacy of Reading and
Writing Skills for a Task.
NVVK, 2008. (Red) Met handen en voeten, Communiceren met buitenlandse of
anderstalige arbeidskrachten, NVVK info, Februari 2008, pp. 42–43.
OECD, 2003. Guidance for Industry, Public Authorities and Communities for
Developing SPI Programmes Related to Chemical Accident Prevention,
Preparedness and Response, 2003, OECD Environment, Health and Safety
Publications, Series on Chemical Accidents No. 11 (Interim version), pp. 8, 19,
179–188.
OECD, 2008. Guidance on Developing Safety Performance Indicators, Related to
Chemical Accident Prevention, Preparedness and Response, for Industry, second
ed. OECD Environment, Health and Safety Publications, Paris, series on Chemical
Accidents No. 19, pp. 21, 24.
OECD-EU, 2008. Seminar on Safety Performance Indicators, 8 October 2008,
Bordeaux, France.
OHSAS 18001 (nl), 2007. Arbomanagement systemen – Eisen, November 2007,
Nederlands Normalisatie Instituut, Delft, ICS 03.100.01; 13.100.
Reason, J., 1997. Managing the risks of organisational accidents. Aldershot. Ashgate.
pagina 142–146. ISBN 1-84014-105-0.
Schein, E.H., 1996. Three Cultures of Management: The Key to
Organisational Learning. Sloan Management Review; 1996; 38, 1; ABI/INFORM
Global, pp. 9–20.
Shell, 2002. Shell Exploration & Production, Understanding your HSE culture, Hearts
& Minds Programme. <www.energyinst.org.uk/heartsandminds>.
Stichting Lezen en Schrijven, 2004. <http://www.lezenenschrijven.nl/en/illiteracy/
scope-of-problem/>.
Sugito, A., 2004. Teksten op een goudschaal Een studie naar het vereiste NT2-niveau
voor het lezen van overheidsbrochures. Lienden, BureauTaal.
Swain, A.D., Guttmann, H.E., 1983. Handbook of Human Reliability Analysis with
Emphasis on Nuclear Power Plant Applications (NUREG/CR-1278). Washington,
DC: USNRC.
Wagenaar, 1982. Checklist voor de evaluatie van geschreven procedures, voorlopige
versie okt. 1982 Instituut voor zintuigfysiologie TNO, Soesterberg.
Zakaluk, B.L., Samuels, S.J. (Eds.), 1988. Readability: it’s Past Present & Future
International Reading Association. Newark, Delaware.
P. Lindhout et al. / Safety Science xxx (2010) xxx–xxx 13
ARTICLE IN PRESS
Please cite this article in press as: Lindhout, P., et al. Controlled readability of Seveso II company safety documents, the design of a new KPI. Safety Sci.
(2010), doi:10.1016/j.ssci.2010.02.011