PresentationPDF Available

Towards a Validated Readability Index for Dutch: Fact(or)s and Figures



Towards a Validated Readability
Index for Dutch:
Fact(or)s and Figures
Suzanne Kleijn (
Henk Pander Maat
Ted Sanders
Utrecht Institute of Linguistics OTS Utrecht University
NWO Begrijpelijke Taal
Readability Index for Dutch (LIN)
LIN: LeesbaarheidsIndex voor het Nederlands
Project partners: Utrecht University, Radboud
University, CITO and Nederlandse Taalunie
Goal: to build a new and improved readability formula
automated (online tool)
based on real comprehension and processing data
provides an interpretable readability level prediction of a
text (i.e., ‘Your text is suited for readers of level X’)
Readability research
Using objective, quantitative measures to predict the
difficulty level of text (e.g., word length)
Two different areas of application:
1. Readability prediction: assessing whether a text is
appropriate for a certain target group.
2. Readability improvement: diagnosing (potential)
problems for a certain target group for improvement
Some issues in traditional readability
Predictors are not causally relevant to comprehension
Predominately use expert judgments to index texts
Content (message) and style (manner) are
• Higher-level text features like coherence are ignored
• Reader-text interactions are ignored
Effects on on-line processing are ignored
Our approach to readability
‘Causally inspired’ predictors (incl. high-level text
Real comprehension and processing data of target
Multiple text versions to separate effects of content
and style
Regard for reader-text interactions by including
readers with different skills
Steps to building the index
Step 1: Build a tool to automatically extract features
from texts (T-Scan)
Step 2: Relate these features to comprehension and
processing data
Step 3: Analyze data and build readability index
• T-Scan is a tool which automatically extracts 400 text
features from Dutch text.
It currently provides features describing lexical
complexity, sentence complexity, referential and
relational coherence, concreteness, person-oriented
writing and word prediction.
During the break: T-Scan demo by Henk Pander Maat
From T-Scan to data collection
• T-Scan only tells us the value of features and not its
relation to readability.
Empirical data
Comprehension cloze tests
On-line processing eye-tracking
Data collection
Cloze study:
2850 Dutch 8th-10th grade students
Enrolled in different levels of Dutch secondary education
(vmbo-b/k/g; havo; vwo)
60 texts
2 text versions to separate effects of content and style
4 cloze texts per person
30 40 items per cloze text
Cloze fragment
Data collection (2)
Eye-tracking study:
181 Dutch 9th grade students
Enrolled in different levels of Dutch
secondary education
(vmbo-k; havo; vwo)
8 texts taken from the cloze study
Multiple choice questions after each
Screen presentation
Separating content from style
All 60 texts were manipulated to create 2 text
20 texts on lexical complexity
20 texts on syntactic complexity
20 texts on relational complexity
Our lexical manipulation
Text were manipulated to create a lexically easy and a
lexically difficult version.
20% of content words were replaced by a more frequent
or less frequent synonym.
Manipulated words in ‘easy’ text version are on average
14 times more frequent than in the ‘difficult’ text version
(using Subtlex NL)
Natural language: no stilted or archaic language
Text content was left intact
While minimizing (systematic) confounds:
Content, word length, syntactic structure, argument
overlap and type-token ratio were kept constant
between text versions.
Lexical manipulation
Higher frequency text version
Lower frequency text version
1. Rabies is een infectieziekte die de hersenen
“Rabies is an infectious disease that damages/impairs the
2. Iemand heeft genoeg geld/voldoende middelen om er een
tijdje tussenuit te kunnen.
“Someone has enough money/sufficient means to take a
break for a while.”
Combined results of cloze and eye
Increasing word frequency has a positive effect on
comprehension and on-line processing:
Higher cloze scores
Higher multiple choice score
Shorter reading times
Clear main effects of Educational level and Grade in
the expected directions
Conclusion lexical manipulation
Word frequency affects readability, but the effects are
relatively small.
Word frequency is only one measure of lexical
complexity a combination of features may prove far
more successful
Where are we now?
We have T-Scan to automatically extract text
We have empirical data which show which features
influence comprehension and text processing.
We know how they affect different level readers.
All that is left is to actually build the index (LIN)
ResearchGate has not been able to resolve any citations for this publication.
ResearchGate has not been able to resolve any references for this publication.