ArticlePDF Available

Checking an integrated model of web accessibility and usability evaluatin for disabled people

Authors:
  • University of Perugia. Italy

Abstract and Figures

A combined objective-oriented and subjective-oriented method for evaluating accessibility and usability of web pages for students with disability was tested. The objective-oriented approach is devoted to verifying the conformity of interfaces to standard rules stated by national and international organizations responsible for web technology standardization, such as W3C. Conversely, the subjective-oriented approach allows assessing how the final users interact with the artificial system, accessing levels of user satisfaction based on personal factors and environmental barriers. Five kinds of measurements were applied as objective-oriented and subjective-oriented tests. Objective-oriented evaluations were performed on the Help Desk web page for students with disability, included in the website of a large Italian state university. Subjective-oriented tests were administered to 19 students labeled as disabled on the basis of their own declaration at the University enrolment: 13 students were tested by means of the SUMI test and six students by means of the 'Cooperative evaluation'. Objective-oriented and subjective-oriented methods highlighted different and sometimes conflicting results. Both methods have pointed out much more consistency regarding levels of accessibility than of usability. Since usability is largely affected by individual differences in user's own (dis)abilities, subjective-oriented measures underscored the fact that blind students encountered much more web surfing difficulties.
Content may be subject to copyright.
Checking an integrated model of web accessibility and usability
evaluation for disabled people
STEFANO FEDERICI
1,5
, ANDREA MICANGELI
1,2
, IRENE RUSPANTINI
3,4
,
STEFANO BORGIANNI
1
, FABRIZIO CORRADI
4
, EMANUELE PASQUALOTTO
4
,&
MARTA OLIVETTI BELARDINELLI
1,2,4
1
ECONA, University of Rome ‘La Sapienza’, Rome, Italy,
2
CIRPS, University of Rome ‘La Sapienza’, Rome, Italy,
3
Technology and Health Department, Istituto Superiore di Sanita`, Rome, Italy,
4
Department of Psychology, University of
Rome ‘La Sapienza’, Rome, Italy, and
5
Department of Human and Educational Sciences, University of Perugia, Italy
Abstract
A combined objective-oriented and subjective-oriented method for evaluating accessibility and usability of web pages for students
with disability was tested. The objective-oriented approach is devoted to verifying the conformity of interfaces to standard rules
stated by national and international organizations responsible for web technology standardization, such as W3C. Conversely,
the subjective-oriented approach allows assessing how the final users interact with the artificial system, accessing levels of user
satisfaction based on personal factors and environmental barriers. Five kinds of measurements were applied as objective-
oriented and s ubjective-oriented tests. Objective-oriented evaluations were performed on the Help Desk web page for students
with disability, included in the website of a large Italian state university. Subjective-oriented tests were administered to 19
students labeled as disabled on the basis of their own declaration at the University enrolment: 13 students were tested by
means of the SUMI test and six students by means of the ‘Cooperative evaluation’. Objective-oriented and subjective-oriented
methods highlighted different and sometimes conflicting results. Both methods have pointed out much more consistency
regarding levels of accessibility than of usability. Since usability is largely affected by individual differences in user’s own
(dis)abilities, subjective-oriented measures underscored the fact that blind students encountered much more web surfing
difficulties.
Keywords: Psychotechnologies, web accessibility and usability, user satisfaction, space and disability
The theoretical model
The contrast between accessibility and usability is
often reduced superfic ially to that of objectivity and
subjectivity. This does not shed sufficient light on the
complex interaction between technol ogy and user
[1 3]. According to this perspective, accessibility
refers to the environmental characteristics of en-
trance/exit movements. Applying this view to user/
computer systems, it therefore only concerns both
hardware and software of the technological product.
On the other hand, usability pertains to a system
ability to perform the task for which it was designed,
when it is utilized by a specific user [4]. Conse-
quently, usability does not pertain at all to the
technological aspects of a machine functioning, but
to the cognitive aspects of the individual differences.
Defined in this bi-polar manner, accessibility
might be established as the objective end of the user
interaction while usability could be correlated to the
subjective aspects, as determined by users’ inherent
individual differences. From this pers pective, a
technological product is reduced to a neutral entity
that functions independently from its user in a
neutral environment. As a result, a machine could
be perfectly accessible but not usable.
In contrast, the systemic-constructivist perspective
[5] proposes to overcome the false scientific dichot-
omous comparison between object and subject. Each
entity is not considered separately from its observer
Correspondence: Stefano Federici, ECONA, Rome, Italy. E-mail: stefano.federici@uniroma1.it
Disability and Rehabilitation, July 2005; 27(13): 781 790
ISSN 0963-8288 print/ISSN 1464-5165 online # 2005 Taylor & Francis Group Ltd
DOI: 10.1080/09638280400014766
Disabil Rehabil Downloaded from informahealthcare.com by Prof. Stefano Federici on 12/08/10
For personal use only.
during the interpretative/reconstructive process be-
cause the entity is known by the subje ct only as an
observed and perceived object.
From this viewpoint, accessibility and usability are
not understood as characteristics regarding two
separate interacting entities but rather as one system
where both object and subject are just moments in a
multiphase process of empirical observation. This
prevents the existence of user-less technological
products thereby guaranteeing that the accessibility
of a machine always refers only to the possible
entrance and exit of a signal needed to fulfill the task
for which it was designed, and that it is in constant
relation either to its designer or to its user. In this
sense, a machine cannot be accessible and yet
unusable at the same time.
According to this model, accessibility and usability
do not refer to the objective and subjective factors of
the user/technology rapport, but rather to a bidirec-
tional way of observing the interaction. In effect, this
represents two prospective points from which the one
and only observed reality of the user/technology
system is drawn. Accessibility of an environment is
therefore defined based on how it allows the user to
initiate and terminate the operation that completes
the machin e’s task (functioning construct) while its
usability is based on the user’s perception of the user/
technology interaction (user performance). The func-
tioning construct of a machine is the basis for standard
rules, (e.g., Web Content Accessibility Guidelines Work-
ing 1.0 [WCAG 1.0]) against which access ibility
levels are controlled and assess ed. The user perfor-
mance in relation to functioning construct of a machine
allows us to deduce scales (e.g., efficiency, satisfac-
tion, cognitive load, helpfulness) of usability scores.
The systemic-constructivist m odel is compatible with
a universal model of disability whereby abi lity/
disability are viewed within a continuum. Using
ability/disability to refer to an individual functioning
in a real context can only have a theoretical interest
since nobody has a complete absence of disability or
complete absence of ability [6 8]. Therefore, ability/
disability are referred to by the activities performed
by an individual, originating from the environment
and valued by a predetermined functioning construct.
These activities can change the topology of an
environment, and the construct with respect to the
process and measure expected of its functioning.
Experimental plan
Our experimental design was drawn in such a way
that data regarding accessibility and usability dimen-
sions assess the disabled-user/web-interface entire
system. This system is a help desk web page ‘Servizio
Handicap’ (SH), part of the official website of a large
Italian public university. In order to reach this aim,
the experimental design was based on an integration
of objective-oriented and subjective-oriented methods
applied to the observed system.
The objective-oriented method was utilized in order
to verify the compliance of the interface SH with the
standard rules issued by the Web Content Accessibility
Guidelines Working Group. The method was applied
through semi-automatic analytic and empirical tests
onto HTML web page developing code and the SH
functioning, using different internet browsers. Since
this method was principally applied to the SH-
interface features of the observed system, we propose
this as the system’s accessibility parameter.
The subjective-oriented method was utilized in order
to assess how the end user interacts with the artificial
system. It was performed through psychometric
analytical tests and empirical proofs of observation
of behaviour. Since this method was principally
aimed to observe the user beha viour in the system,
measuring the satisfaction levels, we propose these as
usability levels of the student-disabled/SH-interface
system.
The two methods were applied to the observed
system during two experimental Phases, A and B,
through five kinds of measures:
A Objective-oriented tests
1. Preliminary evaluation of the compliance
with the W3C’s WCAG 1.0 using Watch-
fire
1
Bobby
TM
.
2. Accessibility by means of graphic browser
Microsoft Internet Explorer 5.
3. Accessibility by means of textual browser
Lynx 2.8.4.
B Subjective-oriented tests
4. Submission of standardized psychometric
tool: ( Software Usability Measurement In-
ventory) developed by the Human Factors
Research Group’ of the University College,
Cork, with the collaboration of the
MUSiCproject’ .
5. Direct observation of the user’s behavior
by the ‘Cooperative evaluation’ test.
The two methodologies required multidisciplina ry
expertise in psychotechnologies in order to utilize
measures and data score. Phase A was led by a group
of engineers with computer science expertise and
Phase B was led by a group of psychologists with
psychotechnology expertise.
Phase A: Objective-oriented tests
Specific aims
The specific aims were as follows:
782 S. Federici et al.
Disabil Rehabil Downloaded from informahealthcare.com by Prof. Stefano Federici on 12/08/10
For personal use only.
1. To identify the principal kinds of barriers in the
website SH.
2. To assess the compliance with the W3C’s
WCAG 1.0.
These Phase A goals were reached using the SH
home page as the starting point. The path from the
University home page toward the SH web site was
not tested.
Tools and methods
Accessibility evaluation was performed according to
the WCAG 1.0 by W3C. This document comprises
14 guidelines each consisting of various checkpoints.A
priority level is associated to each checkpoint.
The three priority levels are defined as follows:
1. Priority 1: A web content developer must satisfy
the checkpoint as a basic requirement for
accessibility.
2. Priority 2: A web content developer should
satisfy the checkpoint in order to remove
significant barriers to access information in
the document.
3. Priority 3: A web content developer may address
this checkpoint so as to improve access to web
document for a larger user population.
The 14 WCAG are reported here as identified by the
WAI:
1. Provide equivalent alternatives to auditory and
visual content.
2. Don’t rely on color alone.
3. Use markup and style sheets and do so properly.
4. Clarify natural language usage.
5. Create tables that transform gracefully.
6. Ensure that pages featuring new technologies
transform gracefully.
7. Ensure user control of time-sensitive content
changes.
8. Ensure direct accessibility of embedded user
interfaces.
9. Design for device-independence.
10. Use interim solutions.
11. Use W3C technologies and guidelines.
12. Provide context and orientation information.
13. Provide clear navigation mechanisms.
14. Ensure that documents are clear and simple.
Guidelines 1 11 are mainly focused on elegant
transformation. Following these guidelines the web
page developer guarantees user content comprehen-
sion regardless of page layout, access devices, or user
operating context. These guidelines primarily affect
the code development for web pages.
Guidelines 12 14, on the other hand, assess the
basic principles of understandable and navigable web
content design, such as: Using clear language, and
providing simple navigation mechanisms. These
accessibility and usability issues are more related to
website design rather than to code development.
The Watchfire
1
Bobby
TM
4.01 testing software was
used in order to verify the SH’s conformance to the
WCAG checkpoints. It automatically points out
barriers to accessibility, detected in the code, and
highlights crucial page elements that the analyzer
successively has to check manually.
Since accessibility evaluation by automatic tools
cannot identify all accessibility issues, a human
review was performed as follows:
A Evaluating accessibility through a graphical user
interface browser (Microsoft Internet Explorer 5):
1. Turning off images in the browser in order
to verify that the text-equivalent was appro-
priate and informative with respect to the
image content.
2. Testing with different screen r esolution to
verify that the content was always percei-
vable and that a graceful transformation was
guaranteed. The resolutions, considered
most frequently used by the statistical data
provided by Web Counter (http://counter.-
search.bg), are 1024 6 768, 800 6 600,
1280 6 1024, 640 6 480.
3. Turning off the Cascading Sty le Sheets
(CSS) in the browser to make sure that
the content remained understandable.
4. Verifying that color combinations and text-
background contrast were adequate.
B Examining the website by means of the text-
based browser Lynx in order to assess whether
the content is understandable when the page
layout is ignored.
Phase A: Objective-oriented tests
Semi-automatic validation through software Bobby
TM
WCAG 1.0 rule s discourage web developers using
table elements in HTML since some older screen-
readers may not handle side-by-side text correctly. In
the SH, the page layout is designed with nested
tables which make content comprehension complex
for those using textual browser where a linearization
of tables occurs.
The priority 1 checkpoint is not satisfied only in
one case: for one image the text equivalent is not
provided. That image was used for decorative and
not informative purpose, thus the alternative text was
not a funda mental element. However, it is worth
noting that when a user employs a screen-reader a
Web accessibility and usability evaluation for disabled people 783
Disabil Rehabil Downloaded from informahealthcare.com by Prof. Stefano Federici on 12/08/10
For personal use only.
synthetic voice always alerts him that an image is
present, thus useless visual elements should be
avoided so as not to reduce content understanding.
Priority 2 checkpoints results show that form
controls are not provided with labels: these elements
indicate the purpose of a specific control and,
thereby, facilitate blind user in orienting themselves
throughout the page.
Among priority 3 checkpoints the one dealing with
table summary is not satisfied. The summary attribute
furnishes information about table content in few
words. This technical expedient describes the content
to screen-reader users and improves understanding
and navigability. Summary is particularly useful when
employed to distinguish between mere layout tables
and data tables.
Manual accessibility evaluation
Several documents are provided in pdf downloadable
format by the SH site. To dat e, the pdf format is not
automatically recognized by the screen-reader. It is
only possible upon the installation of a plug-in or use
of an on-line converter which are not available for a
lot of users. To supply the same document in another
format, the textual (.doc, .rtf, .txt, etc.) is one
solution. In particular, this solution ou ght to be
supplied for the file ‘directory tape library.pdf’
present in the support section for blind students.
Moreover, some ambiguities regarding names and
links are present:
1. Some textual links are associated to the same
page.
2. Different links point to the same page.
Graphical browser Microsoft Internet Explorer
A. Viewin g without images
Viewing SH site without images is not compromised.
The textual equivalents of images are properly
provided.
B. Viewing at different resolutions
Priority 2 In the SH site, tables and cells
dimensions are not always expressed as percentages.
Hardcoded table dimensions set in absolute values
can cause undesirable transformations in the page
when users set windows and monitor resolution
different from those of the software developer in the
latter’s attempt at achieving optimal layout.
C. View without CSS
CSS use is partial: Elements such as colours and text
connected to the page format are hardcoded in
HTML code. This mix limits the usefulness of CSS
which utilizes format-related defined settings found
in a single file. This readily allows the user to
override the developer-supplied values with his own
style sheet, based on his preference or/and needs and
to substitute the CSS defined by the developer. Since
different style sheets are not interpreted by browser
in the same way, it is necessary to underline the
importance of the code homogeneity.
Priority 1 The contrast between the link and the
background is low when the CSS are not loaded.
D. Colour use test
Contrasts between text and background are appro-
priate. In the SH site no information is relayed by
colours alone, so that contents are accessible to all
users even if a person cannot see colours.
Textual browser text
The critical element in the web page transformation
is the effect caused by tables linearization. In a
textual browser, the table content will appear
following the cell order (from left to right, from top
to down). In a textual browser, it is not possible to
align the columns. These aspects should be taken
into account to allow better understanding of a web
page. Regarding the SH site table linearization does
not diminish text comprehension.
Reading by screen-reader
Using the screen-reader with both browsers created
some problems. The list of links on the web page
left side contains links with different label s that
point to the same page. This problem diminishes
link comprehension because of the confusion
generated from arriving at the same page via
different link names. One figure was not properly
inserted in the text page which made it difficult to
understand the content when it is read by a screen-
reader.
Results of Phase A: Objective-oriented test
Based on the WCAG 1.0 and our analysis, it is
possible to confirm that the website conforms well to
priority 1: in fact it missed just one textual equiva-
lence of an image. An image incorrectly positioned
within the SH home page text compromised the
comprehension when the page was read by mean of
screen-reader.
Regarding priority 2, several problems were found:
1. Some links are not clearly and uniquely identi-
fied (links with different names point to the
same page) making the navigation confusing.
2. Although some style sheets are used for page
setup, they are only used partially and com-
bined with HTML code, thus limiting their
usefulness.
784 S. Federici et al.
Disabil Rehabil Downloaded from informahealthcare.com by Prof. Stefano Federici on 12/08/10
For personal use only.
The visualization by graphical brows ers is adequate,
with respect to ‘elegant transformation’ encouraged
by WAI guidelines.
Priority 3 requirements are not examined in the
present analysis. Given the nature of the SH site’s
targeted users, we stress the importance of supplying
documentation in different formats in order to meet
different users’ needs. In particular, the use of pdf
format should be limited since it requires the user to
take extra steps as discussed above.
Phase B: Subjective-oriented tests
SPECIFIC AIMS
The aims of the subjective-oriented tests were as follows:
1. To assess the satisfaction levels of disabled
university students in their interaction with SH.
2. To identify the principal issues of usability and
cognitive accessibility through the empirical
observation of the user’s behavior.
3. To assess web content usability.
These Phase B goals were achieved by examining the
user’s behavior in their internet navigation from the
university website. It was through this starting point
that the cognitive accessibi lity levels of the SH
website’s contents were evaluated.
Tools and methods
Two kinds of tools were applied in order to surve y
the SH usability and accessibility from a subjective-
oriented perspective:
1. Analytic proof user satisfaction assessing by
means of the test SUMI adm inistered to
disabled students.
2. Empirical proof user’s behaviour observation
by the Cooperative Evaluation test, performed by
six disabled students in order to evaluate the
outcomes of interaction between them and the
SH.
SUMI
SUMI is a questionnaire for measuring user satisfac-
tion. It is designed to be filled out by end users of a
software product being evaluated. It features 50
items, on three-points Likert scale of answer modality
(1 = Agrees, 2 = Don’t Know, 3 = Disagrees). SUMI
contains five sub-scales and a global scale: efficiency,
affect, helpfulness, control, and learnability. It was
administered to 13 subjects, enrolled at the Universi ty
as disabled: 6 with motor disability, 6 with visual
disability, and 1 with both motor and visual disability.
Cooperative evaluation
The ‘Cooperative evaluation’ is a procedure for
eliciting usability feedback from users while they use
a software product [9]. It is a variant of ‘Thinking
aloud’, in which the user is encoura ged to see himself
as a collaborator in the evaluation rather than just a
subject.
As well as getting the user to think aloud, the
evaluator can ask such questions as ‘Why?’ and
‘What if?’; likewise, the user can ask the evaluator for
clarification if problems arise. This more relaxed
approach has a number of advantages. It is less
constrained and therefore easier for the evaluator,
who is not forced to sit in solemn silence; the user is
encouraged to actively criticize the system rather
than simply quietly suffering through it; and the
evaluator can clarify points of confusion, thus
maximizing the effectivene ss of the approach.
Six subjects were evaluated: two students with
motor disability, four students with visual disability
(one blind students and three with diminished
vision).
Figure 1 shows the map of the where the
Cooperative Evaluation was conducted:
1. The student sat on chair marked number 1;
2. The observers sat on chairs marked number 2;
3. The evaluator, interacting with the student, sat
next to him;
4. The supervisor sat on chair marked number 2
behind the observer;
5. At the number 3 there are the two video
cameras.
During the testing, subjects were surfing on a
Compaq Presario 2800 laptop, set with external
speakers, Microsoft
1
Windows
1
XP Professional oper-
ating system, LCD 15-inch external monitor, and
Logitech cordless keyboard and mouse. In order to
browse on internet, subjects could choose among the
software: Microsoft Internet Explorer 6 ; Netscape
Navigator 7; Opera 7 (free version); Mozilla 1.5;
Loupe 5 demo; IBM Home Page Reader 3 demo; Jaws
5 demo.
As soon as the subject sat down, the observer read
him a short explanatory sheet regarding the goal and
the modality of the test. Subjects were asked to
comment out aloud and were informed that there
was no time-limit imposed for the tests. They were
also informed that it was not their behavior that was
being evaluated but rather that of the SH-interface.
The observer took notes on the steps used by the
student as well as on the comments said aloud while
making sure that no influence or disturbance was
exerted on the user’s behavior. The observer could
only interrupt the user for the following reasons: to
Web accessibility and usability evaluation for disabled people 785
Disabil Rehabil Downloaded from informahealthcare.com by Prof. Stefano Federici on 12/08/10
For personal use only.
obtain a bett er understanding of certa in crucial
browsing aspects or to elicit more comments about
the student’s actions. The average test time was
about 30 minutes.
Results of Phase B: Subjective-oriented tests
SUMISCO data analysis
The data were analyzed by the SUMISCO compu-
terized scoring program, following the SUMI User
Handbook guidelines provided along with the ques-
tionnaire [10]. A standardized score is produced to
be compared to the data. The output is standardized
using the z-transform so that the statistical popula-
tion mean score (or m) is 50 and the population
standard deviation (or s) is 10. If some of the sub-
scales are at or below 50 then they are poor in usability
in that aspect. Sub-scale at or below 40 indicate the
need for remedial action. Good software will achieve
scores of 60 or more in most sub-scales (ivi, 33).
Table I shows the total scores of SUMIS six scales,
and Figure 2 highlights the usability features of each
scale.
The following explains the system usability char-
acteristics of each scale from the data processed:
1. Efficiency: This refers to the user’s perception of
the software capacity to perform the task(s) in a
quick, effective, and economical manner. The
scale score is 48, which is slightly lower than the
average, indicating that the scale has a rather
low efficiency. This could be because the
software interfa ce works in an inconsistent
way and therefore does not allow the user to
navigate effectively.
2. Affect: This is a psychological term for emo-
tional feeling. In this context it refers to the user
feeling mentally stimulated and pleasant or the
opposite, as a result of interacting with the
software. The scale score is 58, namely high:
users enjoy their sessions with this software,
they find it mentally stimulating to use; it is
satisfying and attractive.
3. Helpfulness: This refers to the user ’s perceptions
that the software communicates in a helpful
way and assists in the resolution of operational
problems. The scale score is 46, namely low:
the web page is not very helpful as information
is not seen consistently.
Figure 1. Cooperative Evaluation map setting.
Table I. Total scores of SUMI’s scales
Scale UF Ucl Medn Lcl LF
1. Efficiency 79 55 48 41 19
2. Affect 86 64 58 52 26
3. Helpfulness 82 53 46 39 22
4. Control 72 50 44 38 24
5. Learnability 74 67 60 53 38
Global 76 60 54 48 26
UF = Upper Fences
Ucl = Upper Confidence Limits
Medn = Median
Lcl = Lower Confidence Limits
786 S. Federici et al.
Disabil Rehabil Downloaded from informahealthcare.com by Prof. Stefano Federici on 12/08/10
For personal use only.
4. Control: This sub-scale refers to the user’s
feeling that the software is responding in a
normal and consistent way to inputs and
commands. It is not difficult to make the
software work, the user can get their work done
with ease. The scale score is 44, namely low:
users judge it to be unstable and unreliable
since they do not always get an adequate
response from deterministic actions.
5. Learnability: This sub-scale refers to the feeling
that the user has that it is relatively
straightforward to become familiar with the
software and that its tutorial interface, hand-
books etc. are readable and instructive. The
scale score is 60, namely high: the web page is
easy to get into.
6. Global scale: It represents a general usability
measure. The Globa l scale is a weighted sum of
the most important usability items in the SUMI
scale. The scale score is 54, namely almost high:
the SH web page has a good usability level.
Cooperative evaluation results
Starting from the search engine Google
TM
home page
(www.google.it), the subje cts were invited to search
the help desk for students with dis ability of the
University website. Five out of six subjects located
the SH web page within the ten minutes allotted. It
was not easy searching for the SH web page since the
path preferred by the subjects starts from their
respective Faculty home pages which do not always
contain a link to the page in question.
Those subjects who used the screen-reader for
browsing did not reach the SH web page from the
University home page, since the text, provided as an
equivalent alternative to the image link connecting to
the student section, was evidently different from the
graphic one and did not use conventional words.
This resulted in confusion and thus led the blind
students to discard it.
Moreover, the subjec ts met problems accessing
several ot her sections from the University home
page. This is because it is designed with frames
which prevent one from easily jumping from one
area to the next within the site. It must be
mentioned that the use of frames is not even
necessary since they are not utilized as content
containers.
Once the subjects got to the SH site, they were
interviewed in accordance with the evaluation pro-
tocols:
Figure 2. Usability features of SUMI’s scales.
Web accessibility and usability evaluation for disabled people 787
Disabil Rehabil Downloaded from informahealthcare.com by Prof. Stefano Federici on 12/08/10
For personal use only.
1. What they thought the name of this University
web site section was;
2. What kind of resources they would have and
would have wanted to find there.
Then, they were asked to navigate the site, and to
decide freely when to stop. Once they had finished,
the subjects were interviewed again about their
experience with the SH interface. Afterwards, the
two experimenters organized the data and watched
the videotapes recorded duri ng the experimental
sections in order to gather information which may
have been missed during the experimental se tting.
The SH site is designed in four areas (see Figure
3): on the top there are two horizontal areas reaching
the entire page width Area 1 and 2 and below that
there is the real site divided in two vertical areas
Area 3 and 4.
1. Area 1: The University ID logo appears on the
left side while the Help Desk’s toll free number
is a little bit to the right of the center.
2. Area 2: This contains five links: Students; Home;
Disability Help Desk (SH); Toll free number;
Contacts.
3. Area 3: This contains the navigation menu of
the section which provides single click passages
to almost all of the site contents, a slogan
image, and three links to external resources.
4. Area 4: The principal area, bordered by a
beveled-angle box, gathers all of the site
contents.
The Cooperative Evaluation results are divided into
three categories:
1. User’s behavior while browsing within the SH.
2. Browsing-specific problems.
3. User satisfaction with the SH contents.
The outcomes of the evaluation of the users’
interaction with the SH point out that:
1. Students with motor disability did not show
particular problems in navigation. Instead, they
easily learned the navigation paths, accomplish-
ing the committed tasks effortlessly.
2. On the other hand, students with visual
disability, using a screen-reader, found barriers
because of Areas 1 and 2. In particular, Area
caused misunderstanding of the page identifi-
cation because of the University title in the
site ID. Moreover, the lack of a means to
bypass Areas 1, 2, and 3 constrains the
subjects to navigate through all the areas at
all times and thus unreasonably delaying
direct access of the contents in Area 4
(navigation goal of any link).
The identification of the browsing problems high-
lights these following points:
1. The page download and disp lay speed is fast
thus providing users with a feedback in a
reasonable time, thanks to an economical use
of graphic elements.
2. The font size is freely changeable (flexibility
and efficiency in use). The home page width is
adjustable, however the tables within the page
do not automatically resize proportionally,
rendering the layout not ‘liquid’. Although we
did not find this to be a problem in our
setup: 15-inch monitor with a resolution of
1024 6 768 pixels and the highest color quality,
the lack of liquid layout could cause problems
for users wi th low resolution monitors, espe-
cially when reading contents of Area 4.
3. The design is minimalist. Three of the four
sections contain a link and/or reference to the
toll free number help desk: redundant informa-
tion slows down web page scanning (particularly
using screen-reader).
4. Using the screen-reader, scanning links of Area 3
is often interrupted by the link set on the
graphic pointer, slowing down the browsing
and delaying the presentation of the searched
content.
The evaluation of the web-page content s shows us a
low level of satisfaction:
Figure 3. Layout of SH web page.
788 S. Federici et al.
Disabil Rehabil Downloaded from informahealthcare.com by Prof. Stefano Federici on 12/08/10
For personal use only.
1. Although the contents are easily reachable from
the Area 3 menu, where titles are clear and self-
evident, blind subjects must listen all the time
to all the links of Areas 1, 2, and 3, before
arriving at Area 4 for the demanded content.
2. Subjects showe d a low level of satisfaction
because of a generalization of the contents, a
scarcity of information, and the constant
referral to the help desk office or call center
for furthe r information.
3. Blind students expressed dissatisfaction be-
cause much of the contents are in pdf format,
hence only readable through Adobe Acrobat
#
Reader. Moreov er, several blind students re-
marked on the incoherence of the pdf format
use for looking up the catalogue of the
audiotape courses in the section regarding
support for blind students.
Conclusion
The results of the two test phases are in accordance
with the systemic-constructivist model adopted in this
study regarding two aspects:
1. According to our experimental plan, both
accessibility and usability were proposed as two
distinguishably observable and assess able reali-
ties, measured in Phases A and B with different
methods, because of the disciplinary expertise of
different research groups. Nevertheless, results
proved difficult to show that accessibility and
usability are separated entities in interaction.
Instead, they reveal a much more dynamic and
complex reality that is attributable to the user/
technology rapport as discussed above.
In both objective-oriented and subjective-oriented
validation methods, problems found regarding
accessibility and usability prevented us from
cleanly segmenting the observed data into the
two categories, funct ioning construct and user
performance, which are theoretically and metho-
dologically distinct. In both Phases of the
analysis, we had to resort to the systemic
interaction which revealed similar, if not iden-
tical, problems regarding accessibility, that is,
the want of system functioning expected. Similar
problems regarding accessibility surfaced during
usability tests, of which specific examples are the
difficulty encountered in navigating with the
left-side menu of the SH, while testing user
behavior via Cooperative Evaluation.
2. As a direct consequence of this previous point
and as confirmed by the res ults, the integration of
the objective-oriented and subjective-oriented ap-
proaches as well as the research groups’ different
disciplines and methodologies have proven to be
the appropriate choice for highlighting the
observed system’s complexity, disabled-stu-
dents/SH-interface. This is true even if each
approach has previously defined only one
system parameter to be observed, such as the
dependent variable of how the software works
and the modality of the user’s navigation. This
demonstrates the difficulty of selecting (with
accessibility and/or usability) only one system
parameter without having to constantly refer
back to the system in its entirety, such as the
indispensable signifier of the observed para-
meter.
As seen from the objective-oriented and sub-
jective-oriented test results, problems relating to
the following were found: the use of a screen-
reader (due to redundant links), the navigation
(the side menu bars with unclear and/or redun-
dant links), and the content availability in only
one format (pdf). In both objective-oriented and
subjective-oriented analyses, the obtained results
would be attributable to both accessibility and
usability, if according to current ergonomic
definitions. Both analyses also show that the
existence of disabled-stud ent/SH-interface in-
teraction problems cannot be attributed to only
just one parameter. According to modern
epistemology, the results force us to a ‘‘recon-
struction of the field of study based on new
factors, a reconstruction which modifies some of
the most fundamental theoretical generalizations
of the field’’ [11].
We therefore maintain that an integrated model for
validating usability and accessibility of websites offer
great opportunities for observing user/technology
systems, even in situations where observed data are
to be classified in only one of the two aforemen-
tioned categories. The formulation of web design
standard rules (accessibility) could be accepted
more universally when it takes better into account
the different individual users’ predispositions in
their use of web technology (usability). We would
like to point out that the law 9/04, recently
promulgated by the Italian Parliament, regarding
user-friendliness of software systems for disabled
users, is in accordance with the ample vision of
accessibility. Individual users’ characteristics, dis-
ability issue in particular, are definitely included in
the definition of the aforemen tioned law. The more
restrictive definitions are found on the usability
side. In fact, regarding accessibility, the legislation
states that ‘‘the capacit y of the software systems
must provide services and useful information,
without discrimination, even for those who, due to
their disabilities, need assistive technology or
particular configurations’’. (art. 2).
Web accessibility and usability evaluation for disabled people 789
Disabil Rehabil Downloaded from informahealthcare.com by Prof. Stefano Federici on 12/08/10
For personal use only.
It is necessary to take into account the user
entrance in the functioning construct of a machine,
particularly in cases where individual functioning
require use of assistive technologies. The usability of a
technological product is very much tied to its capacity
to neutralize and break down barriers that restrict
users from accessing it in order to take advantage of it.
Therefore, an integrated model of validation is the
most suitable methodology when analyzing complex
systems of user/technology interaction.
With regards to the functionality of the SH website,
results show barriers in the form of both navig ation
and inaccessible content due to format. Moreover, the
presented information is general and non-exhaustive.
Overall, the site is only accessible on a superficial level
in that some pertinent information remains hidden
(please see results regarding compliance with the WAI
Standard rules). This shows that the relationship
between the functioning construct (how the site ought
to work) and the user performance (visually and
motor disabled students) contains numerous barriers.
The following steps must be undertaken in order to
improve the accessibility and usabil ity of the SH web
site so as to increase students’ level of satisfaction:
1. Eliminate Areas 1 and 2 and clean up links.
2. Allow a quicker way of get ting the information
requested (without the need to read the entire
menu).
3. Use cascading links to improve the site’s
presentation and navigation.
4. Offer formats other than pdf for downloadable
files.
5. Enrich the content with m ore information on
services.
References
1. Annett J. Subjective rating scales: Science or art? Ergonomics
2002;45:966 987.
2. Annett J. Subjective rating scales in ergonomics: A reply.
Ergonomics 2002;45:1042 1046.
3. Kirakowski J. Is ergonomics empirical? Ergonomics,
2002;45:995 997.
4. Marcolin F, Mian G, Ossicini A, Luisi F, Pischiottin S, Vecchi
Brumatti L. Glossario di ergonomia. Milano: INAIL; 2002.
5. Olivetti Belardinelli M. La costruzione della realta`. Torino:
Boringhieri; 1973.
6. Zola IK. Toward the necessary universalizing of a disability
policy. Milbank Quarterly 1989;67(suppl):401 428.
7. Bickenbach JE, Chatterji S, Badley EM, U
¨
stu¨ n TB. Models of
disablement, universalism and the international classification
of impairments, disabilities and handicaps. Soc Sci Med
1999;48:1173 1187.
8. World Health Organization. International classification of
functioning, disability and health: ICF. Geneva: WHO; 2001.
9. Monk A, Wright P, Haber J, Davenport L. Improving your
human computer interface: A Practical Technique, New York:
Prentice Hall; 1993.
10. Kirakowski J. SUMI User Handbook. 2nd ed. Cork: Human
Factors Research Group, University College Cork; 1998.
11. Kuhn TS. La struttra delle rivoluzioni scientifiche. 4th ed.
Torino: Einaudi; 1978.
790 S. Federici et al.
Disabil Rehabil Downloaded from informahealthcare.com by Prof. Stefano Federici on 12/08/10
For personal use only.
... 8), diritto sancito dalla Convenzione dell'ONU sui diritti delle persone con disabilità (UN, 2006). L'accesso alla tecnologia coinvolge di fatto molteplici dimensioni − la consapevolezza dell'uso, la disponibilità e la qualità del computer e della rete, l'accessibilità relativa all'input e all'output del sistema che consentono a particolari (gruppi di) utenti di utilizzare tutte le strutture del sistema, l'adattabilità alle esigenze degli utenti (Boot et al., 2018), l'usabilità relativa alla capacità di farlo in modo efficace, efficiente e con soddisfazione (Federici et al., 2005) -e la letteratura di settore si è pertanto occupata di riconoscere il divario significativo esistente tra necessità, accesso o utilizzo della tecnologia digitale e la disponibilità di tecnologie calibrate sui bisogni degli adulti con disabilità (Boot et al., 2018;Moisey & van de Keere, 2007). Essa ha inoltre analizzato gli ostacoli ai bisogni insoddisfatti (in termini di disponibilità e accesso alla tecnologia) e il loro impatto sullo sviluppo delle persone con disabilità (Boot et al., 2018;Palmer et al., 2012a;2012b;Wehmeyer et al., 2008). ...
Article
Full-text available
This paper describes the Accessible Information Material project aimed at promoting digital literacy for adults with intellectual disabilities and/or with low levels of literacy. The paper, that is introduced by a brief discussion of the role of ICT in promoting the inclusion of disabled people, intends to make a contribution to the design of digital literacy practices. The creation of easy-to-read material and the development of a path for the training of adults with intellectual disabilities are designed as tools to facilitate access and utilization of digital technology and to support their inclusion.
... Several standardized tools have been developed to measure satisfaction, realization of benefit and perceived usability of user with and without disabilities [5][6][7][8][9][10][11]. It is also well known that if the UX of a product is assessed at the end of the design process, product changes are much more expensive than if the same evaluation were conducted throughout the development process (i.e., according to a usercentered design, UCD) [5,12]. ...
Chapter
Full-text available
To fully model the perceived experience of a user, practitioners should include a set of repeated objective and subjective measures in their evaluation protocols to enable satisfaction and benefit analysis as a “subjective sum of the interactive experience.” It is also well known that if the UX of a product is assessed at the end of the design process, product changes are much more expensive than if the same evaluation were conducted throughout the development process. In this study, we aim to present how these concepts of UX and UCD inform the process of selecting and assigning assistive technologies (ATs) for people with disabilities (PWD) according to the Matching Person and Technology (MPT) model and assessments. To make technology the solution to the PWD’s needs, the MPT was developed as an international measure evidence-based tool to assess the best match between person and technology, where the user remains the main actor in all the selection, adaptation, and assignment process (user-driven model). The MPT model and tools assume that the characteristics of the person, environment, and technology should be considered as interacting when selecting the most appropriate AT for a particular person’s use. It has demonstrated good qualitative and quantitative psychometric properties for measuring UX, realization of benefit and satisfaction and, therefore, it is a useful resource to help prevent the needs and preferences of the users from being met and can reduce early technology abandonment and the consequent waste of money and energy.
... As can be seen by our indicator selection, we did not consider the overall accessibility of the website design or whether there were assistive technologies. The accessibility of the information itself and of the website have been addressed in many studies (Domínguez Vila et al., 2018;Akgül and Vatansever, 2016;Buzzi et al., 2011;Shi, 2011;Jaeger, 2006;Federici et al., 2005), which can be checked for further details. In this study, we only considered the provision of information about physical accessibility, in other words, if the information is available and can be found by a person that has reduced mobility but can access the website regardless of the existence of any specific assistive tool. ...
Thesis
Full-text available
Accessibility refers to how easily someone can reach their destination. It also can be used as an indicator of transport performance and the quality of land use. Moreover, it is well-known that accessibility strongly influences access to basic needs, such as jobs, education, and health care. Unsurprisingly, personal mobility is one of the fundamental human rights and it is vitally important to ensuring a productive and dignified life. According to the World Health Organization, 15% of the world’s population has some type of disability, and forecasts show that this number is likely to rise as the population ages. Moreover, studies have shown that people with reduced mobility often face difficulties over simple daily trips, due to inadequate infrastructure in urban areas. Thus, to safeguard equity of mobility, it is imperative to provide urban environments and transport systems, particularly public ones, that are accessible to everyone, without discrimination. Urban railway systems are a common alternative in urbanized and heavily populated cities, due to their high capacity, safety and ease of use. Because of the central role played by this mode, strategies such as Transit-Oriented Development (TOD) have emerged and gained popularity. TOD refers to the creation of dense, diversified, integrated and connected urban communities around transport stations, which helps to reduce the demand for transport, by favoring the use of public transport and active mobility. Despite its significance, studies suggest that worldwide less than half of the urban railway systems are fully accessible to people with disabilities. In this context, and considering both the importance of the topic and the scarcity of literature about it, this thesis aims to propose a method to evaluate the accessibility level of urban railway stations for people with reduced mobility, in the context of Transit-Oriented Development urban design. The novel method that we are proposing in this study comprises over 40 indicators distributed in three categories: (1) Information about accessibility; (2) Accessibility of urban railway stations and trains; and (3) Accessibility of station surroundings and transfers. To develop the indicators, we examined many international guides, manuals, standards and assessment tools. The list of indicators was also assessed by a specialist in accessibility for people with disabilities. To determine the weighting of the categories and subcategories under analysis, we used an expert assessment panel and an analytical hierarchy process (AHP-Express) model. The panel was composed of acknowledged experts with diverse professional backgrounds. We also tried to include professionals with disabilities, as they would be able to provide some feedback from their own user experience. We performed a global assessment using the method developed for the information online category and we submitted the complete method to an application in a few Taipei city metro stations. Finally, the thesis also proposes an Accessibility Index, which can be used to classify the stations and rank them according to their overall accessibility. The results of this study suggest that this novel method can be valuable for researchers and decision-makers during different stages of the TOD implementation process. In addition to being an easy-to-use tool to measure the current accessibility status of urban railway stations, the proposed method also enables classification considering multiple categories, which can help to identify critical points that need to be prioritized.
... Due to the globalizing progress of civilization, nowadays people's work is becoming disconnected from their place of residence. This results in people with disabilities or other difficulties not having to be confined to a specific location, but being able to perform their jobs remotely, from anywhere in the world (Federici et al., 2005). ...
Article
Full-text available
Purpose: The purpose of the research is to present the algorithm for the validation of economic and financial websites, allowing to check the degree of accessibility of these websites in terms of WCAG standards with particular attention to people with disabilities and the elderly. The developed method is automatic and allows to obtain both a general score for the analyzed page and a detailed list of analyzed aspects. Design/methodology/Approach: A two-step needs analysis process was conducted to develop the algorithm. In the first phase, the literature was analyzed for selected aspects of accessibility particularly relevant to people with disabilities and the elderly. Findings: The results made it possible to determine the general directions of the algorithm, which are particularly important for the mentioned users. The next stage was the analysis of selected aspects of the websites in the context of their relation to the groups of recipients and the possibilities of analysis by the algorithm. The result of this stage is a set of 9 aspects, within which the individual elements checked by the algorithm were dissected. Practical Implications: The results obtained indicate that the evaluation made by the algorithm coincides with the evaluation of the accessibility of the pages made by the experts. The algorithm allows us to quickly analyze web pages for accessibility and to detect general trends that characterize a given theme of pages. It allows to make general recommendations for improvements and good practices for building accessible pages according to WCAG recommendations. Originality value: The algorithm created is a proprietary project that allows the automatic evaluation of websites. It is based on 9 aspects that make up websites, with the possibility of expanding to more. It can be used to supplement expert knowledge or be used as a general tool to inform about the state of accessibility for a given page.
... Most disabilities have a wide range of conventional sensory human dysfunctionality. Regardless of the sensory disability or its severity the accessibility "should offer equal opportunity to access information, removing some barriers to the communication and interaction that faced by users with disabilities in the real world" (Domínguez Vila, Alén González, & Darcy, 2018a;Federici et al., 2005;Król & Zdonek, 2020;Kuzma, 2010;Nuñez, Moquillaza, & Paz, 2019). WAVE Accessibility testing enables optimum accessibility. ...
Article
Full-text available
The aim of this research intends to investigate Taiwanese egovernment websites with regards to Accessibility and Usability. The automated WAVE Accessibility tool and Nielsen’s 10 Heurestic Principle based surveys of foreign users of the English egovernment websites were used to test Accessibility and Usability respectively. The study found poor results that were unanimous across the board for Usability and total of just over 100 errors for Accessibility was the best performing website. The study proves that further research and investigation is necessary if Taiwan wants to meet its own governmental ambitions with regards to egovernment.
... The CSUQ (Computer System Usability Questionnaire) (Lewis 1995), (Federici et al. 2005) instead consists of 19 statements whose answers, as for the previous questionnaire, express a degree of agreement on a scale of 1-5 and are divided into four areas. The SYSUSE is used to evaluate the usefulness of the system, the INFOQUAL measures the quality of the information, the INTERQUAL the quality of the interface while the OVERALL is used to measure general satisfaction. ...
Article
Nowadays, new technologies are taking place for enabling conversational interactions between users and bots. Creating conversational interfaces (CI) paves the way for new design challenges. Designers need to use appropriate design specifications for creating a logically sound dialogue endowed with proper visual cues in the case of chatbots that captures the entire user experience. To do it they need the help of domain experts. Experts in mobility, environment, energy, e-health, weather, etc., are the only ones who well-know the specific domain in which the bot will act. Current tools used to design interactive flows of dialogue entail deep programming skills their users need to know. Starting from this consideration, the paper proposes a design environment that offers experts the possibility to design a flow of dialogue without getting lost in technicalities. By using visual language, the expert can specify intents, actions, entities and parameters at the base of the flow of dialogue on which the bot will be created. Then, our engine automatically translates into a bot compliant with the Google DialogFlow technology. Finally, the paper present a preliminary analysis of the system carried out to test its features and how these are reflected in its ease of use.
... As can be seen by our indicator selection, we did not consider the accessibility of the website design or whether there were assistive technologies. The accessibility of the information itself and of the website have been addressed in many studies [8,[82][83][84][85][86], which can be checked for further details. In this study, we only considered the provision of information about physical accessibility, in other words, if the information is available and can be found by a person that has reduced mobility but can access the website regardless of the existence of any specific assistive tool. ...
Article
Full-text available
It is estimated that more than one billion people worldwide have some form of disability, and that number is expected to rise as the population ages. A lack of accessible transport can represent a challenge to commuting citizens and it can also inhibit tourists with reduced mobility. Online information about accessibility is the first point of contact that tourists have with their destination and it should therefore be considered an attribute of accessibility. In that context, this paper aims to: (a) propose a method for classifying the information about accessibility provided by the official websites of public transport systems; and (b) present and discuss the results of the application of the method to the official websites of 212 urban rail systems around the world. The results suggest that, despite it being the first indication of destination accessibility, many cities do not provide or provide unclear or insufficient information for tourists with reduced mobility on their official websites. Moreover, few official websites provide information on accessibility around stations or in the case of transfers. This novel method proved to be suitable for classifying the websites, as well as identifying aspects of the information provision that can be improved.
Chapter
This chapter presents an overview of accessibility and usability for educational computer-based games and the first survey of the accessibility and usability of digital educational games. The overview includes a discussion of accessibility and usability, both in general and in the specific context of educational games, as well as a brief presentation of issues relating to game design, including of mobile games. Since there are no previous studies of the accessibility and usability of educational computer-based games, studies of the accessibility and usability of the related areas of virtual learning environments, digital games for entertainment and PDF documents, are also presented. The overview of accessibility and usability and the results of the survey are used to draw up a structured list of 62 guidelines and recommendations, organised into three categories at the first level and ten at the second level. These guidelines and recommendations are illustrated by an example of their application to a fictitious new educational game.
Article
Comments on the target paper are acknowledged. Whilst there is still some concern that subjective rating scales are scientifically suspect, the general view is that verbal reports, including ratings, constitute objective data, which can be of considerable value in ergonomics research and practice. The main anxiety attached to their use is to ensure acceptable levels of reliability and validity. Quantification in the strict sense can be achieved by some measures but is by no means essential for all scientific and practical purposes. The value of ordinal and qualitative data obtained by subjective judgements should not be underestimated, especially in predicting future performance. Many commonly used constructs such as fatigue, stress, mental workload, usability, etc. are complex and multidimensional, often combining both ‘subjective’ and ‘objective’ measures. The validity of individual dimensions and complex constructs lies principally in their relationships with other variables of interest in the context of the specific investigation. The question of design standards based partly or wholly on such scales should therefore be treated with some caution.
Article
Issues confronting people with disabilities do not result solely from physical or mental impairment, but from the fit between impairments and practically every feature of the social, economic, physical, and political environment. Changes in housing, transportation, and employment policies would augment the quality of daily living for those with disabilities today and in the future. With the entire population facing chronic illness and activity limitations, a universal approach to disability is virtually required, rather than policies focusing exclusively on a person's special needs. The absence of such a universal perspective will lead to the expansion and perpetuation of the segregated and unequal society visible at present.
Article
A review and critique of models of disability is presented, tracing the development of frameworks and classificatory instruments (International Classification of Impairments, Disabilities and Handicaps) over the past 20 years. While the 'social' model is now universally accepted, it is argued that universalism as a model for theory development, research and advocacy serves disabled persons more effectively than a civil rights or 'minority group' approach. The development of the revised International Classification (ICIDH-2) is discussed in this light.
Article
Subjective rating scales are widely used in almost every aspect of ergonomics research and practice for the assessment of workload, fatigue, usability, annoyance and comfort, and lesser known qualities such as urgency and presence, but are they truly scientific? This paper raises some of the key issues as a basis for debate. First, it is argued that all empirical observations, including those conventionally labelled as 'objective', are unavoidably subjective. Shared meaning between observers, or intersubjectivity, is the key criterion of scientific probity. The practical steps that can be taken to increase intersubjective agreement are discussed and the well-known sources of error and bias in human judgement reviewed. The role of conscious experience as a mechanism for appraising the environment and guiding behaviour has important implications for the interpretation of subjective reports. The view that psychometric measures do not conform to the requirements of truly 'scientific' measurement is discussed. Human judgement of subjective attributes is essentially ordinal and, unlike physical measures, can be matched to interval scales only with difficulty, but ordinal measures can be used successfully both to develop and test substantive theories using multivariate statistical techniques. Constructs such as fatigue are best understood as latent or inferred variables defined by a set of manifest or directly observed indicator variables. Both construct validity and predictive validity are viewed from this perspective and this helps to clarify several problems including the dissociation between measures of different aspects of a given construct, the question of whether physical (e.g. physiological) measures should be preferred to subjective measures and whether a single measure of constructs which are essentially multidimensional having both subjective and physical components is desirable. Finally, the fitness of subjective ratings to different purposes within the broad field of ergonomics research is discussed. For testing of competing hypotheses concerning the mechanisms underlying human performance, precise quantitative predictions are rarely needed. The same is frequently true of comparative evaluation of competing designs. In setting design standards, however, something approaching the level of measurement required for precise quantitative prediction is required, but this is difficult to achieve in practice. Although it may be possible to establish standards within restricted contexts, general standards for broadly conceived constructs such as workload are impractical owing to the requirement for representative sampling of tasks, work environments and personnel.
La costruzione della realta `
  • Olivetti Belardinelli
Olivetti Belardinelli M. La costruzione della realta `. Torino: Boringhieri; 1973
Cork: Human Factors Research Group
  • J Kirakowski
  • Sumi User
  • Handbook
Kirakowski J. SUMI User Handbook. 2nd ed. Cork: Human Factors Research Group, University College Cork; 1998.
La struttra delle rivoluzioni scientifiche
  • T S Kuhn
Kuhn TS. La struttra delle rivoluzioni scientifiche. 4th ed. Torino: Einaudi; 1978.
Toward the necessary universalizing of a disability policy Milbank Quarterly
  • I K Zola