Evaluation of health care related web resources based on web citation analysis and other quality criteria.
ABSTRACT Automatic system for ranking of educational health care related web resources based on quality criteria was developed. Automatic retrieval of new web resources from well-known directories followed by their automatic evaluation is a keystone of the system. The system provides this information: a) location of the resource and its author, b) number of back links to the resource, c) checks presence of electronic labels of quality. Currently the system encompasses more than 500 links to educational web resources divided into 47 medical specialities. The resources in all categories are ranked according to their link popularity, the electronic labels of quality are presented. History of link popularity is recorded and might be displayed at every resource. There are 2 language editions. The system operates automatically, the editors check and correct the retrieved values. The described system adds to webcite indexing criteria for objective evaluation of quality of webpages. It is useful for the selection of optimum education resources in health care.
- SourceAvailable from: ncbi.nlm.nih.gov[Show abstract] [Hide abstract]
ABSTRACT: The Columbia University Infobutton Manager (IM) is a system that provides dynamically generated, context-specific links between clinical information systems and online information resources. The resources range from local documents, to commercially available document sets and search engines. The links provided by the IM can be reliably created, but there is no guarantee that they will function reliably, since the resources to which they point are subject to unannounced changes and failures. We have developed a set of tools to audit the links periodically to determine if the resources are available and if the IM has sufficient information to generate all the links needed by its users. These tools have been in use since February, 2006 and have provided timely warnings on many occasions. These warnings have allowed us to correct problems with resource access before they became apparent to our users and often before the resource maintainers were aware of the problems. The tools have thus helped us provide clinicians with a dependable level of service.AMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symposium 02/2007;
Evaluation of health care related web resources based on web citation
analysis and other quality criteria
P. Kasal, A. Janda, J. Feberova, T. Adla, M. Hladikova, J.P. Naidr, R.Potuckova
Institute of Medical Informatics, 2nd Medical Faculty, Charles University, Prague
V Uvalu 84 150 06 Prague 5, Czech Republic
Automatic system for ranking of educational health care
related web resources based on quality criteria was
Automatic retrieval of new web resources from well-known
directories followed by their automatic evaluation is a
keystone of the system. The system provides this
information: a) location of the resource and its author, b)
number of back links to the resource, c) checks presence of
electronic labels of quality.
Currently the system encompasses more than 500 links to
educational web resources divided into 47 medical
specialities. The resources in all categories are ranked
according to their link popularity, the electronic labels of
quality are presented. History of link popularity is recorded
and might be displayed at every resource. There are 2
language editions. The system operates automatically, the
editors check and correct the retrieved values.
The described system adds to webcite indexing criteria for
objective evaluation of quality of webpages. It is useful for
the selection of optimum education resources in health care.
The Internet is a very powerful information medium,
however, the quality of the content is extremely variable and
this fact limits its use as a source of important information.
This feature of the net becomes crucial in the health-care
related webpages where dangerousness of misinformation
is obvious and may confuse or even harm the user. It is
virtually impossible to control the content of web pages, so
it would be useful to have some tools to measure the quality
of provided information, e.g. in the form of rating system
based on standard criteria. Many attempts have been made,
and core standards that can help to achieve this goal have
For example survey of published criteria for evaluating
health related web sites was performed by Kim et al. 
They found 29 rating tools whose authors agreed on the key
criteria: criteria dealing with content, design, aesthetic of
site, disclosure of authors, sponsors,
currency of information, authority of the source, ease of use
and accessibility and availability. They concluded that
creation of any evaluative static tool would be difficult
and inappropriate. This opinion is supported also by
Gagliardi and Jadad  who found 47 rating instruments
appearing on websites in 1998, and in survey repeated in
2002 they found that most of the previously evaluated tools
no longer exist.
The dynamic behaviour of the Internet limits efficacy of
establishment of the quality criteria and the rating tools that
The lack of evaluated source with links to educational
webpages for professionals in
implementation of a new system - Citmed. The main
criterion for adding a resource to the
list was a high number of links to the webpage – “link
Another marker of the quality taken into consideration in the
selection process was a presence of electronic labels of
quality: HON, HardinMD, Medical Matrix.
This project is conducted under the auspice of the Ministry
of Health of Czech Republic and is a part of the initiative of
European Union „e-Europe+, Healthcare on- line“.It is
available at http://www.citmed.cz.
There is another system interconnected with Citmed
focused on general quality evaluation of webpages, named
Rankmed. Preliminary results of these projects have
already been presented.[11, 15]
health-care led to
The system Citmed was developed to enable automatic
retrieval of new web resources according to set criteria
followed by their evaluation. The automatic engine provides
1) Location. It finds a location of the resource by inquiry to
(query: http://www.alexa.com/data/details?url=url ).
2) Link popularity (back links). It traces new links to the
particular webpage. To exclude selfcitation, only external
links to the site are counted. Only one link a server (to
prevent multiplication of information in the same place) and
links created in the past 12 months are calculated. Server
AlltheWeb (http://www.alltheweb.com ) is used. We have
chosen this server according to the results of previous
research done by our Institute. Quantification of links to
the website depends on the scope of the webspace the engine
searches, by no means the number
Proceedings of the 2005 IEEE
Engineering in Medicine and Biology 27th Annual Conference
Shanghai, China, September 1-4, 2005
0-7803-8740-6/05/$20.00 ©2005 IEEE.
of links displayed in Citmed and obtained from AlltheWeb
are the total number of links linked to the particular site.
Previous research states that it would be necessary to
combine results from at least five large search engines to
cover majority of the resources on the web. Due to
technical reasons, however, we selected only one, the most
suitable, engine. The link popularity displayed is therefore
just a part of the citation of the resource and it must be
treated as a
3) Electronic labels of quality. MedicalMatrix. It retrieves
stars at http://www.medmatrix.org. It is a representative
of post publication editorial process. The target audience of
this web service are physicians and health-care workers, the
editorial board assigns respective number of stars to the
given source. Number of the stars corresponds with
reliability, accurateness, currentness, usefulness and extent
of the site and its design.  HONcode. It is checked
whether HON code logo was assigned to the page at
http://www.hon.ch/HONcode/Hunt. The presence of this
label symbolizes adherence to HON Code of conduct that
adjut authority, complementarity, confidentiality, attribution,
justifiability, transparency of authorship and honesty in
advertising and editorial policy of the site. HardinMD.
The presence of
http://www.lib.uiowa.edu/hardin/md is verified.
HardinMD's Clean Bill of Health award is given only to the
top-level pages that have connection rates of at least
4) New resources. The system selects new links for
Only links that are not already in the database or that were
not marked to be ignored are proceeded. Besides URL the
engine retrieves other data, e.g. metadata from the html
source of the page (description, keywords, title).
It is then used for categorization of the link. The engine
always caches the whole site and works in off-line mode.
The following directories are scanned:
a) Karolinska Institutet - http://www.mic.ki.se
Collecting of links is focused on the category “Netsim“ with
educational links and main part of the medical library,
category „Diseases/Disorders and Related Topics“.
It is a directory of directories. All resources with Clean Bill
of Health award are retrieved.
c) MedicalMatrix - http://www.medmatrix.org
Resources with minimum number of stars (set
by administrator of the system) are retrieved.
d) MedHunt - http://www.hon.ch/Medhunt
Particular categories are put into the searching form by the
administrator. The links that have relevance over a set limit
HardinMD's logo at
Currently system Citmed encompasses more than 500 links
to educational web resources divided into 47 medical
specialities. The found resources are sorted according to
the keywords to relevant categories. The information
retrieved automatically by the system: a) identification of the
resource (location of the resource and its author), b) impact
of the resource on the users (number of back links to the
source), c) electronic labels of quality, relative proportion of
citation of the resource to the other resources in Citmed.
d) and retrieval of new resources. In all categories the
resources are ranked according to their link popularity. The
name, the origin and the type of a document (I – image, Q -
test/quiz C - case report, G – guidelines, T – textbook, X –
other) are displayed. Notable resources are marked by
editors of Citmed. There are 2 language editions (English
The history of link popularity is recorded and might be
obtained by clicking on a certain mark at every resource.
The system operates automatically, however, the editors
check and correct the retrieved values.
Fig.1. Citmed – automatic ranking system
Fig. 2. Citmed – example of one resource
Eysenbach and Diepgen compiled a comprehensive
analysis of quality management of medical information on
They discussed future of evaluation of web pages and
suggested a decentralized approach towards quality control.
They mentioned different electronic labels and apart from
other suggestions they list possible indirect quality
indicators suitable for dynamic automatic selection by
software. On the first place of the list is a „webcite index“,
analogous to the Science Citation Index, that could be
compiled from the absolute number of hyperlinks to a
certain website or new
hyperlinks established over period of time. Similarly, web
impactfactor (WIF), was previously introduced by
Ingwersen as a tool to measure impact of the web
The attraction of this parameter is the possibility of
automation of its measurement and the fact that we not only
assess the authors, content and structure of the webpage, but
we also measure the impact of the publication
on its readers, because we may expect that web masters will
only cite or ink to pages they think are relevant and of higher
value (it is similar assumption as in printed media). Likewise
printed medical journals,
medical documents on the Internet could then be ranked by
their citation analysis. Research supporting this hypothesis
was conducted by Lei Cui who proved that webcite
index of health-care related webpages correlates with their
evaluation by some of the largest rating systems on the
Citmed records link popularity of the monitored site from
the input date to the database onwards. The trend of this
value is displayed. Because of the dynamic character of the
Internet, this feature seems to have a key role in evaluation
of citation as was shown by Kasal et al., who documented
dependency of link popularity on the length of time from
date of publication on the Internet. After steep increase of
hyperlinks linked to the particular site in the first
6 months, plateau period follows as is shown in Fig 3. This
fact discriminates the new resources in favour of the older
ones. However, the later decrease of citation during a longer
time period might be expected.
Fig. 3. History of link popularity
Number of links to Human Anatomy Online (1997-2001), available at:
http://www.innerbody.com/htm/body.html . Measured with Altavista server.
Adapted from Kasal et al.
Another problem with citation analysis arises if we look at
the different types of documents. As was documented by
Kasal et al.and is shown in Fig. 4 The link popularity
differs in individual types of documents and its mutual
comparison is therefore difficult.
Fig. 4. Number of hyperlinks linked to different types of resources on
Internet. A pilot study of system Citmed. Adapted from Kasal et al.
Unclearness of methodology for site extent measuring
caused that in contrast to WIF, only pure link popularity in
the citation analysis was used and the size of the site was
ignored in the calculation. More research in this field is
necessary and we plan to focus on this issue in the future
Several resources on the Internet evaluate citation popularity
by means of links. We find this way a democratic
method for defining web resources of high quality, because
the Internet community defines the best resources by
recommendation through linking to them.
Moreover, we believe that the fact that these usage index
correlates with evaluation by third parties, qualifies it as a
However, in all cases the webcite indexing is performed
without other evaluation. The described system Citmed, list
of knowledge web medical resources, adds another criteria
in an objective evaluation of webpage quality, useful for the
selection of optimum education resources.
We would like to thank the Ministry of Health of Czech
Republic for effort they made while developing the database
environment for project Citmed.
Published criteria for evaluating health related web sites:
review. Bmj Vol. 318, pp. 647-9, 1999.
 A. Gagliardi, A.R. Jadad, Examination of
instruments used to rate quality of health information on
the internet: chronicle of a voyage with an unclear
destination. Bmj Vol. 324, pp. 569-73, 2002.
 B.F. Schloman, Quality of health information on
the Web: where are we now? Online J Issues Nurs Vol. 8,
pp. 8, 2003.
 G. Eysenbach, T.L. Diepgen, Towards quality
management of medical information on the internet:
evaluation, labelling, and filtering of information. Bmj Vol.
317, pp. 1496-500, 1998.
 P. Kasal, J.P. Naidr, M. Hladikova, J. Feberova, T.
Adla, A. Janda. The Use of Web Links for Evaluation of
Internet Education Resources. Telecommunications for
Education and Training(TET) 2001, Prague, Charles
 L. Cui, Rating health web sites using the principles
of citation analysis: a bibliometric approach. J Med Internet
Res 1, E4, 1999.
 HON Code Principles, Health On the Net
Foundation (1997), Accessed
 Hardin MD, Hardin Library for the Health
Sciences, University of Iowa (2001), Accessed on- line
[2004-01-19]. Available at:
P. Kim, T.R. Eng, M.J. Deering, A. Maxfield,
on-line [2004-01-19]. Available at:
 T. Adla, P. Kasal, A. Janda, M. Hladikova, J.P.
Naidr, J. Feberova, Weights of criteria for evaluation of
health resources on
the web. Technol Health Care 11, pp. 367-368, 2003.
 A. Janda, P. Kasal, T. Adla et al., CITMED -
searching and presenting health care related web resources
quality criteria. Technol Health Care 11, pp. 378-379, 2003.
 S. Lawrence, C.L. Giles, Searching the world wide
Web. Science 280, pp. 98-100, 1998.
 E. Garfield, Significant journals of science. Nature
Vol. 264, pp. 609-15, 1976.
 P. Ingwersen, The calculation of web impact
factors. Journal of Documentation 54, pp. 236-243,
15] T. Adla, P. Kasal, M. Hladíková, A. Janda, J.P.
Naidr, J. Feberová, P. Kubu, R. Pot??ková. Use of the
automated quality evaluation system for
the comparison of health care web pages. Mednet
2004, 9th World Congress on the Internet in Medicine,
Buenos Aires, Argentina, In: Technol Health Care.
Ranking system, Medical Matrix (2002), Accessed