Information and its handling and transmission form an essential part of health care and are reflected in professional standards. Automated information systems in health care—health informatics services—will improve these functions and bring new opportunities through the harnessing of modern information and communications technologies. Thus, computer support is now essential in many parts of medicine, the US Institute of Medicine has long espoused the value of computerised patient records,1 and many countries have developed strategies on this topic, and there are countless health related internet sites.However, as new information and communication technologies in health bring new opportunities, they also bring new risks. Emphasis has rightly been placed on ensuring appropriate levels of confidentiality in electronic information systems—to the point that the highly exacting requirements being demanded by independent commentators and professional bodies2 are difficult to satisfy without jeopardising the functioning of core services 3 4 or the interests of the most vulnerable groups.5 In contrast, much less thought has been given so far to ensuring the appropriateness of the design and integrity of functioning of health informatics services.
Like drugs 40 years ago, products in health informatics are unregulated with regard to safety and efficacy
A European project has now recommended ways of accrediting healthcare related software, telemedicine, and internet sites
A scheme like CE marking of electrical goods is recommended for software, national regulatory bodies should be identified for telemedicine, and a European certification of integrity scheme developed for websites
Importance of quality assurance of health informatics systems
If informatics systems are increasingly essential in the delivery of health care then their integrity and quality must be of equal importance, but this has been scarcely recognised to date. In 1963 the then UK secretary of state for health stated to the House of Commons: “The House and the public suddenly woke up to the fact that any … manufacturer could market any product, however inadequately t sted, however dangerous, without having to satisfy any independent body as to its efficacy and safety and the public was almost uniquely unprotected in this respect.”6 That statement related to drugs, being triggered by the thalidomide disaster, and the situation was changed rapidly. However, the same situation applies today with regard to electronic health informatics products and services, which are now the most important unregulated healthcare resource—in sharp contrast to drugs, medical devices, and licensed health professionals.
When errors and failures have occurred it has generally been in the interests of suppliers, provider organisations, and clinicians to quietly rectify or remove the flawed systems rather than draw attention to them. This, however, allows for unidentified and thus unquantified errors to be dispersed, with potential risk to patient health. Box 1 gives published examples of such health threatening errors in computer software. In a modern consumerist environment, however, this situation is unacceptable, as shown by the public furore over the software that miscalculated the risk of Down's syndrome in pregnancies.10
Box 1: Examples of health-threatening software errors
Errors in updated embedded clinical coding software giving false plain language representation of diagnoses, United Kingdom7Errors in reference database calculation of Down's syndrome screening, giving false negatives8Age cohort of women omitted from call up for cervical screening, Grampian Region, Scotland9Error in software calculating risk of Down's syndrome led to falsely low calculation of risk for 150 women, Sheffield10
The TEAC-Health project
Recently, a European project—towards European accreditation and certification of telematics services in health (TEAC-Health)—was conducted to investigate the issues, and we report its core findings here. The findings outlined in the project report11 have recently been formally accepted by the European Commission, which intends to examine in detail the steps required for their implementation (Jean-Claude Healy, head of health applications unit, Information Society Directorate-General, personal communication, 2000).
The project arose from an expert conference at Turku University in 1997 organised by JF, which resulted in several published articles.12–14 The work of the project was undertaken by representatives of five European countries; details of the membership and working reports can be found on the Multimedica website.15
Classification of health informatics services
For the project, we classified health informatics services into three categories—software and related services, telemedicine, and internet sites. Although many services combine more than one of these elements, the quality assurance and regulatory components for each need to be considered separately as the issues are quite distinct. We also felt it inappropriate to consider processes of quality assurance and verification solely in the health sector and therefore looked at commercial approaches such as regulation in the financial sector and at other areas of public risk such as air traffic control and food safety.
In the health sector, precedents have been set in the regulation of drugs and medical devices, but neither of these is directly applicable to health informatics services. Safety control of new drugs now depends largely on controlled trials, which are neither feasible nor affordable as a mandatory control for clinical software or internet sites. Regulation of medical devices has several similarities, but key differences are the much wider range of user proficiency and circumstances of use of informatics systems compared with medical devices and the difficulties of ensuring structured user training and education.
A taxonomy of risk assessment
We next considered how best to categorise risk in health informatics services, as it is only by identifying risk that appropriate control methods can be identified. For medical devices, the regulations are clear and helpful: they require that a device's manufacturer or supplier identifies the risk level as determined by the type of product and how life critical are the circumstances of its use.16 We concluded that risk in health informatics services depends on a combination of type of user, circumstances of use, type of use, and nature of the system. For example, a failure in an automated appointments system can have serious consequences by passing undetected, whereas an experienced clinician may filter out spurious results from a diagnostic support tool used merely as an aide memoire. The table shows the different levels of risk associated with different health informatics services.
View this table:View PopupView InlineLevels of risk in the use of health telematic systems
Quantification of the problem
We sought to identify and quantify the risks attributable to informatics services, and the degree of concern they produced. A comprehensive literature search and a small targeted survey of European opinion leaders from health and consumer domains showed that the problem was, if anything, greater than anticipated.15
Many of the problems identified when using clinical software are resolved between supplier and user on condition that there is no publicity, while the problems that are not identified cannot, by definition, be reported. Thus the literature will substantially underestimate these problems, but some errors have been reported (see box 1), as has the adverse outcome of software upgrades producing erroneous printed interpretations of previously recorded diagnostic data.17
Less has been published about the risks of telemedicine services because of their comparative newness. However, we identified concerns about authenticity and risks in telemedicine services, including email consultations, other than those within a single provider organisation or on a closed, point to point basis.15 There are indications that a quarter of those offering telemedicine consultations directly to the general public do not hold the qualifications they claim (S Schanz, personal communication, 2000), and others may be offering advice beyond their qualifications. Studies have shown there is wide variation in the quality of advice provided, and, although guidance may generally be sound, the occurrence of so many outliers is an unacceptable and avoidable risk. 18 19
Services on the world wide web are the most obvious risk, as anyone can publish any information they like. Much of this information is valuable and the internet allows freedom of expression for patient support groups and leaders in alternative therapies, but studies have shown that both misleading and life threatening advice is readily available. 20 21 A figure of 1400 “suspicious” websites was reported by the coordinator of a study for the G8 group of countries, with a 21% increase in that number annually,22 and a recent US study found errors and contradictions even within sites.23 Yet, by its very nature, the internet cannot be controlled or censored.
Our survey of opinion leaders, for which we used a “snowball sample” method, yielded 54 respondents, of whom 36 (67%) indicated that they had experienced one or more problems with health telematics services. Of the 74 problems reported, 10 adversely affected patient safety, four adversely affected optimum treatment of a patient, and 31 adversely affected the health professional's duty of care to a patient. Of all the respondents, 19 were “very concerned” about the current lack of quality assurance of telematics services and a further 22 had some concerns, giving a total of 41 (76%) “concerned.”
TEAC-Health recommendations for clinical software
In view of the need to avoid identified risks to the public, and the professional opinion in favour of some form of regulation, we concluded that specially crafted regulation was needed based on existing European experience with product control and monitoring health risks. The components suggested are as follows.
Applying this publicly understood and reliable mark on approved goods is a well established process in Europe based on clear regulation24 and with variants for medical devices.16 However, further research is needed on the specific criteria to accommodate clinical software. This will require a “notified body” to have overall responsibility and to identify and monitor essential requirements for these products and services. As concurrent verification of design and quality is far more effective than retrospective testing, the necessary identification of control measures for production and quality assurance will itself yield invaluable standards for clinical software developers.
A legally underpinned requirement for accurate and detailed labelling is a key element of our proposed solution, as this will enable purchasing organisations and clinical users to know much more about the software product. Identification of named responsible individuals will also substantially increase the commitment to ensure quality of design and manufacture. The exact requirements will need further discussion and definition, but box 2 shows a suggested list.
Box 2: Suggested labelling requirements for clinical software
Country of originIdentity of legal person or company responsibleIntended purpose (such as clinical advice, decision support, prescribing advice)Competence of intended end user (such as general practitioner, endocrinology specialist, triage nurse)Assumed knowledge of user (such as specific clinical qualification)Identity and registration body of health professional responsible for supervising the clinical element of the designKey sources of clinical logic or knowledge (such as citation of published material, authorship of in house clinical design)Extent of previous use or in house testing of this version“Hotline” telephone number for postmarketing surveillance
RETURN TO TEXT
“Hotline” for postmarketing surveillance
An essential part of CE marking is postmarketing surveillance, in particular the requirement that the supplier provides a “hotline” telephone number to which any problem or concern can be reported. It is also a statutory requirement of CE marking that all serious incidents are reported by the supplier to a “competent authority,” and this process is liable to unannounced audit on site.
National hotlines and monitoring organisations
Based broadly on existing models for drug products and medical devices, national hotlines and monitoring organisations are necessary for clinical software to ensure that problems such as adverse interactions between different products (see box 1) can be identified speedily. They are of proved benefit for other clinical products and already apply to health software in Sweden.
In house software and informatics services
Software and services developed by particular healthcare organisations for their own use cannot readily be subjected to compulsory CE marking as they are not marketed products. However, our proposed regulation would bring two safeguards. Firstly, the identification of professional standards would form a yardstick for identifying reasonable practice and duty of care should there be a formal complaint or litigation. Secondly, in house products could be submitted voluntarily to the verification process.
TEAC-Health recommendations for telemedicine
Telemedicine presents an entirely different situation because telecommunications based services that cross legislative boundaries are almost free of regulation. Thus, providers of healthcare services could escape regulation, particularly when moving to the internet. Since this leaves patients at risk, some control mechanisms are needed. In principle, legislation should be independent of the communication medium used—namely, the same ethical principles and liabilities should apply to telemedicine as to conventional patient care. Because telemedicine services can readily cross international boundaries, international coordination or coregulation is needed in Europe and beyond. Similarly, in countries such as the United States regulation is at the state level, leading to complex and unwieldy situations that hamper legitimate national providers and thus also patients.
Key elements of regulating telemedicine services should include international agreement as to whether such services are delivered under the law of the supplier or that of the consumer. The European Permanent Committee of Physicians (EPCP) now favours accepting European law that it is the supplier's legal system that applies (Ä Markku, chairman, EPCP, personal communication, 2000). Secondly, labelling (as above) with legal sanctions should be required, linked to a code of conduct, which needs to be developed. Box 3 shows proposed key elements.
Box 3 : Key elements of proposed labelling requirements and code of conduct for telemedicine
Healthcare professionals should state their full name and qualificationsThe professional body responsible for monitoring clinical practice must be identifiedRecords must be kept to an agreed standard, with the database maintained and protected according to European standards for data protectionTelemedicine traffic should be strongly encryptedTelemedicine service providers should be required to register with a national agency for the provision of the services, related to international standards and qualifications which need to be developedServices should be provided in accordance with stated technical standards (including those for equipment, telecommunication, and data interchange) together with stated practice standards (such as for image labelling and agreed terminology)
RETURN TO TEXT
A global regulatory framework is also important. There are clear and effective global conventions and supervisory organisations for both civil aviation and food standards, both of which operate on an evidence based principle, obtaining and interpreting emergent scientific evidence in order to formulate new standards that then become the basis for universally agreed international regulation. Delivery of telemedicine services internationally puts individual patients at risk of injury or death through incompetent or malicious unregulated providers, but, because the transactions are individual and confidential, adverse outcomes are not as conspicuous as in domains such as civil aviation. The same situation applied to pharmaceutical products until regulation.6 The global risk to personal health continues unabated in the absence of international agreement on regulation, liability, and control. We consider international telemedicine to deserve at least the same level of regulation as the civil aviation and food sectors. This could also aid the development of national frameworks, especially in countries with largely independent states or provinces.
TEAC-Health recommendations for internet sites
We believe that the cost of developing a system solely to verify the quality of health internet sites would be high and that it would be impractical. The Health on the Net Foundation (HON) has for some time been promoting a voluntary code of conduct, and there have been several overlapping initiatives in the United States (see box 4), but their main drawback is that there is no external verification and so the system is open to abuse and, indeed, offers false security.
Box 4: Voluntary initiatives for codes of conduct for health internet sites
Health on the Net Foundation (HON). www.hon.chSwiss based organisation, European focusInternet Health Coalition. www.ihealthcoalition.comUS based organisation, European inputAmerican Medical AssociationPrinciples governing AMA publications websites. pubs.ama-assn.org/ama_web.htmlJAMA special communication. Guidelines for medical and health information sites on the internet. jama.ama-assn.org/issues/v283n12/ffull/jsc00054.htmlHealth Internet Ethics. www.hiethics.comUS based organisationQuackwatch—Your Guide to Health Fraud, Quackery, and Intelligent Decisions. www.quackwatch.comUS based initiative
All sites accessed 20 June 2001
RETURN TO TEXT
However, the need for independently verified sites is common to many other internet activities, including retailing.25 As with CE marking and other recognised quality standards, the power of effective regulation depends on the universality of use leading to public recognition. We studied earlier attempts to identify high quality sites to the public, the best known being filtering mechanisms and rating systems.11 Both have drawbacks.
Most filtering excludes inappropriate items but also excludes many relevant sites, as it is difficult to develop a 100% specific yet sensitive filter that does not filter out required material. For example, a filter designed to protect against pornography will exclude sites with the word “breast,” but it will also filter out important medical sites. Such “heuristic” filtering depends on finding and interpreting key words. The alternative, “filtering in,” requires the site to undertake self rating honestly and accurately.
Rating systems depend on third parties such as informed users to provide a rating and score for each individual site, but this raises questions of ensuring objectivity, impartiality, and common clinical and cultural values to the extent that there are now proposals for rating the raters. Moreover, this leaves most sites unrated. Clearly, these methods are not feasible to aid general public users, nor indeed most health professional users unfamiliar with the intricacies of the internet. Box 5 summarises the issues.
Box 5: Impediments to voluntary quality assurance for websites
No closed industrial or commercial groupingVoluntary initiatives may reflect sponsors' interests or valuesEnforcement and sanctions are difficult to applyConsumer confusion with numerous initiatives
UndiscriminatingMay exclude relevant sites
Requires major expert resourcesImposes values of ratersSlow to cover new sitesSites can change rapidly after rating
Monitoring or reporting apparently adverse sites
Cannot be comprehensiveBased on personal values
Allows inaccurate (and malevolent) sites to remain unchallengedConsumers continue to be at risk
RETURN TO TEXT
The EuroSeal proposal
We have therefore proposed development of a new European system and standard, entitled the EuroSeal. 12 15 This would be a seal supplied to a website by an accredited agency (the approach fundamental to CE marking). Once attached to the site, its integrity would be verified by secure single socket layer or similar secure software, as currently happens with secure trading sites. The seal would be provided at two levels, the higher of which would require independent onsite verification (for a higher fee). The verification processes would be open and transparent—by clicking on the EuroSeal symbol, visitors to the site would see details of the site inspections, drawn in real time from the records of the accrediting body (as applies with current secure links for web commerce), as well as the code(s) of conduct to which the site adhered.
Codes of conduct
These are an important element of the EuroSeal approach, as they would form the basis on which the third party assessed a site provider's claims and decided whether to award the EuroSeal. Each health professional body would be able to devise its own codes of conduct and standards, and viewers would know against which code the EuroSeal had been applied. This approach would also allow special interest groups—such as ethnic groups, those with particular religious beliefs, and advocates of alternative medicine—to devise their own codes of conduct. Patient support groups could also devise codes of conduct, provided they met a prescribed framework and standard for codes.
Thus, the EuroSeal approach would not only provide a simple, clear, and universal public safeguard without seeking censorship but would also be socially progressive, enabling positive support and selection for special interest groups and minorities. As a mark of high integrity, it would be sought after by sites and looked for by search or filter by viewers.
Health informatics systems are invaluable to aid health care. Moreover, they bring intrinsic advantages, such as electronic records being more accessible than paper ones and, if properly protected and encrypted, being more secure from damage or prying. However, this is no excuse not to address current known and avoidable risks.
The TEAC-Health project has clearly shown that public safety and professional integrity are threatened by the lack of regulation of health informatics services. These risks will increase rapidly as health informatics services expand and as telecommunications and globalisation radically change attitudes to and delivery of health care. 26 27 Initiatives to date have been based on restricted research, lacked consideration of overall feasibility and other issues, or depend on the (usually unpublished) integrity and values of a secondary service provider. The strategic proposals we describe, which have now been welcomed by the European Commission, form an evidence based solution.
Competing interests JW has a small part of the equity of Medix, an internet service provider for doctors, and receives research and consultancy funding from various commercial sources.
References1.↵Dick RS, Steen EB eds.The computer-based patient record—an essential technology for health care.Washington DC: National Academy Press,1991.2.↵Anderson RJ.Security in clinical information systems.London: British Medical Association,1996.3.↵Anderson RRigby M.Keeping confidence in confidentiality: linking ethics, efficiency, and opportunity in health care computing—a case study. In: Anderson R ed.Personal medical information—security, engineering, and ethics; personal information workshop, Cambridge, UK, June 21–22, 1996 proceedings.Berlin: Springer-Verlag,1997:129–150.4.↵Anderson RRoberts R, Thomas J, Rigby M, Williams J.Practical protection of confidentiality in acute care. In: Anderson R ed.Personal medical information—security, engineering, and ethics; personal information workshop, Cambridge, UK, June 21–22, 1996 proceedings.Berlin: Springer-Verlag,1997:67–78.5.↵Cesnik B, McCray AT, Scherrer J-RRigby M, Hamilton R, Draper R.Towards an ethical protocol in mental health informatics. In: Cesnik B, McCray AT, Scherrer J-R eds.Medinfo 98 9th world congress in medical informatics, proceedings.Amsterdam: IOS Press,1998:1223–1227.6.↵House of Commons official report (Hansard). Session 1962–6., 1963 May 8.London: HMSO,1963.7.↵Hawking M.Code conversions, data stability, and the future—an agenda for discussion.J Inf Primary Care1995;June:3–5.8.↵Cavalli P.False-negative results in Down's syndrome screening.Lancet1996;347:965–966.OpenUrlMedlineWeb of Science9.↵Computer error leads to smear recalls failure.Health Serv J1998;106: 6.OpenUrl10.↵Wilkinson P.Down's test leaves 150 women in abortion fear.Times,2000 May31: 1, 3.11.↵Forsström J, Rigby M, Roberts R, Nilsson S, Wyatt J, Beier B, et al.Towards evaluation and certification of telematics services for health (TEAC-Health)—key recommendations.Turku: University of Turku,1999.12.↵Forsström J.Why certification of medical software would be useful?Int J Med Inf1997;47:143–152.OpenUrlMedline13.Wyatt J.Quantitative evaluation of clinical software, exemplified by decision support systems.Int J Med Inf1997;47:165–173.OpenUrlMedline14.↵Forsström J, Rigby M.Considerations on the quality of medical software and information services.Int J Med Inf1999;56: 1–3,169–76.15.↵Multimedica.Towards evaluation and certification of healthcare applications in Europe. (accessed 10 Aug 2001)16.↵EU Council.Directive 93/42/EEC concerning medical devices.Brussels: European Commission,1993.17.↵Anderson RHawking M.Organisation of general practice: implications of IM&T in the NHS. In: Anderson R ed.Personal medical information—security, engineering, and ethics; personal information workshop, Cambridge, UK, June 21–22, 1996 proceedings.Berlin: Springer-Verlag,1997:56–65.18.↵Eysenbach G, Diepgen TL.Responses to unsolicited patient e-mail requests for medical advice on the world wide web.JAMA1998;280: 15,1333–5.OpenUrl19.↵Sandvik H.Health information and interaction on the internet: a survey of female urinary incontinence.BMJ1999;319:29–32.OpenUrlFREE Full Text20.↵Impiccatore P, Pandolfini C, Casella N, Bonati M.Reliability of health information for the public on the world wide web: systemic survey of advice on managing fever in children at home.BMJ1997;314:1875–1879.OpenUrlFREE Full Text21.↵Weisbord SD, Soule JB, Kimmel PL.Poison on line—acute renal failure caused by oil of wormwood purchased through the internet.N Engl J Med1997;337:825–827.OpenUrlCrossRefMedlineWeb of Science22.↵Rogers R.A global information society for health—recommendations for international action.Br J Healthcare Computing Information Manage1999;16:28–30.OpenUrl23.↵Berland GK.Health information on the internet: accessibility, quality, and readability in English and Spanish.JAMA2001;285:2612–2621.OpenUrlFREE Full Text24.↵Council of the European Communities.Decision of 22 July 1993 concerning the modules for the various phases of the conformity assessment procedures and the rules for the affixing and use of the CE conformity marking, which are intended to be used in the technical harmonization directives.Brussels: European Commission,1993.25.↵Institute of Chartered Accountants in England and Wales.International chartered accountancy bodies launch webtrust—a worldwide web assurance service. (accessed 15 Aug 2001).26.↵Rigby M.The management and policy challenges of the globalisation effect of informatics and telemedicine.Health Policy1999;46:97–103OpenUrlCrossRefMedlineWeb of Science27.↵Rigby M, Roberts R, Thick MRigby M.And into the 21st century—telecommunications and the global clinic. In: Rigby M, Roberts R, Thick M eds.Taking health telematics into the 21st century.Oxford: Radcliffe Medical Press,2000