Content uploaded by Anne G Ekeland
Author content
All content in this area was uploaded by Anne G Ekeland
Content may be subject to copyright.
International Journal of Technology Assessment in Health Care, 28:1 (2012), 44–51.
c
Cambridge University Press 2012
doi:10.1017/S0266462311000638
A Model for Assessment of Telemedicine
Applications: MAST
Kristian Kidholm
Odense University Hospital
kristian.kidholm@ouh.regionsyddanmark.dk
Anne Granstrøm Ekeland
Norwegian Center for Intergrated Care and Telemedicine
Lise Kvistgaard Jensen, Janne Rasmussen, Claus Duedal Pedersen
Odense University Hospital
Alison Bowes
University of Stirling
Signe Agnes Flottorp
NorwegianKnowledgeCentrefortheHealthServices
Mickael Bech
University of Southern Denmark
Objectives: Telemedicine applications could potentially solve many of the challenges faced by the healthcare sectors in Europe. However, a framework for assessment of these technologies is need by
decision makers to assist them in choosing the most efficient and cost-effective technologies. Therefore in 2009 the European Commission initiated the development of a framework for assessing
telemedicine applications, based on the users’ need for information for decision making. This article presents the Model for ASsessment of Telemedicine applications (MAST) developed in this study.
Methods: MAST was developed through workshops with users and stakeholders of telemedicine.
Results: Based on the workshops and using the EUnetHTA Core HTA Model as a starting point a three-element model was developed, including: (i) preceding considerations, (ii) multidisciplinary
assessment, and (iii) transferability assessment. In the multidisciplinary assessment, the outcomes of telemedicine applications comprise seven domains, based on the domains in the EUnetHTA model.
Conclusions: MAST provides a structure for future assessment of telemedicine applications. MAST will be tested during 2010–13 in twenty studies of telemedicine applications in nine European
countries in the EC project Renewing Health.
Keywords: Telemedicine application, HTA, EUnetHTA, Assessment model
The European Commission has expressed strong support for
telemedicine as a solution to the challenges faced by healthcare
systems with aging populations, increasing numbers of patients
with chronic conditions and decreasing supply of human re-
sources (3).
Telemedicine is here defined as the delivery of healthcare
services through the use of information and communication
technologies in a situation where the actors are at different
locations. The term telemedicine application refers to the overall
intervention or service and not just to the telemedicine device
used as part of the service.
The EC has pointed out that despite the benefits and maturity
of the technology, telemedicine services are still not widespread
in Europe. Small scale services, not integrated into local health-
care systems, dominate the scene.
Several stakeholders and experts participated in the workshops and provided continuous
feedback to our work. We have received valuable comments from Finn Børlum Kristensen, Danish
National Board of Health, Coordinating Secretariat, EUnetHTA and Catalan Agency for Health
Technology Assessment (CAHTA) in Spain. Flora Giorgio and Francisca Rosario Garcia Lizana,
European Commission Information Society & Media DG, have given helpful comments during the
process of the study. Without their contribution it would not have been possible to achieve the
results described in the article.
The project was funded by the EU under SMART 2008/0064 and was conducted as part of the
MethoTelemed study.
Similarly, several literature reviews on the effectiveness
of telemedicine demonstrate lack of evidence. Examples in-
clude a systematic review identifying 1,300 papers making
claims about telemedicine outcomes. Assessment of these
showed that only 46 reported effects on clinical outcomes
(12). Another review analyzed the suitability of telemedicine
as an alternative to face-to-face care, concluding that es-
tablishing systems for patient care using telecommunica-
tions technologies is feasible; the studies, however, pro-
vided inconclusive results regarding clinical benefits and
outcomes (6).
The quality of telemedicine studies has been questioned in
reviews and there is debate about appropriate research method-
ologies and techniques. One review found that economic anal-
ysis of telemedicine had not yet met accepted standards (23);
others suggested that few existing studies are well-designed
(13;14). Finally, one review called for the development of an
“evaluation template” (2).
Against this background, the EC decided to support the
development of guidelines for consistent assessment of
the outcomes of telemedicine (3). In 2009, the EC initiated the
MethoTelemed project, aiming to provide a structured frame-
work for assessing the effectiveness and contribution to quality
of care provided by telemedicine applications. The framework
was to be based on the users’ need for information for decision
making on whether or not to use new telemedicine applications.
44
Model for assessment of telemedicine: MAST
Simultaneously the framework was to be based on a systematic
review of relevant scientific literature.
Therefore, as part of the development of MAST, a system-
atic literature review of reviews on impacts (all types of out-
comes included in the reviews) of telemedicine services with
the purpose of identifying the “state of the art” was carried out
by Ekeland et al. (10).
Interventions included all e-health interventions; informa-
tion and communication technologies for communication in
health care; Internet-based interventions for diagnosis and treat-
ments; and social care if an important part of health care and in
collaboration with health care for patients with chronic condi-
tions. Each potentially relevant systematic review was assessed
in full text using a revised check list from EPOC (Cochrane
Effective Practice and Organization of Care Group) to assess
quality. Qualitative analysis of the included reviews was in-
formed by principles of realist review.
Reviews from 2005 and onward were included, and sev-
enteen data sources and databases of literature were searched
without language restrictions. The literature review was struc-
tured according to the PRISMA checklist. The main results from
the review of 80 systematic reviews were that despite the large
number of studies on the effects of telemedicine, high quality
evidence to inform policy decisions is still lacking. Large stud-
ies with rigorous designs are needed to improve evidence and
stronger focus is needed on economic analysis and patients’ per-
ception as well as an understanding of telemedicine as complex
development processes.
The purpose of this article is to present the Model for AS-
sessment of Telemedicine applications (MAST), developed in
the MethoTelemed study. The presentation includes description
of the methods used, the objective of the model, the elements
included and relationships to other models and frameworks.
METHODS
The MAST model was developed through user and stakeholder
workshops and on the basis of a systematic literature review
(10) as described above.
To assess the users’ and stakeholders’ needs for informa-
tion on the content and effects of telemedicine applications
in decision making, two workshops were held in 2009 at DG
Information Society and Media in Brussels. Supplementary
File 1, available at www.journals.cambridge.org/thc2012006,
includes a list of all participating stakeholders in both
workshops.
At the first workshop, twenty participants represented key
stakeholder groups (service providers, users, and policy direc-
tors). They were presented with results from the literature review
and introduced to the EUnetHTA Core Model (15;16).
The participants were divided into two groups and asked to
apply their knowledge and experiences, while setting aside their
usual professional roles. They were asked to play the roles of
members of the decision-making body of a health organization
which had been approached by a doctor with a proposal for a
telemedicine solution. The participants’ job was to determine
the type, amount and quality of information needed to decide
whether or not to implement the service. The participants were
also asked to evaluate the EUnetHTA model and the usability
of the nine domains with regard to telemedicine represented by
two cases presented at the workshop (see Supplementary File
2, available at www.journals.cambridge.org/thc2012007).
The facilitators used “brown paper” sessions for structur-
ing the group discussions, described in, for example (1). Each
participant wrote examples of information needed on self-stick
notes. These notes were then placed on nine sheets of brown
paper representing each domain in the EUnetHTA Core Model.
Next, a facilitator helped the group analyze the results and re-
group the notes. Finally the group agreed on the results, and the
outcomes were presented in a plenary session.
At the second workshop a new group of eighteen stakehold-
ers discussed and validated the results from the first workshop.
As background they had the results from the previous workshop
and the literature review. The draft model was described and the
participants were divided into two groups discussing the model
and the content of the suggested domains and recommended
methods for data collection. Finally, results were presented in
a plenary session where suggestions for improvements were
reviewed.
During the workshops, two members of the team took notes
capturing the discussion and oral comments of the participants.
Also, all the notes and papers from the brown paper sessions
were collected at the end of each workshop. Subsequently, the
team used the notes and papers to prepare a summary and col-
lectively synthesize the results. The results were presented in
a report for each workshop, which was forwarded to the re-
spective participants for comment and correction before being
finalized and used by the team for the next stage of work.
Workshop results were discussed by the project team (the
authors of this article) during video conference meetings, face-
to-face meetings and meetings with EC representatives. The
results of the literature review were also used, as the articles
identified formed the basis for describing the content of each
domain and for selection of topics and outcome measures.
RESULTS
Results from the Workshops
In the first workshop, the EUnetHTA Core Model was accepted
as a reasonable starting point, albeit needing some adjustments.
First, it was suggested that an assessment should start with
strategic consideration of the organizational level (local, re-
gional, national) at which the assessment should be carried out.
Second, the model should in addition to the EUnetHTA Model
focus on specific aspects of telemedicine including: Economic
sustainability for the institution involved; Patients’ perceptions
45 INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 28:1, 2012
Kidholm et al.
of the telemedicine application; Safety: Risk of harms, loss of
data, network problems, data safety, etc.; Effects on workflow
and co-operation between primary and secondary care; and Eth-
ical and legal aspects of telemedicine.
Examples of ethical aspects discussed included equality of
patient access to telemedicine services, for example, for frail
patients with low socio-economic status, cognitive impairment,
living in geographically remote areas; differences in patients’
ability to use telemedicine services; patients’ dignity in rela-
tion to being monitored constantly in their own environment;
as well as the patient’s right to refuse treatment which used
telemedicine.
Finally, the transferability of results from small scale as-
sessments of telemedicine was discussed and highlighted as an
important subject in new assessments.
At the second “validation workshop” the draft model was
discussed. Main comments were as follows: The purpose of
the model should be clearer; Description of the purpose of the
application should be part of the preceding considerations; All
domains needed further development and examples of outcome
measures; and Potential outcomes for the relatives of the patients
should be included.
The possibility of using the domains in the EUnetHTA Core
Model was also discussed. However, the proposed seven do-
mains were considered relevant and sufficient by the stakehold-
ers in assessment of telemedicine.
The Objective of the Assessment
Based on the workshops, the project team concluded that if the
objective of an assessment of telemedicine applications is to
describe effectiveness and contribution to quality of care and to
produce a basis for decision making, then the relevant assess-
ment framework fulfilling this objective is a multidisciplinary
process which summarizes and evaluates information about the
medical, social, economic, and ethical issues related to the use
of telemedicine in a systematic, unbiased, robust manner.
Key concepts in this statement of principle are “multidisci-
plinary” and “systematic, unbiased and robust.” The first con-
cept suggests that the assessments should include all important
outcomes of the applications for patients, clinicians, healthcare
institutions and society in general. The others suggest that as-
sessments should be based on scientific studies and methods
and on scientific criteria for quality of evidence. This statement
of principle is based on the definition of HTA in the EUnetHTA
project.
The Elements in MAST
Figure 1 presents the three elements in MAST. The assessment
must start with preceding considerations to determine whether
it is relevant for an institution to carry out the assessment.
After the preceding considerations, the multidisciplinary
assessment is carried out to describe and assess the different
outcomes of the telemedicine application. As shown in Figure 1,
the outcomes and description of patients and the telemedicine
application can be divided into seven domains based on the
EUnetHTA Core Model and results from the stakeholder work-
shops. The domains are described in Table1. Finally, in relation
to the description of the outcomes, an assessment should also
be made of the transferability of the results. Issues to consider
are described below.
Note that the term “assessment model” prescribes a struc-
ture of aspects or outcomes of telemedicine applications which
should be included in a certain order (with the preceding con-
siderations before the multidisciplinary assessment) in the as-
sessment.
Step 1: Preceding Considerations
Preceding considerations start by determining the aim of the
telemedicine application and the relevant alternatives. This ini-
tial description is used to get a first view of the characteristics
of the patients, the primary outcomes and whether compari-
son should be made with usual care, an upgraded system or a
different technology.
Second, the following conditions should be considered to
identify barriers and issues that must be addressed before a
multidisciplinary assessment of the outcomes of a telemedicine
solution should be made:
Legislation.
Before the introduction of a telemedicine service, a
provider must assess whether its implementation accords with
national and regional legislation. This may include inter alia
legislation regulating medical care provision; accreditation sys-
tems for care providers and liability for care provision.
Reimbursement.
Reimbursement is the principle by which national
or regional health authorities’ and insurance bodies pay, for ex-
ample, hospitals for their services. Many countries apply varia-
tions of the DRG-system (Diagnosis Related Groups), where
hospitals are paid fully or partially for each discharge per-
formed. In some cases, telemedicine does not change the DRG-
tariff of a service, but in other cases the DRG-tariff is reduced.
Some telemedicine services have no DRG-tariff.
This can have a significant impact on the result of the busi-
ness case in the economic analysis of telemedicine applications
and should be considered before an assessment.
Maturity.
The development of telemedicine applications takes
time. Therefore evaluation of telemedicine applications must
consider their maturity or degree of development over
time (23).
A review by Taylor (22) suggests that evaluations of
telemedicine should first try to establish its safety, that is,
whether its use may result in immediate risk or harm to pa-
tients. Only then should evaluations of feasibility or practicality
be carried out. Finally, an evaluation of the effectiveness of
telemedicine can be done to determine whether the applica-
tion is worthwhile. Thus, only in this phase of the development
INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 28:1, 2012 46
Model for assessment of telemedicine: MAST
Preceding consideration
•Purpose of the telemedicine application?
•Relevant alternatives?
•International, national, regional or local level of assessment?
•Maturity of the application?
Multidisciplinary assessment
1. Health problem and characteristics of the application
2. Safety
3. Clinical effectiveness
4. Patient perspectives
5. Economic aspects
6. Organisational aspects
7. Socio-cultural, ethical and legal aspects
Transferability
assessment
- Cross-border
- Scalability
- Generalizability
Figure 1. The elements in MAST.
of the application, studies of the outcomes can be carried out
summatively.
Similarly, Drummond et al. (7) describe frequent modifica-
tions of new devices as a general problem in the design of the
economic evaluation of medical devices. If the development of
a new telemedicine application is not in a substantial “steady-
state” period, an evaluation based on a randomized controlled
trial (RCT) is particularly problematic.
Before an assessment of the outcomes of telemedicine is
initiated it is therefore important to determine whether the ap-
plication is mature, that is, whether the application has been
used on a sufficiently large number of patients, such as in a
cohort study, for most problems to have been addressed.
If the application is still being developed and needs im-
provement, an assessment based on MAST should not be started.
Instead other kinds of assessments should be carried out, such
as formative studies conducted during the development of a
service, often more than once, for in-house staff of the service
with the intention of improving the service (20).
Number of Patients.
Implementation of telemedicine often involves
large investments in equipment and integration with other in-
formation systems. Often it is necessary to educate clinical staff
and change the organization of work. Thus, the fixed costs of
implementing telemedicine can be substantial. It is, therefore,
important that assessment of telemedicine applications includes
a sufficient number of patients to inform the economic analysis,
thereby making it possible to approximate the estimated costs
to the cost in real-life use of the technology. This means for ex-
ample that if a hospital does not have enough relevant patients
per year, cooperation with other hospitals may be necessary.
Notice that modeling approaches (e.g., a Markow model) are
also an option, but this requires information on the cost vari-
ables to include and their sensitivity toward changes in number
of patients, which is often not available.
Step 2: Multidisciplinary Assessment
As illustrated in Figure 1, multidisciplinary assessment involves
assessment of the outcomes of the specific telemedicine ap-
plication compared with one or more comparators, where the
outcomes are divided into seven domains.
In Table 1 the definition and topics of each domain are
described. Topics are defined as issues for consideration within
the domain.
When choosing between different study designs and meth-
ods for data collection within each domain, the general prin-
ciple is that designs and methods must follow the state
of the art research methods within the domains and pro-
duce valid and reliable estimates of the outcomes of the
telemedicine application. However, it should be noted that
telemedicine is a complex intervention with interacting com-
ponents, and this may necessitate adjustment of the design
of new studies (5). Reporting of results should also fol-
low general guidelines for reporting health research (21)
such as guidelines for reporting of RCTs (4) and economic
evaluation (8).
The choice of outcome measures should be based on careful
consideration of the aim of the specific application, the patient
group and the organization using the application.
Step 3: Assessment of Transferability
The transferability of results from studies of healthcare tech-
nologies from one setting to another is a general problem, as
pointed out in the description of the EUnetHTA Core model.
Special attention should be paid to this in assessments of
telemedicine applications.
One reason is that implementation of telemedicine in health-
care systems is generally a process which affects the organiza-
tion. To reach the full potential of telemedicine, adjustments
must often be made in the distribution of tasks between differ-
ent healthcare professions (task shifting) and in communication
between professionals. Similarly, interoperability—the degree
47 INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 28:1, 2012
Kidholm et al.
Table 1. The Domains in MAST
Domain Definition Topics
1. Health problem and description of the application Description of the health problem of the patients expected
to use the telemedicine application and the application
being assessed incl. description of the current use.
•Health problem
•Description of the application
•Technical characteristics
•Current use of the application
2. Safety Identification and assessment of harms •Clinical safety (patients and staff)
•Technical safety (technical reliability)
3. Clinical effectiveness Effects on the patients health •Effects on mortality
•Effects on morbidity
•Effects on health related quality of life (HRQL)
•Behavioural outcomes
•Usage of health services
4. Patient perspectives Issues related to the perception of the patient or the
relatives of the telemedicine application including the
patients and relatives acceptance of the technology.
•Satisfaction and acceptance
•Understanding of information
•Confidence in the treatment
•Ability to use the application
•Access and accessibility
•Empowerment, self-efficacy
5. Economic aspects A societal
economic evaluation
comparing a telemedicine
application with relevant alternatives in terms of costs and
consequences and a
business case
describing the
expenditures and revenues for the healthcare institutions
using the telemedicine application.
Economic evaluation:
•Amount of resources used when delivering the
application and comparators
•Prices for each resource
•Related changes in use of health care
•Clinical effectiveness
Business case:
•Expenditures per year
•Revenue per year
6. Organizational aspects Assessment of what kind of resources have to be mobilized
and organized when implementing a new technology, and
what kind of changes or consequences the use can further
produce in the organisation.
•Process
•Structure
•Culture
•Management
7. Socio-cultural, ethical, and legal aspects The socio-cultural aspects include the social-cultural arenas
where the patient lives and acts during use of the
application. The ethical analysis appraises the ethical
questions raised by the application itself and by the
consequences of implementing it or not. Legal aspects focus
on the legal obligations which must be met and any specific
legal barriers that may exist to the implementation of the
application.
•Ethical issues
•Legal issues
•Social issues
of integration with other clinical or administrative systems—is
crucial to realization of the benefits for patients and healthcare
institutions
A second reason is that the number of patients using a
telemedicine application often determines the costs per patient,
with some positive economics of scale. A third problem is that
technical infrastructures often vary between countries and this
can create substantial differences in the possibilities and costs
per patient of introducing the same service to different countries.
Therefore, new studies based on MAST should include in-
formation that could be relevant for others in an assessment
of the transferability of the results. For example, estimation of
costs of using telemedicine should be followed by an assessment
of how the cost per patient is expected to vary with the number
of patients, for example, by estimation of a cost-function.
Similarly, if an assessment is based on systematic litera-
ture review, any potential problems with validity and reliability
of the studies included must be described. A practical tool for
INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 28:1, 2012 48
Model for assessment of telemedicine: MAST
combining results from different studies is meta-analysis (9).
This approach includes statistical methods for handling differ-
ences between the interventions being studied.
If the studies included have been conducted in other
countries, other issues to consider include the possibilities for
cross border transfer of results and whether differences in,
for example, legislation, reimbursement or organization of the
healthcare sector prevent transfer of the results. The EUnetHTA
project has developed an “adaptation toolkit” with several
questions that can be answered to assess the transferability of
results from other studies within five domains (11).
DISCUSSION
MAST provides a structure for future assessment of
telemedicine applications, adapted to the users’ and stakehold-
ers’ need for information in decision making and based on the
EUnetHTA core model and a systematic literature review.
Assessment of the opinion of stakeholders on a complex
issue like the need for information as a basis for decision mak-
ing is difficult and therefore a qualitative approach like “brown
paper” sessions was used. However, the use of a small sample
size can be criticized and it would have been interesting to com-
bine the qualitative approach with a more quantitative collection
of information from each stakeholder, for example, by use of
questionnaires.
Other models and frameworks for assessment of similar
health technologies can be found in the literature. The relation
between MAST and important examples of other models are
discussed below.
EUnetHTA
The EUnetHTA Core Model is a general framework intended
to facilitate international collaboration and sharing of results
in HTAs, see Lampe et al. (15;16). The model includes nine
domains and describes several topics and issues within each
domain.
To a high degree MAST is based on HTA as described in the
EUnetHTA project. In addition to the support for this approach
from the stakeholders at the workshops, the main reasons for
using this as the starting point are as follows: HTA also aims to
produce a basis for decision making; HTA is a familiar concept
to stakeholders in the EU, national health authorities, industry,
and health professionals; and MAST can be a starting point for
a future EUnetHTA model for telemedicine and thereby prepare
the way for one common model for assessment of telemedicine
across the EU.
MAST deviates from the Core Model by placing “Current
use of the technology” and “Description and technical charac-
teristics of technology” in one domain. Similarly ethical, social,
and legal aspects are combined in the seventh domain. Finally,
the patients’ perception of the technology is a separate domain
as requested at the stakeholder workshop.
The reasons for combining these five domains into two do-
mains in MAST are, first, a wish from the stakeholders to sim-
plify the model. Second, the description of the health problem
and the technology (including the current use) constitute a com-
mon basis for the assessment of the effects of the technology.
The ethical, social, and legal aspects of telemedicine include
broader societal aspects of telemedicine and the information
basis is typically from experts or legal documents and not from
data collection at patient level. The division of outcomes into
domains can always be discussed, but the seven domains in
MAST were chosen on the basis of the stakeholders requests
at the workshops. The ongoing empirical test of MAST (see
below) will show the general usability of the model in practice.
INAHTA Approach
As part of an INAHTA project Ohinmaa et al. (18) describes
an approach to assessment of telemedicine, which includes a
business case and subsequent evaluation of technical aspects,
effectiveness, user assessment, economic evaluation and sen-
sitivity analysis. Thus, this approach has a special focus on
the business case. Generally the topics mentioned are included
in the MAST domains. The problems with generalizability of
results from local studies are also included in both approaches.
National Telehealth Outcome Indicator Project
This Canadian project aimed to produce national guidance on
specific outcome indicators for evaluating telehealth applica-
tions. The guidance, described by Scott et al. (19), was based
on a systematic literature review and a workshop with national
experts and describes the development of 34 approved outcome
indicators divided into four categories: quality, access, accept-
ability and costs. Although the division of outcomes is different,
several outcomes are included in the domains in MAST. The
outcome “access,” constitutes a theme in itself in the guidance,
whereas in MAST an improvement in patients’ access to treat-
ment can be included as a clinical effect, an economic effect
caused by increase in the number of users, a reduction in trans-
portation costs or increased patient satisfaction.
Empirical Test of MAST
Empirical tests of the usability of MAST are needed to ensure
its usefulness. An empirical test will be carried out in 2010–
13 in the Renewing Health project initiated by the European
Commission, see www.renewinghealth.eu.
In this project, assessment will be made of telemedicine
applications for patients with diabetes, heart failure, and COPD
in nine EU countries. Based on the project an evaluation will
be made of the model and necessary adjustments. Criteria for
success are that an assessment based on MAST can be car-
ried out within a reasonable time frame and resources and that
stakeholders find the results sufficient for decision making.
49 INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 28:1, 2012
Kidholm et al.
CONCLUSION
The overall aim of the MethoTelemed project was to provide a
structured framework for assessing the effectiveness and con-
tribution to quality of care of telemedicine applications, based
on need for information by users and stakeholders in decision
making. On the basis of the EUnetHTA Core Model we have
produced a draft model.
We hope that the model will form the basis for future deci-
sions on whether or not to implement telemedicine services in
healthcare systems and that MAST can help decision makers in
choosing the most efficient technologies to be used in the most
cost-effective way.
Similarly, private and public manufacturers of telemedicine
can use MAST as a structure for data collection and descrip-
tion of the outcomes of their applications for patients, hospitals
etc. They can also find help in the MAST manual, for example,
examples of outcome measures within each domain and meth-
ods for data collection (17). Several practical guides for data
collection are also available, see www.renewinghealth.eu.
SUPPLEMENTARY MATERIAL
Supplementary File 1
www.journals.cambridge.org/thc2012006
Supplementary Files 2
www.journals.cambridge.org/thc2012007
CONFLICT OF INTEREST
The project was funded by the EU under SMART 2008/0064 as
part of the MethoTelemed study. Kristian Kidholm, Anne Eke-
land, Lise Kvistgaard Jensen, Janne Rasmussen, Claus Duedal
Pedersen, Alison Bowes, and Signe Flottorp have received
grants and travel support to their institutes from Commission
Information Society & Media DG; Alison Bowes has also re-
ceived grant and travel support to her institute from the Uni-
versity Hospital of North Norway, and Mickael Bech reports
having no potential conflicts of interest.
CONTACT INFORMATION
Kristian Kidholm, PhD, HTA Consultant, Department of Re-
search and HTA, Odense University Hospital, Odense, Denmark
Anne Granstrøm Ekeland, PhD, Special Consultant and
Researcher, Norwegian Center for Integrated Care and
Telemedicine, Tromsø, Norway
Lise Kvistgaard Jensen, MSc, Coordinator, Department of Re-
search and HTA, Janne Rasmussen, MA, Project Manager,
Claus Duedal Pedersen, MSc, Chief Consultant, Center for Clin-
ical Innovation, Odense University Hospital, Odense, Denmark
Alison Bowes, BA, PhD, Professor, School of Applied Social
Science, University of Stirling, Stirling, Scotland, United
Kingdom
Signe Agnes Flottorp, MD, PhD, Professor, Norwegian Knowl-
edge Centre for the Health Services, Oslo, Norway
Mickael Bech, PhD, Professor, Institute of Public Health, Uni-
versity of Southern Denmark, Odense, Denmark
REFERENCES
1. Arlbjørn JS. Process optimization with simple means: The power of
visualization. Ind Commer Train. 2011;43:151-159.
2. Barlow J. Building an evidence base for successful telecare implemen-
tation – updated report of the Evidence Working Group of the Telecare
Policy Collaborative chaired by James Barlow—November 2006.
http://www.ssiacymru.org.uk/media/pdf/f/4/APPENDIX_B_CSIP_
Telecare.pdf. (accessed August 1, 2011)
3. Commission Communication: Telemedicine for the benefit of pa-
tients, healthcare systems and societies. COM/2008/689 final. http://
eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2008:0689:
FIN:EN:PDF. (accessed August 1, 2011)
4. CONSORT statement for RCTs. http://www.consort-statement.org/.
5. Craig P, Dieppe P, Macintyre S, Mitchie S, Nazareth I, Mark P. Devel-
oping and evaluating complex interventions: The new Medical Research
Council guidance. BMJ. 2008;33:979-983.
6. Currell R, Urquhart C, Wainwright P, et al. Telemedicine versus face to
face patient care: Effects on professional practice and health outcomes.
Cochrane Database Syst Rev. 2000;2:CD002098.
7. Drummond M, Griffin A, Tarricone R. Economic evaluation for
devices and drugs – same or different? Value Health. 2009;12:402-
404.
8. Drummond M, Manca A, Sculpher M. Increasing the generalizability
of economic evaluations: Recommendations for the design, analysis,
and reporting of studies. Int J Technol Assess Health Care. 2005;21:
165-171.
9. Egger M, Smith GD, Altman D, eds. Systematic reviews in health care.
Meta-analysis in context. London: BMJ Books; 2001.
10. Ekeland AG, Bowes A, Flottorp S. Effectiveness of telemedicine:
A systematic review of reviews. Int J Med Inform 2010;79:736-
771.
11. EUnetHTA. HTA Adaptation toolkit – work package 5. October 2008.
http://www.eunethta.eu/upload/WP5/EUnetHTA_HTA_Adaptation_
Toolkit_October08.pdf. (accessed August 1, 2011)
12. Hailey D, Roine R, Ohinmaa A. Systematic review of evidence for the
benefits of telemedicine. J Telemed Telecare. 2002;8(Suppl 1):1-7.
13. Hersh WR, Helfand M, Wallace J, et al. Clinical outcomes resulting from
telemedicine intervention: A systematic review. BMC Med Inform Decis
Mak. 2001;1:5.
14. Hersch WR, Hickam DH, Severance SM, Dana TL, Krages KP, Helfand
M. Telemedicine for the Medicare population: Update. 2006 Evidence Re-
port/Technology Assessment No. 131 (Prepared by the Oregon Evidence-
based Practice Center under Contract No. 290-02-0024.) AHRQ Publica-
tion No. 06-E007. Rockville, MD: Agency for Healthcare Research and
Quality.
15. Lampe K, M ¨
akel¨
a M, Garrido MV, et al. The HTA Core Model: A novel
method for producing and reporting health technology assessments. Int J
Technol Assess Health Care. 2009;25:9-20.
16. Lampe K, Anttila H, Pasternack I. HTA core model hand-
book. https://fio.stakes.fi/htacore/handbook.html (accessed February 18,
2009).
17. MAST-Manual. http://www.renewinghealth.eu/project-overview/
overview/assessment-method. (accessed August 1, 2011)
18. Ohinmaa A, Hailey D, Roine R. Elements for assessment of telemedicine
applications. Int J Technol Assess Health Care. 2001;17:190-
202.
19. Scott RE, McCarthy FG, Jennett PA, et al. Telehealth outcomes: A syn-
thesis of the literature and recommendations for outcome indicators. J
Telemed Telecare. 2007;13(Suppl 2):1-38.
INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 28:1, 2012 50
Model for assessment of telemedicine: MAST
20. Scriven M. Evaluation thesaurus. Thousand Oaks, CA: Sage Publications;
1991.
21. Simera I, Moher D, Hirst A, et al. Transparent and accurate
reporting increases reliability, utility, and impact of your research:
Reporting guidelines and the EQUATOR Network. BMC Med. 2010;8:
24.
22. Taylor P. Evaluating telemedicine systems and services. J Telemed Tele-
care. 2005;11:167-177.
23. The Lewin Group, Inc. Assessment of approaches to evaluating
telemedicine. Prepared for: Office of the Assistant Secretary for Planning
and Evaluation, Department of Health and Human Services. Contract
Number: HHS-10-97-0012, 2000.
24. Whitten PS, Mair FS, Haycox A, May CR, Williams TL,
Hellmich S. Systematic review of cost effectiveness stud-
ies of telemedicine interventions. Br Med J. 2002;324:1434-
1437.
51 INTL. J. OF TECHNOLOGY ASSESSMENT IN HEALTH CARE 28:1, 2012
