Content uploaded by André Tomlin
Author content
All content in this area was uploaded by André Tomlin on Jan 04, 2016
Content may be subject to copyright.
BJPsych Advances (2016), vol. 22, 55–63 doi: 10.1192/apt.bp.113.012377
55
ARTICLE
SUMMARY
The internet provides access to what is often a
bewil dering array o f medical knowl edge on mental
health, some (but not all) of which is evidence
based. As well as information for clinicians, there
has been a dramatic increase in the variet y and
qualit y of information available for patients and
carers. In this ar ticle we discuss the advantages
and limitations of the types of information
available, with suggested sites and strategies for
assessing their relative merits.
LEARNING OBJECTIVES
• Understand the different types of e-learning
available and their relative advantages and
disadvantages
• Be aware of the need to assess the quality of
each source of e-learning and the availability of
assessment tools to do this
• Gain a working knowledge of terms used in
e-learning, their uses and applications
DECLARATION OF INTEREST
A.T. is Managing Director of Minervation Ltd.
Use of the internet has expanded exponentially over
the past decade, increasing worldwide by 806%
between 2000 and 2015. The biggest percentage
increase over that time has occurred in Africa, but
the greatest absolute numbers using the internet are
in Asia (48% of total users). By geographic region,
North America has the greatest penetrance, with
88% of the population using the internet (Internet
World Stats 2015). In the UK in 2014, 83% of
homes had internet access, 50% of adults used
internet banking and 53% accessed social media
sites (Cotton 2014); 93% now have a mobile phone
(Ofcom 2015).
The internet is a powerful tool that allows access
to multiple types of learning, of which some, but
by no means all, use evidence-based information.
The term e-learning refers to the use of internet
technologies to deliver a broad array of different
teaching tools that enhance knowledge and
performance; it is also called web-based learning,
online learning, distributed learning, computer-
assisted instruction and internet-based learning
(Ruiz 2006). Many sites are part of blended
learning, where e-learning is combined with a more
traditional face-to-face instructor-led training, for
example using an online tutorial to supplement a
lecture (see Table 1 for definitions and examples).
Using the internet to access mental health
information
The vast number of information sites on the
internet can be overwhelming for both patients and
clinicians alike. For example, a simple search via
Google (accessed April 2015) using the search terms
‘evidence-based mental health’ yielded more than
60 000 000 results. A specific clinical question such
as ‘evidence for exercise in depression’ yielded more
than 89 000 000 results. The records retrieved range
from reviews of the evidence and primary research
articles in peer-reviewed journals to newspaper
reports and advertisements for treatment centres.
So, the internet user is swamped with an array of
sites of variable quality that are neither necessarily
pertinent to the original question nor ranked in
order of reliability.
Paradoxically, this array of sites and information
means that, although clinicians have a large number
of questions regarding patient care, they pursue
answers to these questions via the internet in only
half of cases (Ely 2005). The most common reason
for clinicians not researching an answer online is
the perception that they will not be able to find it
among the multitude of possible information sites
(Ely 2005). This is confirmed, as in a quarter of
cases when clinicians do look for an answer they
cannot find it (Ely 2007).
In addition, the immediate access to knowledge
gives the often erroneous impression that the
information found is the most up-to-date available.
For example, a study in which emergency doctors
used internet search engines to answer clinical
questions showed that they found the correct
answers in only 59% of cases. However, their
confidence in the accuracy of their answers was
far higher: they believed that 92% of their answers
were reliable enough to use in patient care (Krause
2011). A key challenge, therefore, is not establishing
whether information is available via the internet,
but managing that information in a way that allows
access to high-quality, evidence-based data.
Evidence-based mental health and
e-learning: a guide for clinicians
Katharine A. Smith, André Tomlin, Andrea Cipriani & John R. Geddes
Katharine Smith is an honorary
consultant psychiatrist at the
National Institute for Health
Research (NIHR) Oxford cognitive
health Clinical Research Facility
(CRF) and the Department of
Psychiatry at Oxford University.
André Tomlin is an information
scientist with 20 years’ experience
working in evidence-based
healthcare. He has worked in the
NHS and for the University of
Oxford, where he helped to set
up the Centre for Evidence Based
Mental Health. In 2002, he became
Managing Director of an Oxford
University spin-out company called
Minervation, which designs and
builds health websites for charities
and the public sector. These include
the recently launched National
Elf Service, a series of blogs
using simple, clear and engaging
summaries of evidence-based
health research designed to help
professionals keep up to date.
Andrea Cipriani is Associate
Professor at the Department of
Psychiatry at Oxford University, an
honorary consultant psychiatrist
with the Oxford Health NHS
Foundation Trust and Editor-in-
Chief of Evidence-Based Mental
Health (http://ebmh.bmj.com). His
research focuses on the evaluation
of treatments in psychiatry, trying
to develop the methodology of
evidence synthesis to better inform
clinicians in their daily decision-
making.John Geddes is Head of
the Department of Psychiatry at
Oxford University and an honorary
consultant psychiatrist with the
Oxford Health NHS Foundation
Trust, where he provides clinical
care for patients with mood
disorders and specialises in bipolar
disorder. He is also Director of
Research and Development at the
Oxford Health NHS Foundation
Trust and Director of the NIHR
Oxford cognitive health CRF.
Correspondence Dr Katharine
Smith, NIHR Oxford cognitive health
Clinical Research Facility, Warneford
Hospital, Oxford OX3 7JX, UK. Email:
katharine.smith@psych.ox.ac.uk
BJPsych Advances (2016), vol. 22, 55–63 doi: 10.1192/apt.bp.113.012377
56
Smith et al
TABL E 1 Useful terms in e-learning
Term Description
Web 1.0 The original type of webpage
Common in the 1990s
Usually a static site with no user involvement
Web 2.0 A more dynamic user application, first described around 2004
Encourages (and relies on) user interaction and feedback to facilitate the creation of user-generated content
Responds more quickly to external events
Examples are blogs, wikis and social network sites
Blog Contraction of ‘web log’
Easy to start
Allows anyone a voice
Blogs vary in size and frequency of updating
Written by a ‘blogger’ (who is ‘blogging’)
Front page has the latest posts; old posts are archived in date order
Readers can leave comments
Blogs are highly interlinked, followed and referenced by other bloggers
Many blogs follow medical issues and can be a useful way to follow developments in a particular area, e.g. www.researchblogging.org
RSS Stands for ‘really simple syndication’
A web-feed format
Notifies subscribers when new content or blogs are published on the topic covered by the feed
Typically, users have several RSS feeds which are updated and sent frequently to an email address or RSS reader, e.g. feedly.com
Allows information on a particular subject to be sent from different sites as new information becomes available on the web
Wikis From the word for ‘hurry’ in Hawaiian
A website allowing collaborative creation
Contribution or modification can be made by anyone allowed access
Information can be wider and more detailed than is possible from a traditional authorship
Information is only as reliable as the quality of the contribution and monitoring by those using the site
Example is Wikipedia
Microblogs, tweets Consist of brief text updates, links, audio or video ‘tweets’ which can be collated into a ‘twitter feed’
Users can ‘follow’ others to form groups
Distributed to subscribers via a website
Can be used to rate online articles or to give reactions in real time to papers delivered at conferences
Allows dissemination of articles to large numbers of readers
Example microblogging site is Twitter
Twitter hashtag The symbol #, used to mark key words or topics in a tweet
Google Hangout An instant messaging and video chat platform for up to 10 people simultaneously
Social network Web-based services that enable users to communicate with each other by posting information, comments, messages, images, etc.
Their biggest impact in medicine is in facilitating the communication of patient groups
Examples are Facebook, Myspace, LinkedIn
Social bookmarks A method for web users to organise and share bookmarks of web resources
Resources themselves are not shared, only their web addresses
Open-access journal articles Some journals are fully open access (e.g. BJPsych Open and BJPsych Bulletin), some are hybrids (mix of open and subscriber access,
e.g. BJPsych and BMJ ) or delayed open-access journals (e.g. BJPsych Advances)
Open-access articles are usually funded by the authors (who in turn draw the money from their funding body)
Podcasts Audio files made available for download over the internet
Many are updated regularly and users subscribe using a program such as iTunes or via an RSS feed
Many well-established publishers of clinical evidence provide podcasts to complement their journals
MOOCs Stands for ‘massive open online courses’ (Liyanagunawardena 2014)
Recent innovative addition to online learning
Online courses with open registration generally without limits on participant umbers or prerequisites
Free of charge (some courses charge for a certificate of participation)
Start and end dates, but registration is often kept open
Webinar Web seminar – a presentation, lecture or seminar transmitted over the internet using video-conferencing software
Interactive, allowing information to be given, received and discussed
Blended learning Combination of face-to-face and digital learning
For example, the King’s THET Somaliland Partnership (www.kcl.ac.uk/lsm/research/divisions/global-health/partnerships/Somaliland/
somaliland.aspx) combines face-to-face teaching of health professionals in Somaliland with e-tutorials via Medicine Africa (www.
medicineafrica.com)
Gamification The use of game design elements and mechanics in non-game contexts, such as information and learning sites
These techniques are used to increase motivation and engagement in learning
BJPsych Advances (2016), vol. 22, 55–63 doi: 10.1192/apt.bp.113.012377 57
Evidence-based mental health and e-learning
Summary websites
Giving clinicians access to user-friendly summaries
improves clinical knowledge and reduces error
(Hoogendam 2008). The summary websites listed
in Tables 2 and 3 are suggested as a starting point
for exploration of sites of this type. The lists are
not intended to be exhaustive, as websites are
rapidly superseded by new ones, and current sites
change in content, methodological rigour and the
efficiency with which they keep up to date with
the evidence base (Banzi 2011). However, despite
these reservations, it is helpful for clinicians to have
suggestions of websites, and we have divided the
listed sites loosely into those directed primarily at
clinicians (Table 2) and those directed primarily at
the public (Table 3). Inevitably there will be overlap,
with many sites providing information for both.
Given the changing nature and quality of sites, we
would encourage all clinicians to practise ‘evidence-
based-ly’ (Cipriani 2014), by applying the tools of
evidence-based medicine to assess the quality of
sites before using them or recommending them to
patients and carers. We return to the problem of
quality assessment towards the end of our article, in
the section ‘Potential disadvantages of e-learning’.
Web-based resources can be divided by the for-
mat they use to deliver knowledge or by the primary
group for which the information has been prepared.
Different e-learning formats
Summary sites
Summary sites (webpages providing a summary
of knowledge) are the most static form of internet-
based learning about the current state of evidence
in mental health. Nevertheless, they are the most
easily and frequently accessed first port of call,
particularly for clinicians (but also patients) to up-
date themselves quickly. Systematic reviews in the
Cochrane Library, National Institute for Health and
Care Excellence (NICE) guidance and evidence-
based search engines such as the TRIP database
(see Table 2 for website addresses) provide a quick
and easy reference guide. The advantage of these
sites is that the evidence presented has been care-
fully appraised, synthesised and summarised by
qualified experts who follow rigorous methodologi-
cal guidance and make stringent efforts to avoid
bias (Higgins 2011). Some also provide plain lan-
guage summaries for the public.
Although a quick and (mainly) reliable source,
these static websites do not necessarily create
a dynamic learning environment for engaging
an audience with matters of mental health. The
evidence is presented in a concise and technical way
that can make for dense reading. Even if regularly
updated (at varying time periods depending on
the site), there is little real-time interaction or
opportunity for dynamic feedback.
Interactive summary sites
In general, learning is greatly enhanced by
interaction. For example, a systematic review
of continuing education meetings showed that
interactive workshops can result in moderately large
changes in professional practice, whereas didactic
sessions alone are unlikely to change it (Forsetlund
2009). There are a number of information resources
that offer interactive features. Examples include the
journal
Evidence-Based Mental Health
, which has
started a programme of tweet chats and Google
hangouts, the network Students 4 Best Evidence,
which regularly posts blogs and social media
activities, and the Mental Elf, via comments, social
media, note-taking, webinars and gamification.
The advantages are that the interaction involved
TABL E 2 A selection of websites that include information on evidence-based mental
health aimed at cliniciansa
Name Website
Centre for Evidence Based Mental Health www.cebmh.com
BMJ Clinical Evidence clinicalevidence.bmj.com
Cochrane Library www.cochranelibrary.com
CPD Online (Royal College of Psychiatrists) www.psychiatrycpd.org
DynaMed Plus www.dynamed.com
Evidence-Based Mental Health www.ebmh.bmj.com
Health Evidence www.healthevidence.org
Information via professional organisations:
General Medical Council (GMC) www.gmc-uk.org
British Association for Psychopharmacology (BAP) www.bap.org.uk
Royal College of Psychiatrists (RCPsych) www.rcpsych.ac.uk
Royal College of General Practitioners (RCGP) www.rcgp.org.uk
Royal College of Nursing (RCN) www.rcn.org.uk
British Psychological Society (BPS) www.bps.org.uk
British Association for Behavioural & Cognitive
Psychotherapies (BABCP)
www.babcp.com
British Association for Counselling & Psychotherapy
(BACP)
www.bacp.co.uk
King’s Fund www.kingsfund.org.uk
The Mental Elf www.nationalelfservice.net/mental-health
NHS Confederation www.nhsconfed.org/health-topics
NHS Evidence https://www.evidence.nhs.uk
National Institute for Health and Care Excellence
(NICE)
www.nice.org.uk
Students 4 Best Evidence www.students4bestevidence.net
Trainees Online (TrOn) (Royal College of Psychiatrists) www.tron.rcpsych.ac.uk
TRIP database www.tripdatabase.com
a. Websites are rapidly superseded by new ones, and current sites change in content, methodological rigour and the
efficiency with which they keep up to date. We therefore encourage all readers to appraise sites for quality before using
or recommending them.
BJPsych Advances (2016), vol. 22, 55–63 doi: 10.1192/apt.bp.113.012377
58
Smith et al
makes learning an engaging activity that can be
individualised to each learner’s needs. However, as
with all interactive modes of learning, particularly
in real time, there is limited opportunity for editing
of information: the learner’s experience relies
heavily on the expertise and skills of those who
engage in the process.
New ways to access mental health information
As internet access has dramatically increased, so
has the role of internet users. Ofcom (2015) reports
that 61% of the UK adult population use their
mobile phone to access the internet, so the digital
divide of access is fast closing. However, a new
divide is opening up between those who use the web
as consumers (i.e. finding information, using email,
online shopping) and those who are contributors (i.e.
bloggers, people actively commenting and engaging
in public debate, for example via social media).
In the past 5 years we have seen a dramatic
increase in the use of blogs and social media by
clinicians, researchers and the general public. This
democratisation of research has expanded the dis-
cussion about new evidence beyond the traditional
intellectual and literal paywalls of publishers and
academic journals. Access to full-text research
remains a barrier to many, but science bloggers now
provide an alternative way of keeping up to date
with the latest reliable evidence.
Twitter
Nowhere online is this debate more vibrant and
immediate than on Twitter, which has become the
primary social network for keeping up to date with
new mental health research. Twitter subscribers
create their own personalised feeds of information
by following organisations and individuals who
share their areas of interest. Examples include real-
time tweet chats (e.g. #WeNurses, #EBMHchat),
and live tweeting from conferences (e.g. the Royal
College of Psychiatrists’ International Congress
#RCPsychIC) and organisations with automated
feeds of publications (e.g. @OxPsychiatry).
Discussions often arise around specific themes
(e.g. using a Twitter hashtag to follow a debate at
a conference or via an online journal club) and are
usefully archived and organised for posterity.
Webinars
An alternative is the use of webinars, which involve
video conferencing via routes such as Skype, Google
Hangouts and WebEx. They are hosted by a range of
organisations, some of which are open access (such as
the Google Hangouts organised by
Evidence-Based
Mental Health
(http://blogs.bmj.com/ebmh) or the
public campfires organised by National Elf Service),
whereas others require membership or subscription
(such as the BMJ Masterclasses webinars at http://
masterclasses.bmj.com/webinars).
Gamification
Game-based learning is an accessible and engaging
method of learning. An example is the Mental Elf
(www.nationalelfservice.net/mental-health), where
members’ website activity is automatically compiled
using gamification techniques, so users can learn
about new research while progressing through an
engaging ‘game’. Educational apps are already a
popular way of providing learning to the public
(see ‘Monitoring systems and apps’ below) and can
also be useful for professionals. For example, the
Resuscitation Council (UK) in combination with
the production company UNIT9 has developed
Lifesaver (www.life-saver.org.uk), accessible via the
website or a downloadable app, for professionals and
the public to learn cardiopulmonary resuscitation
(CPR). Lifesaver is a live-action movie showing real
people in real places played like a game with different
options for what you might choose to do next. The
user learns by doing, as the video shows the positive
and negative consequences of each decision made.
TABL E 3 Some websites aimed at informing the public about evidence-based mental healtha
Name Website
*Alzheimer’s society www.alzheimers.org.uk
Anxiety UK www.anxietyuk.org.uk
Big White Wall www.bigwhitewall.com
Bipolar UK www.bipolaruk.org.uk
Carers UK www.carersuk.org
Centre for Mental Health www.centreformentalhealth.org.uk
Depression Alliance www.depressionalliance.org/information
Driving and medical conditions www.gov.uk/browse/driving/disability-health-condition
*healthtalk.org www.healthtalk.org
The Mental Elf www.nationalelfservice.net/mental-health
Mental Health Foundation www.mentalhealth.org.uk/publications
*Mind www.mind.org.uk
*NHS choices www.nhs.uk/Conditions/Pages/hub.aspx
*NHS choices: Behind the Headlines –
mental health
http://www.nhs.uk/news/pages/newsarticles.
aspx?TopicId=Mental+health
*NICE (National Institute for Health
and Care Excellence)b
www.nice.org.uk
*Rethink Mental Illness www.rethink.org
*Royal College of Psychiatrists:
Mental Health Information for all
www.rcpsych.ac.uk/expertadvice.aspx
SANE www.sane.org.uk
Time to change www.time-to-change.org.uk
Young Minds www.youngminds.org.uk
*These websites have been certified as a quality provider of health and social care information by The Information
Standard (www.england.nhs.uk/tis).
a. Websites are rapidly superseded by new ones, and current sites change in content, methodological rigour and the
efficiency with which they keep up to date. We therefore recommend that clinicians assess the quality of sites before
recommending them to patients and carers.
BJPsych Advances (2016), vol. 22, 55–63 doi: 10.1192/apt.bp.113.012377 59
Evidence-based mental health and e-learning
Information directed primarily at
practitioners/clinicians
Websites providing clinical information
Table 2 lists a selection of websites directed
primarily at clinicians. However, the accessibility
of the internet means that information is no longer
categorised purely as suitable for one group or
another (unless the website has a restricted access
to a certain group) and many of these sites are used
by the public and incorporate lay information and
summaries.
Websites aimed specifically at delivering CPD
and learning packages
A large number of websites and webpages deliver
training and learning packages. Many have been
developed to help clinicians meet their continuing
professional development (CPD) requirements –
either those set by their employer or regulatory
organisation, or those they have set themselves
for their own development. In terms of delivery of
learning, these packages vary enormously. Some
use methods similar to a traditional lecture, with
a static set of information slides and an assessment
at the end, perhaps in the form of multiple choice
questions. The other end of the spectrum is
represented by multimedia online CPD that may
incorporate video and/or audio tools and can be
enhanced by reflective learning and the facility for
the user to personalise and adapt outputs for their
own local environment. For example, the GMC’s
Learning Disabilities website (www.gmc-uk.org/
learningdisabilities) helps health professionals
to learn about good and bad practice through
interactive video tutorials and reflective learning.
The route of access to learning also varies.
Some online CPD can be accessed free, but some
is restricted to certain groups (e.g. all doctors
registered with the General Medical Council (GMC)
can access CPD via www.doctors.net, all members
and associate members of the Royal College of
Psychiatrists can access Trainees Online (TrON)
at tron.rcpsych.ac.uk). Others require an extra
subscription (e.g. the Royal College of Psychiatrists’
CPD Online modules at www.psychiatrycpd.co.uk).
Other learning is restricted by making it available
only to members of a certain National Health
Service (NHS) trust, for example by publishing
it only on that trust’s intranet, and these pages
may or may not link to a stand-alone external
internet site.
In addition, such web-based teaching is often
combined with face-to-face teaching, for example
completing online resuscitation learning before
a hands-on practical assessment (so-called
blended learning).
Online courses aimed at learning the principles of
evidence-based medicine
The advantages of e-learning, including its
flexibility and accessibility, have also been applied
to the teaching of evidence-based medicine (EBM).
Traditionally, EBM has been taught through stand-
alone courses, conferences, workshops, journal
clubs or educational meetings, but these have been
criticised as being insufficiently integrated into
daily clinical practice and clinical postgraduate
training. A systematic review showed that stand-
alone education improved basic knowledge of
EBM, but that the evidence for an improvement
in practice was lacking (Coomarasamy 2004).
Kulier and colleagues have developed a clinically
integrated e-learning course for teaching the basic
principles of EBM to postgraduates and compared
its effectiveness with that of a traditional lecture-
based course of equivalent content (Kulier 2008,
2009; Hadley 2010). They ran the course with
different groups of postgraduate trainees in a
range of European countries, including the UK,
and concluded that it was at least as effective as
traditional teaching and was less costly, easier to
update and well accepted.
Resources available to, or directed at,
patients, carers and the general public
Information websites
Individuals are increasingly making use of online
resources to support their own health, either to
supplement their knowledge about treatment or
instead of seeking help. In the UK, 43% of internet
users report having used the web to access health
information in 2014, compared with 18% in 2007
(Cotton 2014).
Research studies focusing on mental health
information sites for the public have often high-
lighted poor usability and quality (Reavley 2011).
The readability of information provided (the
complexity of its vocabulary and syntax) is often set
at a high level of difficulty, preventing a significant
proportion of patients from understanding it
(Gralton 2010), and its quality can be variable
(Ferreira-Lay 2008). However, these sites have
great potential in providing information to patients
about their diagnosis and treatment options and,
in response to the demand for more information,
an increasing number of higher-quality sites have
been developed in recent years. A selection of these
is given in Table 3. Some of the listed websites have
been approved by The Information Standard (www.
england.nhs.uk/tis), a certification scheme for
health and social care information established by the
Department of Health to help people decide which
BJPsych Advances (2016), vol. 22, 55–63 doi: 10.1192/apt.bp.113.012377
60
Smith et al
information is reliable. Organisations achieving
Information Standard certification have undergone
rigorous assessment to check that the methodology
they have used to produce their information is clear,
accurate, balanced, evidence based and up to date
(note that any user-generated information, such
as comments on a page or on services, blogs and
forums,is excluded from the scope of certification).
It is important to note that Information Standard
certification does not guarantee that every piece of
information produced by a certified organisation
is of high quality; instead it indicates that the
methodology used to produce the information
is reliable. We would encourage clinicians to
assess the quality of any sites themselves before
recommending them to patients and carers.
Websites for lay people can also have a significant
effect in informing others. The interactive nature of
some websites (e.g. via feedback) means that these
sites can help to inform professionals and policy
makers about the views of patients. For example,
healthtalk.org provides information about health
issues by sharing people’s real-life experiences. It
is aimed at the general public, but the patients’
experiences have been used in training health
professionals (www.healthtalk.org/learning-and-
teaching/health-care-students). A Day In The Life
(https://dayinthelifemh.org.uk) is a year-long
project collecting the everyday experiences of people
with mental health problems on four calendar dates
(‘snapshots’) between 2014 and 2015.
Another element has been the development of
online communities such as the Elefriends forum
run by Mind (https://elefriends.org.uk) and the Big
White Wall (www.bigwhitewall.com), which allow
members to talk online to other members and gain
support and advice.
Web-based treatment programs
Web-based programs can be used to deliver
treatments such as cognitive–behavioural therapy
to patients. A full discussion is beyond the
scope of this article, but this is an area that has
expanded exponentially in recent years. However,
despite effective interventions, the transition of
computerised interventions into care is slow,
perhaps because in general patients do not perceive
them to be as acceptable as face-to-face treatment
(Musiat 2014).
Monitoring systems and apps
The combination of smartphone technology, email
and the internet has allowed the development of
remote monitoring systems that supply day-to-
day feedback from patients to supplement the
clinical consultation and provide active evidence
for clinicians and patients to decide together on
treatment plans. An example is the True Colours
self-management system (https://oxfordhealth.
truecolours.nhs.uk/www), developed at the
University of Oxford to monitor prospectively those
with bipolar disorder. Regular prompts sent via
email or text message ask patients to rate their mood
(both depressive and manic symptoms) and answer
any individualised questions that they or their
clinicians wish to be included. Patients can also add
events such as medication changes, psychological
treatments or life events. Results are summarised in
a graph that shows the progression of symptoms and
can be reviewed by both the patient and clinician to
inform treatment decisions (Bopp 2010).
Mobile phone apps† such as MyJourney (aimed
at young people: www.sabp.nhs.uk/eiip/app) and
Moodometer (www.2gether.nhs.uk/moodometer-
app) allow the public to monitor their own mood
and gain a picture of the factors that affect it, as
well as accessing advice. ClinTouch (clintouch.
com) is an example of an app that allows users to
monitor symptoms of psychosis.
Other apps are also available. For example, Doc
Ready (www.docready.org) is a digital tool that
helps young people to prepare and make the most
of mental health-related GP visits. MOMO (www.
mindofmyown.org.uk) is an app to help young
people express their views more clearly, get involved
in meetings and make better decisions with their
social care team.
Apps vary in quality, just as websites do, and it
can be difficult to assess their relative quality beyond
star ratings on retailers’ websites and the subjective
feedback of other users. There has been increased
interest in developing instruments specifically to
classify and rate the quality of apps, and use of scales
such as the Mobile App Rating Scale (MARS) can
provide an indicator of overall quality as assessed
using the subscales of engagement, functionality,
aesthetics and information quality (Stoyanov
2015). Websites such as the NHS Choices Health
Apps Library (apps.nhs.uk) review apps for quality.
What are the advantages of e-learning?
Accessibility
A key advantage of e-learning is that it can be
accessed at any time and is immediately available to
anyone with a connection to the internet. Geography
is no barrier and, in theory, there is no physical limit
to the numbers attending a course or accessing
information (Cook 2007). This immediate access
to educational materials via e-learning is crucial,
as learning is often an unplanned experience,
prompted by clinical dilemmas in diagnosis or
treatment (Ward 2001).
†This topic is discussed by Zhang
et al (2015) Smartphone apps in
mental healthcare: the state of the
art and potential developments.
BJPsych Advances, 21: 354–358. Ed.
BJPsych Advances (2016), vol. 22, 55–63 doi: 10.1192/apt.bp.113.012377 61
Evidence-based mental health and e-learning
Transparency and democracy
E-learning can be accessed anonymously, allowing
the learning to be more transparent and democratic
than in traditional communications. It can inform
not only participants, but the wider community and
policy makers. For example, in the recent Ebola
outbreak, media coverage in the USA and the UK
was criticised for being factually misleading and
overly focused on the small number of cases outside
West Africa. A Twitter chat hosted by
The Lancet
showed that the use of social media can help to
balance the trend for biased reporting: ‘there is no
hierarchy on Twitter – anyone can join in, express
an opinion, or ask a question. All voices are equal’
(Lancet 2014). In only 1 hour, there were more than
300 000 impressions (the total number of times
tweets were viewed), and new ideas emerged from
both the experts and the audience. The separation
between scientists, health professionals and policy
makers was reduced, creating a new diverse
community where everyone had a voice and the
opportunity to contribute and rate the experience
on different parameters.
Immediacy
Online learning is instantly accessible, and the best
types of online learning are updated frequently.
Updating electronic content is easier than
updating printed material (Chu 1998): e-learning
technologies allow educators to revise their content
simply and quickly.
Feedback is a key element of learning (Forsetlund
2009). This can be synchronous in delivery, where
instructors and learners communicate in real time
(e.g. internet chat forums and instant messaging)
or asynchronous, where learners pace themselves
and feedback is delivered via email, online
bulletin boards, newsgroups or other technologies
(Ruiz 2006).
Individualising learning
On more sophisticated sites, users can individualise
their learning to suit their particular needs.
Learners can control the content, sequence, pace,
time and media used in their learning (Chodorow
1996). At a basic level, users might select only
the modules of a course that are relevant to them.
With more advanced sites, users may interact with
tutors to create their own learning programme.
This individualised approach is usually not
possible in group face-to-face teaching, where the
needs of the group take precedence. However, it is
important, because more active engagement and
personalisation of the learning experience motivates
learners to become more engaged with the content
(Clark 2002).
Standards and accountability
E-learning standardises course content and delivery,
unlike, for instance, a lecture given to separate
sections of the same course. Automated tracking
and reporting of learners’ activities reduces the
administrative burden. Outcomes assessment can
be incorporated, to determine whether learning
has occurred. Sites often incorporate automated
log-keeping and assessments providing proof of
learning for CPD logs. These can be personalised,
with individual learning aims created by the user
and reflections on the learning achieved.
Use of innovative methods
Innovative methods include the use of virtual
patients, multimedia and interactivity. For
example, the GMC’s Learning Disabilities website
helps doctors improve the care they offer people
with intellectual disabilities (www.gmc-uk.org/
learningdisabilities). The website uses a video of a
play about a young woman with Down syndrome
who faces problems accessing healthcare. The play
highlights bad practice and the potential pitfalls in
treating people with intellectual disabilities. The
website also contains best-practice examples from
the GMC guidance, alongside video interviews with
experts in the field (patients and professionals).
Potential disadvantages of e-learning
Variable quality
The quality and regulation of the different sites
varies significantly. Accessibility is an advantage,
but it can also be a disadvantage when inaccurate
or misleading information is rapidly disseminated
before it can be corrected (Cook 2007). Users are
often left to assess the quality of websites them-
selves. A number of assessment tools have been
developed and, although they vary, most cover one
or more of the following: accuracy, completeness,
readability, accountability, and design and
technical criteria (Reavley 2011). One of the most
commonly used is DISCERN (Box 1), a 16-item
checklist (Charnock 1999) designed to be used
by consumers without content expertise to assess
the quality of healthcare information. DISCERN
primarily assesses the reliability of information,
and acts as a good checklist for this, but it pays less
attention to accessibility and usability, which are
also essential elements for healthcare information.
Although applicable to the internet, it was not
specifically designed for internet-based information.
Online tools are also available, such as the LIDA
instrument, which uses a series of questions to
assess the accessibility, usability and reliability of
information presented on healthcare websites via
BJPsych Advances (2016), vol. 22, 55–63 doi: 10.1192/apt.bp.113.012377
62
Smith et al
a set of free tools to help website developers (www.
minervation.com/lida-tool). Codes of conduct have
also been developed, including the eHealth Code
of Ethics (www.ihealthcoalition.org/ehealth-code-
of-ethics) and the Health on the Net Foundation
(www.hon.ch).
It is also important to remember that the use of
evidence-based terminology (such as ‘systematic
review’, ‘randomised controlled trial’) is not a
guarantee of quality. Critical appraisal tools such
as the Critical Appraisal Skills Programme (CASP)
checklists (www.casp-uk.net) can help readers
decide how much weight to give to a particular
study or summary.
Usability
Although there is potential for innovative and
exciting learning, many websites are poorly
designed, with a static interface and little interaction.
Setting up high-quality web-based learning can
be costly of both money (e.g. funding multimedia
additions, actors simulating patients) and time (e.g.
maintaining online discussion forums). In addition,
the quality may vary. Live discussions such as
Twitter chats will only be as good as the quality of
the participants and the nominated ‘expert’. Finally,
technological problems can mean the teaching is
not available, either unintentionally, or because
many NHS trusts do not allow access to certain
sites or technology.
Conclusions
The internet has great potential as an immediately
accessible source of high-quality evidence-based
information on mental health, but there are a number
of potential pitfalls. To avoid an overwhelming
array of sites, most users turn to summary sites
for information or interaction. These sites are only
as reliable as the methods used to synthesise the
information, and so it is the responsibility of the
clinician to assess each site (and to continue to do so
as more information is added) in order to evaluate
the quality and ‘evidence-based’ credentials it
claims. Notwithstanding all the potential pitfalls,
the advantages of embracing the new digital age
in mental health are worth actively pursuing:
creative and interactive use of the internet allows
the dissemination and discussion of high-quality
information in an immediate, instantly accessible
and democratic format.
Acknowledgements
K.S. and A.C. acknowledge support from the
National Institute for Health Research (NIHR)
Oxford cognitive health Clinical Research
Facility. J.G. is an NIHR Senior Investigator. The
preparation of this article was supported by the
NIHR Collaboration for Leadership in Applied
Health Research and Care Oxford at Oxford Health
NHS Foundation Trust. The views expressed are
those of the authors and not necessarily those of
the NHS, the NIHR or the Department of Health.
References
Banzi R, Cinquini M, Liberati A, et al (2011) Speed of updating online
evidence based point of care summaries: prospective cohort analysis.
BMJ, 343: d5856.
Bopp JM, Miklowitz DJ, Goodwin GM, et al (2010) The longitudinal
course of bipolar disorder as revealed through weekly text-messaging: a
feasibility study. Bipolar Disorders, 12: 327–34.
Charnock D, Shepperd S, Needham G, et al (1999) DISCERN: an
instrument for judging the quality of written consumer health information
on treatment choices. Journal of Epidemiology and Community Health,
53: 105–11.
Chodorow S (1996) Educators must take the electronic revolution
seriously. Academic Medicine, 71: 221–6.
Chu LF, Chan BK (1998) Evolution of web site design: implications for
medical education on the internet. Computers in Biology and Medicine,
28: 459–72.
Cipriani A, Furukawa TA (2014) Advancing evidence-based practice to
improve patient care. Evidence-Based Mental Health, 17: 1–2.
Clark D (2002) Psychological myths in e-learning. Medical Teacher, 24:
598–604.
BOX 1 Summary of the DISCERN checklist
of questions to assess the qualit y and
reliability of health information
Section 1: Reliability
• Are the aims clear?
• Does it achieve its aims?
• Is it relevant?
• Is it clear what sources of information were used?
• Is it clear when the information used was produced?
• Is it balanced and unbiased?
• Does it provide details of sources of support and further
information?
• Does it refer to areas of uncertainty?
Section 2: Quality of information
• Does it describe how each treatment works?
• Does it describe the benefits of treatment?
• Does it describe the risks of treatment?
• Does it describe what would happen if no treatment is
used?
• Does it describe how treatment choices affect quality
of life?
• Is it clear that there may be more than one possible
treatment choice?
• Does it provide support or shared decision-making?
Section 3: Overall rating
(After Charnock 1999)
MCQ answers
1 a 2 c 3 e 4 c 5 a
BJPsych Advances (2016), vol. 22, 55–63 doi: 10.1192/apt.bp.113.012377 63
Evidence-based mental health and e-learning
Cook DA (2007) Web-based learning: pros, cons and controversies.
Clinical Medicine, 7: 37–42.
Coomarasamy A, Khan KS (2004) What is the evidence that postgraduate
teaching in evidence based medicine changes anything? A systematic
review. BMJ, 329: 1017.
Cotton R, Irwin J, Wilkins A, et al (2014) The Future’s Digital: Mental
Health and Technology. NHS Confederation.
Ely JW, Osheroff JA, Chambliss MA, et al (2005) Answering physicians’
clinical questions: obstacles and potential solutions. Journal of the
American Medical Informatics Association, 12: 217–24.
Ely JW, Osheroff JA, Maviglia SM, et al (2007) Patient-care questions
that physicians are unable to answer. Journal of the American Medical
Informatics Association, 14: 407–14.
Ferreira-Lay P, Miller S (2008) The quality of internet information on
depression for lay people. Psychiatric Bulletin, 32: 170–3.
Forsetlund L, Bjørndal A, Rashidian A, et al (2009) Continuing education
meetings and workshops: effects on professional practice and health care
outcomes. Cochrane Database of Systematic Reviews, 15 (2): CD003030.
Gralton E, Sher M, Lopez CD (2010) Information and readability issues for
psychiatric patients: e-learning for users. The Psychiatrist, 34: 376–80.
Hadley J, Kulier R, Zamora J, et al (2010) Effectiveness of an e-learning
course in evidence-based medicine for foundation (internship) training.
Journal of the Royal Society of Medicine, 103: 288–94.
Higgins JPT, Green S (eds) (2011) Cochrane Handbook for Systematic
Reviews of Interventions: Version 5.1.0 (updated March 2011). Cochrane
Collaboration.
Hoogen dam A, St alenhoe f AF, Robbe PF, et al (2008) A nswers to
quest ions posed duri ng daily patient c are are more like ly to be answer ed
by UpToDate than PubMed. Journal of Medical internet Research,
10: e29.
Internet World Stats (2015) Internet usage statistics. Miniwatts
Marketing Group (http://www.internetworldstats.com/stats.htm#links).
Accessed 14 Sept 2015.
Krause R, Moscati R, Halpern S, et al (2011) Can emergency medicine
residents reliably use the internet to answer clinical questions? Western
Journal of Emergency Medicine, 12: 442–7.
Kulier R, Hadley J, Weinbrenner S, et al (2008) Harmonising evidence-
based medicine teaching: a study of the outcomes of e-learning in five
European countries. BMC Medical Education, 8: 27.
Kulier R, Coppus SF, Zamora J, et al (2009) The effectiveness of a
clinically integrated e-learning course in evidence-based medicine: a
cluster randomised controlled trial. BMC Medical Education, 9: 21.
Lancet (2014) The medium and the message of Ebola. Lancet, 384: 1641.
Liyanagunawardena TR, Williams SA (2014) Massive open online courses
on health and medicine: review. Journal of Medical Internet Research,
16: e191.
Musiat P, Goldstone P, Tarrier N (2014) Understanding the acceptability of
e-mental health: attitudes and expectations towards computerised self-
help treatments for mental health problems. BMC Psychiatry, 14: 109.
Ofcom (2015) Facts & figures. Ofcom (http://media.ofcom.org.uk/facts).
Accessed 14 Sept 2015.
Reavley NJ, Jorm AF (2011) The quality of mental disorder information
websites: a review. Patient Education and Counseling, 85: e16–25.
Ruiz JG, Mintze MJ, Leipzig RM (2006) The impact of e-learning in
medical education. Academic Medicine, 81: 207–12.
Stoyanov SR, Hides L, Kavanagh DJ, et al (2015) Mobile app rating scale:
a new tool for assessing the quality of health mobile apps. JMIR mHealth
and uHealth, 3: e27.
Ward JP, Gordon J, Field MJ, et al (2001) Communication and information
technology in medical education. Lancet, 357: 792–6.
MCQs
Select the single best option for each question stem
1 Which of the following statements is true?
a giving clinicians access to user-friendly
summary sites improves clinical knowledge and
reduces error
b all ‘evidence-based’ summary sites keep up to
date with current knowledge
c DISCERN is a checklist designed only for
experienced clinicians to assess the quality of
healthcare information
d Information Standard guarantees that all the
information on a certified site is reliable
e DISCERN focuses primarily on the usability and
accessibility of healthcare websites, rather
than the reliability of information.
2 Which of the following statements is true?
a traditional teaching of evidence-based
medicine (EBM) is clearly superior to e-learning
b stand-alone education in EBM improves basic
knowledge and substantially improves practice
c MARS is an example of a mobile app rating
scale that can be used to assess the overall
quality of an app
d the presence of evidence-based terminology
such as ‘systematic review’ and ‘randomised
controlled trial’ proves that an e-learning site
is reliable
e immediate access to knowledge via the
internet gives the correct impression that
the information found is the most up to date
available.
3 As regards clinicians using the internet
to search for answers to clinical ques tions:
a it is estimated that they use the internet to
search for answers in 80% of cases
b it is estimated that when they use the internet
they find an answer in only a quarter of cases
c one study reported that the accuracy of
answers was 80%
d one study reported that their confidence in their
answers was 50%
e the most common reason clinicians give for not
using the internet was the perception that they
would not be able to find the answer.
4 As regards internet terminology:
a Web 1.0 are more dynamic user applications
than Web 2.0
b blogs, wikis and social net work sites are
examples of Web 1.0
c RSS is a type of web feed
d MOOCs are online courses, usually charging a
fee and restricted in entry requirements
e wikis are static websites written by one exper t
in the area of interest.
5 As regards e-learning:
a it can also be referred to as distributed learning
b searching the internet produces results that are
ranked in terms of reliabilit y
c communicating online (e.g. via a Twitter chat)
prioritises the voices of those with recognised
expertise over those of the public
d a disadvantage is that accurate information
can be disseminated quickly to large numbers
of people
e an advantage is that inaccurate information
can be disseminated quickly to large numbers
of people.
