ArticlePDF Available

Digital NHS Wales: a coding reliability analysis based on the voices of 22 978 patients and clinicians on the benefits, challenges and sustainability of video consulting



Introduction: The use of video consulting (VC) in Wales UK has expanded rapidly. Previous VC evidence has been the subject of small-scale projects and evaluations. Technology Enabled Care Cymru is an all-Wales digital service and rolls out digital interventions and evaluates on large scales, thus capturing representative data sets across Wales, and therefore a wide range of National Health Service (NHS) specialties. Objective: To extract and analyse narrative feedback from patients and clinicians using the NHS Wales VC Service for 6 months (September 2020 to March 2021). Design: A coding reliability approach of a cross-sectional study was conducted. Setting: From all health boards across Wales. Participants: NHS patients and clinicians across primary, secondary and community care settings in Wales. Results: Data were captured on benefits, challenges and sustainability of VC. A coding reliability analysis was used with six domain summaries materialising to include: 'The Ease of VC'; 'The Personal Touches'; 'The Benefits of VC'; 'The Challenges of VC'; 'Technical Quality'; and 'Recommendations & Future Use'. An additional 17 subdomains are included. Direct quotations from patients and clinicians are provided for context. Conclusions: A total of 22 978 participants were included. These data help demonstrate that NHS remote service delivery, via the method of VC, can be highly satisfactory, well accepted and clinically suitable yielding many benefits. Despite this, the data are not without its challenges surrounding engagement and suitability for VC. The NHS Wales VC Service rolled out and evaluated at scale and demonstrates that VC has potential for long-term sustainability. For the future, use a 'blended approach' for NHS appointments that are clinically judged and centred on patient choice.
JohnsG, etal. BMJ Open 2022;12:e057874. doi:10.1136/bmjopen-2021-057874
Open access
Digital NHS Wales: a coding reliability
analysis based on the voices of 22 978
patients and clinicians on the benets,
challenges and sustainability of
video consulting
Gemma Johns , Bethan Whistance, Sara Khalil, Megan Whistance,
Bronwen Thomas, Mike Ogonovsky, Alka Ahuja
To cite: JohnsG, WhistanceB,
KhalilS, etal. Digital NHS
Wales: a coding reliability
analysis based on the voices
of 22 978 patients and
clinicians on the benets,
challenges and sustainability
of video consulting. BMJ Open
2022;12:e057874. doi:10.1136/
Prepublication history and
additional supplemental material
for this paper are available
online. To view these les,
please visit the journal online
Received 30 September 2021
Accepted 31 March 2022
TEC Cymru Informatics, Aneurin
Bevan Health Board, Gwent, UK
Correspondence to
Gemma Johns;
gemma. johns3@ wales. nhs. uk
Original research
© Author(s) (or their
employer(s)) 2022. Re- use
permitted under CC BY- NC. No
commercial re- use. See rights
and permissions. Published by
Introduction The use of video consulting (VC) in Wales
UK has expanded rapidly. Previous VC evidence has been
the subject of small- scale projects and evaluations.
Technology Enabled Care Cymru is an all- Wales digital
service and rolls out digital interventions and evaluates
on large scales, thus capturing representative data sets
across Wales, and therefore a wide range of National
Health Service (NHS) specialties.
Objective To extract and analyse narrative feedback from
patients and clinicians using the NHS Wales VC Service for
6 months (September 2020 to March 2021).
Design A coding reliability approach of a cross- sectional
study was conducted.
Setting From all health boards across Wales.
Participants NHS patients and clinicians across primary,
secondary and community care settings in Wales.
Results Data were captured on benets, challenges
and sustainability of VC. A coding reliability analysis was
used with six domain summaries materialising to include:
‘The Ease of VC’; ‘The Personal Touches’; ‘The Benets
of VC’; ‘The Challenges of VC’; ‘Technical Quality’; and
‘Recommendations & Future Use’. An additional 17
subdomains are included. Direct quotations from patients
and clinicians are provided for context.
Conclusions A total of 22 978 participants were
included. These data help demonstrate that NHS remote
service delivery, via the method of VC, can be highly
satisfactory, well accepted and clinically suitable yielding
many benets. Despite this, the data are not without its
challenges surrounding engagement and suitability for
VC. The NHS Wales VC Service rolled out and evaluated at
scale and demonstrates that VC has potential for long- term
sustainability. For the future, use a ‘blended approach’ for
NHS appointments that are clinically judged and centred
on patient choice.
Since 2020, the National Health Service
(NHS) has seen a paradigm shift in the provi-
sion of healthcare services due to mandatory
social distancing laws introduced because of
the COVID- 19 pandemic.1–3 As a result, the
UK along with the NHS observed a significant
decrease in access to face- to- face appoint-
ments, and therefore an increase in remote
Video consulting (VC) has accelerated
through health and social care as one of
the most common remote methods for
conducting appointments with patients
throughout the NHS, especially in Wales.5 6 VC
within health services has been internation-
ally used for decades, yet the unprecedented
circumstances of the pandemic brought to
light its widespread ability, use, value, benefits
and challenges.1
There is growing evidence that VC can
deliver safe and timely care in many settings
and offer significant benefits to the users.7 8
The use of VC permits services to continue
across a wide range of healthcare condi-
tions, appointment types, sociodemographic
Strengths and limitations of this study
This paper presents patient and clinician free text
narrative boxes on a large scale considering expe-
rience of a relatively new digital service in National
Health Service (NHS) Wales.
The study is representative of Wales, in that it is an
all- Wales study, across all health boards.
The study is a mix of patient and clinician voices
across all types of NHS specialties.
Due to the voluntary and anonymised nature of the
feedback data, it is unclear as to how often recurring
participants completed the feedback, thus potential-
ly skewing the sample.
Due to the size of the study sample it was not pos-
sible to present both the narrative and quantitative
ndings together; however, access to these data is
readily available on our website.
on April 20, 2022 by guest. Protected by copyright. Open: first published as 10.1136/bmjopen-2021-057874 on 20 April 2022. Downloaded from
2JohnsG, etal. BMJ Open 2022;12:e057874. doi:10.1136/bmjopen-2021-057874
Open access
groups and health condition status.9 Furthermore, it
is sometimes considered more suitable for reaching
underserved and isolated populations.4 VC is reported
to provide quality- ensured yet cost- effective care10 and
treatment, while reducing patient waiting times and the
likelihood of Did Not Attends and number of hospital
admissions—ultimately relieving pressure on NHS staff
and services.11
However, the majority of evidence is based on pilot
studies, with small and often highly selected samples, with
limited questionnaire validity, ultimately casting specula-
tion on its use, benefits and challenges across varied care
sectors, specialties and circumstances.12 13
There are often some concerns regarding the use of VC
services within certain professions that rely on face- to- face
physical examinations to make diagnoses, and the increased
level of risk associated.14 These valid uncertainties highlight
situations where sometimes it can be challenging to obtain
the same level of accuracy when taking clinical measures via
VC compared with obtaining them face to face. Not only
this, but personal preferences in clinicians and patients can
dictate whether or not a VC is used.
Therefore, the current evidence base suggests there is a
need to continuously evaluate on a national level to allow for
sustainable VC platforms to be embedded for the long term
into health and social care systems where appropriate.15 16 Due
to the need for a continual evaluation, Technology Enabled
Care (TEC) Cymru as an all- Wales digital service rolled out
the NHS Wales VC Service as a national emergency response
to the COVID- 19 pandemic.17 The evaluation spans a large
and representative scale basis across a wide range of NHS
healthcare sectors across all health boards in Wales. To
contribute to the current evidence base, TEC Cymru works
in partnership with the Welsh Government, academics, third
sector and local health boards and trusts in Wales to adopt a
clinically driven and data- informed approach to their digital
service roll- out, spread and evaluation.
Aims and methods
The aim of this study was to explore the benefits, chal-
lenges and sustainability of VC from the perspective
of Welsh NHS clinicians and patients by conducting a
coding reliability analysis, and presenting the narrative
feedback received from 22 978 participants during a
6- month period (September 2020 and March 2021). This
period was chosen as a ‘mid- point’ from a larger ongoing
VC evaluation during the COVID- 19 pandemic to gain
a better understanding of VC without the influence of
initial issues during the earlier months of VC being rolled
out, and current changes such as VC being scaled up with
a focus on blended consultation approaches.
Design, setting, participants
This paper presents the all- Wales data captured across
all seven health boards and one trust (see online supple-
mental appendix 1) across a range of NHS healthcare
settings within primary, secondary and community care
(see online supplemental appendices 2 and 3). This is
a coding reliability analysis of VC experience feedback
captured in a larger cross- sectional study18 held by the
NHS Wales VC Service, TEC Cymru.17 19 Participant eligi-
bility included NHS clinicians and patients using VC in
NHS Wales (see online supplemental appendices 4–9).
This paper presents national (all- Wales) data from free
text narrative boxes from a cross- sectional feedback study.
The feedback appeared as internet browser pop- ups at
the end of each VC appointment—one per clinician and
patient and completed immediately as live data to reflect
the use, benefits, challenges and sustainability of VC (see
online supplemental files 1 and 2).
Opportunity sampling was used due to accessibility of the
VC intervention and ability to capture data at the end of
each consultation via an online feedback link. There is
acknowledgement of the risks surrounding sampling in
this way, when considering the feedback being completed
by those more willing, thus sharing potentially more
extreme ‘positive’ or ‘negative’ data towards VC, poten-
tially missing out ‘neutral’ responses of those individ-
uals in the middle. To limit this, TEC Cymru conducts
multiple phases of re- evaluation using a phased approach
to their research and evaluation work (see online supple-
mental file 3), which provides ample opportunity across
their digital interventions to explore a wider range of
methodologies and study types.
Patient and public involvement
No patient or public involvement as survey work and
during the emerging roll- out did not have patient and
public involvement (PPI) team., as time has gone on we
now have a specific PPI team and young representatives.
For the data discussed in this paper, there are a total of
22 978 clinician and patient feedback narrative submis-
sions. Using steps for a coding reliability analysis,20 the
data were familiarised by three researchers of the TEC
Cymru team (GJ, BW, MW), codes were manually iden-
tified and generated and placed into an Excel sheet for
manageable order (due to large numbers). Domain
summaries were then generated from the data, reviewed
and defined and the report was produced following
a recursive process of movement between the phases,
ensuring quality and rigour, with an additional 20% vali-
dation check on all data by the national clinical lead for
Wales (AA).
A coding reliability analysis of the free text narrative data
collected at the end of VC feedback was conducted. From
the 22 978 patient and clinician responses captured during
on April 20, 2022 by guest. Protected by copyright. Open: first published as 10.1136/bmjopen-2021-057874 on 20 April 2022. Downloaded from
JohnsG, etal. BMJ Open 2022;12:e057874. doi:10.1136/bmjopen-2021-057874
Open access
September 2020 and March 2021, six domain summa-
ries materialised with an additional 17 subdomains.
These include: the ease of VC; the personal touches; the
benefits of VC; the challenges of VC; technical quality;
and recommendations and future use. Direct quota-
tions from patients and clinicians are provided. Each
quotation is referenced to describe either the patient by
their age range, gender, health board/trust, healthcare
specialty and type of appointment (eg, first appointment,
follow- up), or the clinician by their professional occupa-
tion and the health board/trust in Wales in which they
are based. The domains are analysed in order of the
most common comment/feedback due to the voluntary
Domain summary 1: ease of use
Patients and clinicians generally rate their VC as high in
quality16 and their free text narratives reflect this level in
terms of high satisfaction and acceptability in relation
to both technical and overall experience. For example,
when patients and clinicians rate their VC as ‘excellent’,
‘very good’ or ‘good’ this is often paired with positive
comments in relation to either the VC’s technical perfor-
mance as a VC platform or the overall experience of using
VC as a healthcare delivery service.
Ease of technical use
One of the most common subdomains associated to
the platform’s technical performance was that of ‘ease
of use’. It was often stated that the VC platform used in
NHS Wales (Attend Anywhere) was ‘easy to use’ for both
patients and clinicians.
Easy to use, lots of good information. (Parent of pa-
tient under 12 years, ABUHB, Physiotherapist, paedi-
atrics and child health, Advice)
It was easy to use, and appropriate to use during
the pandemic. (Patient, Female, HDUHB, 25–
44, Midwife, Obstetrics and gynaecology, First
In addition, this ease of use was expressed as a ‘surprise’
to some, in that both patients and clinicians found the
VC platform much easier to use than they initially antici-
pated, and in some instances, this exceeded expectation.
More effective than I expected a non- face to face ap-
pointment to be. (Patient, Male, 45–64, HDUHB)
This is my first experience of a video call, so I was
pleasantly surprised. (Patient, Female, 64–80,
BCUHB, Doctor, Follow- up)
First time to use video call I was very impressed,
better than expected. (Patient, ABUHB, Podiatrist,
Follow- up)
Ease of experience
For some clinicians, it was felt that having access to a VC
platform was ‘easier’ for some of their patients than a
face- to- face appointment would be. This was especially
apparent in terms of patient experience and their personal
circumstances, and those with access difficulties, anxiety
issues or complex home situations that were made more
convenient with VC.
Easier to access with social anxiety. (Doctor, CVUHB)
Very helpful for autistic patient. (Dentist/dental
nurse, SBUHB)
This was also expressed in more depth by the patients
themselves, whom in addition felt VC was better than
attending a face- to- face appointment, such as making
the patient feel safer, less stressed and more empow-
ered, as opposed to their prior experience of face- to- face
Easier and safer than going to the hospital. I didn’t
have to take much time off work. (Patient, SBUHB,
Female, 25–44, Dietician, First appointment)
Just as good as a face- to- face meeting and to be honest
I felt like I was being listened to far more than when
I have been in face- to- face meetings on the same
subject. (Parent of patient, ABUHB, Female, under
12 years, Nurse, Mental health, Advice)
Ease and unique for collaborations
Clinicians comment that the ‘ease’ of the VC plat-
form and its positive associations to patient experience
provides an additional unique opportunity. This opportu-
nity is the ability to link up others to the video call, thus
enabling multidisciplinary appointments to take place.
This is felt to be unique in the sense that this collaborative
approach would not have been possible if conducted face
to face, thus in turn produces additional advantages and
improved outcomes for patients, families and clinicians.
It was easy to join both my patients and other col-
leagues in. (Doctor, HDUHB)
Also, his Wife was able to join session—significant in-
formation shared by Wife today. (SLT, BCUHB)
It also means that in some instances, there is an increase
in patient or parental onus which is perceived as an addi-
tional advantage to patient care.
Parents have to take a more proactive role than they
might in clinical session. (SLT, ABUHB)
Definitely helped with family involvement today.
(Nurse, HDUHB)
Mum appears happy to support and possible not very
involved until now. (SLT, ABUHB)
Domain summary 2: the personal touches
The narrative data highlighted several incidences of
where VC has been able to increase clinician to patient
Communication, personalisation and rapport
Patients commonly expressed how VC helps them to
communicate effectively, to receive a more personalised
on April 20, 2022 by guest. Protected by copyright. Open: first published as 10.1136/bmjopen-2021-057874 on 20 April 2022. Downloaded from
4JohnsG, etal. BMJ Open 2022;12:e057874. doi:10.1136/bmjopen-2021-057874
Open access
and patient- centred approach and build rapport with
their clinicians.
Having a video call made it more personal for me
the support given to me was excellent. (Parent of pa-
tient, CAVUHB, under 12 years, Paediatrics and child
health, Advice)
We have built up a relationship with our clinician
via VC. (Patient, Male, 45–64, CAVUHB, Counsellor,
Mental health)
Patient positivity and appreciation
A strong consensus of patient ‘positivity’ and ‘apprecia-
tion’ towards their clinicians is expressed widely in the
narrative. This positive clinical presence led to many
patients feeling safe, comfortable and supported during
and after their VC. This was particularly evident across
specialties such as mental health and therapies.
Had a really tough week, but [name removed]
was amazing and she listened to me. She gave me
great support and was really kind to me. (Patient,
CAVUHB, 25–44, Psychologist, Mental health,
She was engaging, courteous and professional in
explaining what she felt the issues were with my hand.
I found the whole experience thoroughly satisfactory.
(Patient, CAVUHB, 45–64, Male, Occupational ther-
apist, Trauma and orthopaedics, First appointment)
It was great to feel that someone was there to chat
to, who could see a difference in [patient name
removed], while at the same time supporting us as a
family. (Guardian/carer of patient, SBUHB, 25–44,
Male, Mental health, Follow- up)
Domain summary 3: the benets of VC
The benefits associated to using VC were one of the most
common domains that materialised in the data.
Convenience, safety and home comforts
Many of the patients demonstrate the benefit of conve-
nience when using VC as opposed to a traditional face-
to- face appointment, with additional advantages such as
improved safety and home comforts.
As we live quite far away, the virtual meetings are a lot
more convenient and it’s nice to feel more comfort-
able at home. Thank you. (Patient, CAVUHB, Female,
25–44, Obstetrics and gynaecology, Follow- up)
I felt really comfortable talking to [name removed]. I
was able to get things off my chest, and talk about the
assault more deeply than I have ever done. (Patient,
ABUHB, 45–64, Female, Counsellor, Mental health,
Client is pregnant and so is vulnerable to the COVID
virus. AA means she can continue with therapy without
the additional risks. (Mental health, HDUHB)
Flexibility of VC
For many patients, a benefit of using VC was the flexibility
it allowed. For example, patients reported to be able to
continue ‘getting on with other things’ while waiting in
the ‘virtual waiting room’, which would not have been
possible in a physical location.
I felt it was good as I could start the call and then
get on with things around the house while I wait-
ed. (Parent of patient, BCUHB, Female, under 12
years, Physiotherapist, Paediatrics and child health,
I think that it is excellent to have a consultation this
way. It was easy to log on and saves so much time
for both of us. (Patient, ABUHB, 45–64, Female,
In addition, having three modes of appointment (VC,
telephone or face to face) provided the patient with a
stronger sense of patient choice and flexibility. However,
it was felt that VC, as opposed to a telephone consulta-
tion, allowed comparable aims and goals to be achieved
similar to a face to face.
VC let us achieve patient’s 1st choice, which could not
have been achieved over the phone. (Occupational
therapist, SBUHB)
AA is a way of bridging direct face- to- face and a visual
interaction can be helpful as part of the clinical as-
sessment. (Nurse, HDUHB)
Having a video consultation is so much better than
just a telephone call—it allows you to chat as if it was
in person. (Patient, PTHB, 64–80, Female, Nurse,
Respiratory medicine, Advice)
The ability to be able to visually ‘see’ the patient is
considered imperative to clinicians, as for many health-
care conditions VC is needed to enable visual cues.
Better than just telephone call as could get non- verbal
clues about emotions. (Doctor, ABUHB)
Really useful being able to see patient via system—
really added to consultation, infinitely superior to
telephone consultation. (Doctor, SBUHB)
Time savings
When using VC as opposed to face- to- face appoint-
ments many clinicians and patients expressed that they
had saved time in several ways, and this was a consider-
able benefit to patients, families, clinicians and the NHS
service as a whole.
For example, clinicians felt that the ‘time’ used to
conduct a VC was reduced in comparison to the usual
components of a face to face, for example, logistics. The
time saved from travelling to and from appointments was
able to be combined into the overall virtual consultation
in some cases, ultimately benefiting clinicians’ availability
to attend to other patient needs and clinical tasks.
on April 20, 2022 by guest. Protected by copyright. Open: first published as 10.1136/bmjopen-2021-057874 on 20 April 2022. Downloaded from
JohnsG, etal. BMJ Open 2022;12:e057874. doi:10.1136/bmjopen-2021-057874
Open access
Video consultation reduces time required the next
day. (Midwife, CVUHB)
Video consultation prior, ensures that less time on
home visits. (Health visitor, SBUHB)
In addition, the use of VC lowered the ‘wait times’ in
some instances for patients, in comparison to waiting for
a face- to- face appointment.
Fantastic way to be able to have an appointment
without having to wait months. (Parent of pa-
tient, CTMUHB, under 12 years, Male, Doctor,
Otolaryngology (ENT), First appointment)
Reduces time required for next appointment.
(Audiovestibular medicine, BCUHB)
From the data, ‘travel time’ or ‘time off’ work or school
was perceived as the biggest saving of time in comparison
to attending a face- to- face appointment for patients.
Less travel and disruption of [patient] school day.
(Dietician, ABUHB)
I just had just finished a night shift, and live a fair
distance from work, so doing a video call made my
life a lot easier. (Patient, SBUHB, 45–64)
Not having to travel to the hospital and waiting in
the waiting room was much better, and there was no
stress trying to get around everything all of the time.
(Patient, CVUHB, 45–64)
Clinical value
Many clinicians suggested that VC has the ability to
enhance a clinical session or determine clinical need.
For example, some clinicians demonstrated opportu-
nities to share visual resources immediately within the
appointment. Others reported that VC allowed for an
effective triaging tool to determine the ‘need’ for a face-
to- face appointment as opposed to a remote consultation
It enhanced the clinical session because it added
visual opportunity. (Speech and language therapy,
I could open investigations on screen easily, I shared
internet resources links to patient, and she got them
on screen straight away. (Doctor, BCUHB)
Domain summary 4: the challenges of VC
While there are a number of evident benefits when consid-
ering VC, it is important to highlight the challenges faced
to gain an overall picture of both patient and clinician
experiences which are subject to subtle nuances.
VC is not for everyone or everything
There are some clinical situations and personal circum-
stances which continue to necessitate the need for face-
to- face consultations, where VC does not achieve the
outcomes necessary, or suit the clinical condition or
patient type.
Still needs face- to- face as cannot test hearing over VC.
(Audiovestibular medicine, BCUHB)
Only thing missing was ability to weigh and get height.
(Dietician, ABUHB)
Patient and clinician digital ability
Some issues with patient and clinician user abilities were
also highlighted in the data, clinicians made note that on
occasion patients struggled to undertake VC due to their
lack of technological ability. This affected the potential
quality of the VC and therefore impacted the patients’
opportunity to receive care via VC.
Client unable to get full screen. Client not famil-
iar with using equipment at home. (Counsellor,
Psychiatry and Mental Health, Mental health,
Secondary, SBUHB)
Patient didn’t receive link so unable to do. I think it
may be due to me being unfamiliar with new system
in the end and it worked well. (Physiotherapist, Paedi-
atrics and Child Health, Therapies (AHP), CVUHB)
Engagement over VC
Engagement was a further challenge that clinicians expe-
rienced when using VC. Within therapies for younger
patients where parents were present, clinicians found it
challenging to engage with children via video in the same
way as face to face. Children were reported to be more
distracted during these appointments as the concept of
video was relatively new, therefore parents and clinicians
had to attempt to engage with the child more than via
face to face.
Poor picture quality does not encourage children
who already have attention/listening difficulties to
take part. (Speech and language therapist, ABUHB)
Child had short attention span and parents had to
work hard to keep him in front of camera. (Speech
and language therapist, ABUHB)
Sound quality not adequate at times to determine
success of target so reliant on adult feedback. Child
however responding better than expected although
once attention levels drop it is quite difficult to return
to tasks. (Speech and language therapist, Paediatrics
and child health, ABUHB)
Waiting room issues
A problem that was reported by a small number of
patients was associated to the virtual waiting time being
exceedingly long, or that their appointment was missed
by a clinician, or in some instances, where no clinician
attended at all.
Waited for over 40 minutes in the waiting room.
(Parent of patient, SBUHB, and parent of child un-
der 12 years, Paediatrics and child health, Follow- up)
We waited in the virtual waiting room for an hour
and 30 minutes without anyone answering. We
couldn’t wait any longer and due to our poor internet
on April 20, 2022 by guest. Protected by copyright. Open: first published as 10.1136/bmjopen-2021-057874 on 20 April 2022. Downloaded from
6JohnsG, etal. BMJ Open 2022;12:e057874. doi:10.1136/bmjopen-2021-057874
Open access
connection in our area, I had to use all my monthly
data. (Parent of patient, ABUHB, Parent of child
under 12 years, Male, Paediatrics and child health,
Follow- up)
Domain summary 5: technical quality
When considering the technical aspects of VC, clinicians
and patients tended to rate their VC highly when the
audio and visual picture were of good quality.
Good versus bad quality
For high- quality ratings, these were paired with praise
for VC in the free text narrative box, suggesting that the
audio and visual elements of the VC were of high quality.
Great connection. No glitches very smooth call.
(Parent of patient, ABUHB, under 12 years, Female,
Physiotherapist, First appointment)
The video and picture was perfect, was able to hear
the doctor clearly. (Patient, CAVUHB, 45–64, Female,
Doctor, Follow- up)
However, there were technological challenges reported
within the narrative such as poor connectivity, thus
impacting on visuals and audio.
Lag in audio/video sometimes causes miscommu-
nication or difficulty with younger patients. (SLT,
The video was very choppy and when my therapist
was talking it was delayed video with speech. (Patient,
CAVUHB, Female, 25–44, Counsellor, Mental health,
In some instances, technical issues were associated with
specific device types and their perceived incompatibilities
with the VC platform.
The video call app did not give me the option of using
my inner camera so I had to turn my phone around
so the doctor could see me. (Patient, BCUHB, 25–44,
Ophthalmology, Advice)
Problems with Safari on iPad. (Health visitor, SBUHB)
Session being done on Father’s phone so screen obvi-
ously small for child to watch. (SLT, ABUHB)
Clinical innovation and troubleshooting
Yet, despite these technological challenges, with the right
amount of technical support and appropriate equipment
available, clinicians report to be able to troubleshoot
many issues and continue to use VC in most situations.
Tried to do call with mobile phone and there were is-
sues for the patient not being able to grant access for
use on mobile phone, but the consultation worked
perfectly on their computer. (Dentist/dental nurse,
Issues at the start of the call with the audio but we
disconnected and reconnected and it was then fine.
(Nurse, SBUHB)
Some clinicians were able to troubleshoot the problems
easily to make the consultation work best for them and
their patients.
I was unable to connect through the desktop in clinic
due to computer being extremely slow…. I was luckily
able to connect through my Netbook which supports
the platform. (SLT, CVUHB)
School initially struggling with internet connection
but then able to move to a room with better signal
and VC quality. (SLT, CVUHB)
Domain summary 6: recommendations and future use
When considering the experiences of both clinicians
and patients using VC, it is important to consider how
the narrative can be built on to consider suggestions and
recommendations to ensure that VC is suitable for future
use and in conjunction, blended with face- to- face and
telephone consultations.
Clinical recommendations
One of these suggestions was improved infrastructure
and resources for clinical and administrative staff to have
access to. It was felt that by having better equipment, they
would deliver better patient care via VC. Not only this, but
in some areas the number of devices and access to work-
space was limited and needed significant improvement in
the future.
It would be useful to have 2 microphones so I can
share videos with my clients about EMDR therapy and
PTSD. (Nurse, BCUHB)
Need appropriate screens and two monitors to view
downloads and see patients, desk and chairs at right
height. (Dietician, BCUHB)
Clinicians also suggested that there needed to be an
improvement with the technical support that was on offer
across health boards regarding VC.
Being taught how to split screen so we can write notes
at the same time, rather than making paper notes and
writing up after. (Occupational therapist, BCUHB)
This suggestion of technology support would ensure
clinicians could use VC to the best of their ability, using
all aspects of the platform. Some clinicians suggested new
features that they felt would be useful in ensuring clini-
cian/patient confidentiality and safety.
To be able to blur/add a background when working
from home. (Dietician, SBUHB)
I would like to be able to leave the call screen but still
be able to see patient in a little pop- out screen. (SLT,
Additional administrative support was also suggested
for VC so that they could mirror the way standard face- to-
face consultations were run.
on April 20, 2022 by guest. Protected by copyright. Open: first published as 10.1136/bmjopen-2021-057874 on 20 April 2022. Downloaded from
JohnsG, etal. BMJ Open 2022;12:e057874. doi:10.1136/bmjopen-2021-057874
Open access
For this to work administrative clinic support need-
ed to mirror that provided for face- to- face appoint-
ments. (Nurse, CVUHB)
Patient wants and needs
Patients’ narrative also suggested that technical and
digital skills support would be useful in the future use
of VC. Some patients were slightly unsure of how to use
the technology needed for VC and ran into some issues.
By having support for this, it may lead to an increase in
digital skills for future digital implementations and the
move towards a new NHS digital strategy.
I couldn’t work out how to use the camera on the
front of my and wasn't sure how to connect via my
computer to the appointment. (Patient, ABUHB, 25–
44, Female, Mental health, First appointment)
I was unable to switch my camera to front facing,
so not able to see who I was talking to. (Patient,
CAVUHB, Female, 45–64, Nurse, Otolaryngology
(ENT), First appointment)
Patients provided narrative to suggest a blended
approach of digital healthcare services was needed going
forward. This was due to a large number of patients high-
lighting that VC provided numerous benefits, and help to
supplement the quality of care received from clinicians,
and believed a blended approach of VC and face to face
was the way forward for the future of the NHS Wales
support by clinicians.
Definitely the way forward for consultations, I live 100
miles away so for the purpose of consultation rath-
er than treatment this is brilliant! (Patient, SBUHB,
45–64, Female, Doctor, Plastic surgery)
I think this will be the future. I felt more relaxed
being able to do it from my home. (Patient, BCUHB,
Female, 64–80, Doctor, Obstetrics and gynaecology,
First appointment)
Video consultations act as a useful complement to
face- to- face sessions and home visits. (Audiologist,
The coding reliability analysis of the free text narrative
boxes captured at the end of VC provided feedback
from a large data set of 22 978 clinician and patient
submissions expressing a vast and overall view of VC
experiences in Wales. Six dominant domain summa-
ries and 17 additional subdomains materialised. Due
to the high response rate in free text narrative box
responses, the analysis of the feedback data was able to
be conducted using a coding reliability approach, thus
providing context for each domain and its perspec-
tive, supported by patient and clinician quotation. The
domains that materialised in the analysis provide a
strong sense that the NHS Wales VC Service on a whole
is highly satisfactory, well accepted and clinically suit-
able for a wide range of patient and clinical teams using
the service. Despite this, it is important to draw atten-
tion to the challenges that have also occurred for both
clinicians and patients, such as VC not always being
suitable for every individual or appointment.
The data provide a strong consensus that the VC plat-
form currently being used in NHS Wales is ‘easy to use’
in both technical and experience terms, with the addi-
tional value of its ability of enhanced collaboration, thus
providing a multidisciplinary approach to patient care.
In addition, the data highlight the real life and personal
aspects of VC experience, which suggests that patients
who are using the VC service are satisfied with using it
and provide narrative around its ease of use and person-
alisation felt in their patient care.
In addition, there is a heartfelt sense of patient appreci-
ation and gratitude to their clinicians for their hard work
and dedication to delivering patient care. Furthermore,
the data demonstrate the benefits that are associated
to using VC. These benefits are felt by patients, fami-
lies and clinicians, and the NHS service. Challenges are
also apparent within the data with VC not always being
appropriate for all patients or appointments. This is in
combination with difficulties surrounding engagement,
particularly with children via VC and issues with digital
ability across clinician and patient populations. Although
there were disparities of digital ability that sometimes
hindered a VC appointment, and suggestive of a digital
divide, in this data set and the wider evaluation21 we did
not find this to be the case. The quantitative findings
that run alongside these data provide additional support,
specifically regarding patient representation, concluding
that regardless of patient age, gender, ethnicity, house-
hold income, health condition, disability or place (urban
vs rural), VC can provide a high standard of healthcare
delivery across Wales.18 21 22 Though apparent, the chal-
lenges were heavily outweighed by the number of benefits
experienced from using VC.
The data also present a comparison between good
and bad technical quality on the platform regarding
audio and visuals for both patients and clinicians.
Improvements for future use should encapsulate recom-
mendations such as more resources to be made avail-
able to clinical teams, and that VC platform features
are considered as priority for improvement. It is also
noted that increased technical support and education
is provided to ensure that VC can appropriately be used
in the future, and possibly offered directly to patients,
so that VC is used as a long- term blended approach to
suit patient choice and preferences moving forward.
While challenges have been identified, the data
captured in this study are comparable to previous liter-
ature that suggests the benefits of VC outweigh these
challenges18 22 and can support the use and sustain-
ability of VC in NHS healthcare services. As discussed
within the Results section, VC is not seen to be used
for everything within healthcare, despite the benefits
on April 20, 2022 by guest. Protected by copyright. Open: first published as 10.1136/bmjopen-2021-057874 on 20 April 2022. Downloaded from
8JohnsG, etal. BMJ Open 2022;12:e057874. doi:10.1136/bmjopen-2021-057874
Open access
highlighted within this study. The need to ensure that
VC is offered within every healthcare environment is
pertinent to its sustainable future use along with shared
decisions between clinicians and patients.23
There are many strengths to this study, including its
narrative approach among a very large and representa-
tive sample for Wales. However, the study did have some
limitations. Due to the voluntary and anonymised nature
of the feedback data collection, it is unclear as to how
often recurring participants completed the feedback,
thus potentially skewing the sample. Furthermore, due to
the size of the study, it was not possible to present both the
qualitative and quantitative findings together; however,
access to these data is readily available.22
Originally, the data were broken down into special-
ties, but were regrouped for the purpose of this narra-
tive analysis as most of the data showed little difference
between specialties. While the quotations used within
these data are true of the narrative at the time of collec-
tion, it is important to note that these are fitting of a
time during the pandemic and so reflect this period.
Data are being captured in an ongoing evaluation of
VC within Wales, and suggest that those using VC are
still rating it positively.
While this paper does not cover specific clinical impli-
cations, VC is being used across various specialties and
evaluation is ongoing to explore this further.
Twitter Gemma Johns @teccymru
Contributors GJ contributed to the main design of the study and development
of the research questions, the main structure and write- up of the paper, and nal
amendments to the manuscript. GJ, BW, BT and MW analysed the data with the
supervision of AA, SK and MO. All authors discussed and interpreted the data once
analysed and helped structure the manuscript. AA, SK and MO contributed to the
clinical understanding of the ndings and shaped the discussion, conclusions and
recommendations. AA was responsible for overseeing the full development of the
study design and data collection, the analysis and development and nal sign- off of
manuscript from a clinical and programme perspective. All authors contributed to
proofreading and amendments of the nal manuscript. GJ is the guarantor for this
Funding Technology Enabled Care (TEC) Cymru and its NHS Wales Video
Consulting Service is funded by the Welsh Government.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in
the design, or conduct, or reporting, or dissemination plans of this research.
Patient consent for publication Obtained.
Ethics approval This study involves human participants and TEC Cymru obtained
full ethical approvals and risk assessments from their host Aneurin Bevan
University Health Board Research and Development Department (reference number:
SA/1114/20), and then national approval was obtained from all other health boards
in Wales. Full consent was obtained from all participants. At the end of each
feedback link, a statement of consent and a compulsory tickbox was required prior
to feedback submission. Participants gave informed consent to participate in the
study before taking part.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement Data are available upon reasonable request. All
analysed data are published on the TEC Cymru website in the format of a full
report of all data for the public to view. To access these reports please see https:// Other data can be requested as a reasonable request
to the corresponding author.
Supplemental material This content has been supplied by the author(s). It has
not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been
peer- reviewed. Any opinions or recommendations discussed are solely those
of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and
responsibility arising from any reliance placed on the content. Where the content
includes any translated material, BMJ does not warrant the accuracy and reliability
of the translations (including but not limited to local regulations, clinical guidelines,
terminology, drug names and drug dosages), and is not responsible for any error
and/or omissions arising from translation and adaptation or otherwise.
Open access This is an open access article distributed in accordance with the
Creative Commons Attribution Non Commercial (CC BY- NC 4.0) license, which
permits others to distribute, remix, adapt, build upon this work non- commercially,
and license their derivative works on different terms, provided the original work is
properly cited, appropriate credit is given, any changes made indicated, and the use
is non- commercial. See:
1 Connor MJ, Winkler M, Miah S. COVID- 19 pandemic - is virtual
urology clinic the answer to keeping the cancer pathway moving?
BJU Int 2020;125:E3–4.
2 Ramalho R, Adiukwu F, Gashi Bytyçi D, etal. Telepsychiatry during
the COVID- 19 pandemic: development of a protocol for telemental
health care. Front Psychiatry 2020;11:552450.
3 Leng S, MacDougall M, McKinstry B. The acceptability to
patients of video- consulting in general practice: semi- structured
interviews in three diverse general practices. J Innov Health Inform
4 Whaibeh E, Mahmoud H, Naal H. Telemental health in the context of
a pandemic: the COVID- 19 experience. Curr Treat Options Psychiatry
5 Greenhalgh T, Wherton J, Shaw S, etal. Video consultations for
covid- 19. BMJ 2020;368:m998.
6 John G, Khalil S, Ogonovsky M. Phase 1 report. Chapter 1 live data
- patients & clinicians. The NHS Wales video consulting service,
TEC Cymru, 2020. Available:
7 Donaghy E, Atherton H, Hammersley V, etal. Acceptability, benets,
and challenges of video consulting: a qualitative study in primary
care. Br J Gen Pract 2019;69:e586–94.
8 Wade VA, Karnon J, Elshaug AG, etal. A systematic review of
economic analyses of telehealth services using real time video
communication. BMC Health Serv Res 2010;10:1–3.
9 Smith AC, Thomas E, Snoswell CL, etal. Telehealth for global
emergencies: implications for coronavirus disease 2019 (COVID- 19).
J Telemed Telecare 2020;26:309–13.
10 Hollander JE, Carr BG. Virtually perfect? Telemedicine for Covid- 19.
N Engl J Med Overseas Ed 2020;382:1679–81.
11 Reinhardt I, Gouzoulis- Mayfrank E, Zielasek J. Use of telepsychiatry
in emergency and crisis intervention: current evidence. Curr
Psychiatry Rep 2019;21:1–8.
12 CWTCH Cymru toolkit: step by step guide to using video consulting
in telepsychiatry, 2020. Available:
13 Barsom EZ, van Hees E, Bemelman WA, etal. Measuring patient
satisfaction with video consultation: a systematic review of
assessment tools and their measurement properties. Int J Technol
Assess Health Care 2020;36:356–62.
14 Gallo G, Grossi U, Sturiale A, etal. E- consensus on telemedicine
in proctology: a RAND/UCLA- modied study. Surgery
15 Wherton J, Greenhalgh T. Evaluation of the attend anywhere/near me
video consulting service in Scotland, 2019- 20. Scottish Government,
16 Willman AS. A service user evaluation of eConsult use by defence
primary healthcare primary care clinicians using a mixed- method
approach. medRxiv 2020.
17 The NHS Wales video consulting service, 2020. Available: https://
on April 20, 2022 by guest. Protected by copyright. Open: first published as 10.1136/bmjopen-2021-057874 on 20 April 2022. Downloaded from
JohnsG, etal. BMJ Open 2022;12:e057874. doi:10.1136/bmjopen-2021-057874
Open access
18 Technology enabled care (TEC) Cymru. Available: https://
19 Phase 2A qualitative data. The NHS Wales video consulting service,
technology enabled care (Tec) Cymru, 2021. Available: https://
20 O’Connor C, Joffe H. Intercoder reliability in qualitative
research: debates and practical guidelines. Int J Qualit Method
21 Johns G, Khalil S, Ogonovsky M, etal. Access to the digital NHS is
not much of a problem in Wales. BMJ 2021;374:n2212.
22 Phase 2A quantitative data. The NHS Wales video consulting service,
technology enabled care (Tec) Cymru, 2021. Available: https://
23 Barsom EZ, Jansen M, Tanis PJ, etal. Video consultation
during follow up care: effect on quality of care and patient- and
provider attitude in patients with colorectal cancer. Surg Endosc
on April 20, 2022 by guest. Protected by copyright. Open: first published as 10.1136/bmjopen-2021-057874 on 20 April 2022. Downloaded from
ResearchGate has not been able to resolve any citations for this publication.
Full-text available
Background The rapid spread of the Coronavirus disease 2019 (COVID-19) has forced most countries to take drastic public health measures, including the closure of most mental health outpatient services and some inpatient units. This has suddenly created the need to adapt and expand telepsychiatry care across the world. However, not all health care services might be ready to cope with this public health demand. The present study was set to create a practical and clinically useful protocol for telemental health care to be applied in the context of the current COVID-19 pandemic.MethodsA panel of psychiatrists from 15 different countries [covering all World Health Organization (WHO) regions] was convened. The panel used a combination of reactive Delphi technique and consensus development conference strategies to develop a protocol for the provision of telemental health care during the COVID-19 pandemic.ResultsThe proposed protocol describes a semi-structured initial assessment and a series of potential interventions matching mild, moderate, or high-intensity needs of target populations.Conclusions Telemedicine has become a pivotal tool in the task of ensuring the continuous provision of mental health care for the population, and the outlined protocol can assist with this task. The strength of this protocol lies in its practicality, clinical usefulness, and wide transferability, resulting from the diversity of the consensus group that developed it. Developed by psychiatrists from around the globe, the proposed protocol may prove helpful for many clinical and cultural contexts, assisting mental health care providers worldwide.
Full-text available
Background Video consultation (VC) is considered promising in delivering healthcare closer to the patient and improving patient satisfaction. Indeed, providing care-at-distance via VC is believed to be promising for some situations and patients, serving their needs without associated concomitant costs. In order to assess implementation and perceived benefits, patient satisfaction is frequently measured. Measuring patient satisfaction with VC in healthcare is often performed using quantitative and qualitative outcome analysis. As studies employ different surveys, pooling of data on the topic is troublesome. This systematic review critically appraises, summarizes, and compares available questionnaires in order to identify the most suitable questionnaire for qualitative outcome research using VC in clinical outpatient care. Methods PubMed, Embase, and Cochrane were searched for relevant articles using predefined inclusion criteria. Methodological quality appraisal of yielded questionnaires to assess VC was performed using the validated COSMIN guideline. Results This systematic search identified twelve studies that used ten different patient satisfaction questionnaires. The overall quality of nine questionnaires was rated as “inadequate” to “doubtful” according to the COSMIN criteria. None of the questionnaires retrieved completed a robust validation process for the purpose of use. Conclusion and recommendations Although high-quality studies on measurement properties of these questionnaires are scarce, the questionnaire developed by Mekhjian has the highest methodological quality achieving validity on internal consistency and the use of a large sample size. Moreover, this questionnaire can be used across healthcare settings. This finding may be instrumental in further studies measuring patient satisfaction with VC.
Full-text available
The COVID‐19 pandemic presents an unprecedented challenge to our National Health Service (NHS) (1). As the need to appropriately direct all efforts towards providing emergency supportive care to those suffering, there will be a knee‐jerk tendency to cancel all outpatient activity by NHS trusts. Whilst this appears to be a pragmatic approach to reducing risk of transmission, there will be an unmet cost to those patients who are high‐risk and are already on the cancer referral pathway.
Full-text available
Background Video consultation (VC) is gaining attention as a possible alternative to out-patient clinic visits. However, little is known in terms of attitude, satisfaction and quality of care using VC over a face-to-face (F2F) consultation. The aim of this observational survey study was to compare the attitude and satisfaction with VC amongst patients suffering from colorectal cancer and their treating surgeons at the outpatient surgical care clinic in a tertiary referral centre.MethodsA patient-preference model was chosen following the concept of shared decision making. A total of fifty patients with colorectal cancer were asked to choose between VC- or a F2F-contact during their follow up at the outpatient surgical care clinic and were subsequently assigned to either the VC-group or the F2F-group. Attitude and satisfaction rates of both groups and their surgeons were measured using a questionnaire administered immediately after the consultation.ResultsOut of the 50 patients, 42% chose VC as their preferred follow-up modality. Patients demographics did not differ significantly. Patients who use video calling in their personal life choose VC significantly more often than patients lacking such experience (p = 0.010). These patients scored high on both the attitude- and satisfaction scale of the post-VC questionnaire. Patients who chose a F2F-contact seemed to question the ability of the surgeon to properly assess their healthcare condition by using a video connection more (p = 0.024). Surgeons were highly satisfied with the use of VC.Conclusions Based on patient preference, VC is equivalent to a F2F consultation in terms of patient satisfaction and perceived quality of care. Shared decision making is preferred with regard to which contact modality is used during follow up. For easy uptake in other environments it is to be recommended to facilitate VC using the electronic patient portal.
Background Coronavirus disease 2019 is revolutionizing healthcare delivery. The aim of this study was to reach a consensus among experts as to the possible applications of telemedicine in the proctologic field. Methods A group of 55 clinical practice recommendations was developed by a clinical guidance group based on coalescence of evidence and expert opinion. The Telemedicine in Proctology Italian Working Group included 47 Italian Society of Colorectal Surgery nominated experts evaluating the appropriateness of each clinical practice recommendations based on published RAND/UCLA methodology in 2 rounds. Results Stakeholder median age was 53 years (interquartile range limits 40–60), and 38 (81%) were men. Nine (19%) panelists reported no experience with telemedicine before the pandemic. Agreement was obtained on a minimum of 3 to 5 years of practice in the proctologic field before starting teleconsultations, which should be regularly paid, with advice and prescriptions incorporated into a formal report sent to the patient by e-mail along with a receipt. Of the panelists, 35 of 47 (74%) agreed that teleconsultation carries the risk of misdiagnosis of cancer, thus recommending an in-person assessment before scheduling any surgery. Fifteen additional clinical practice recommendations were re-elaborated in the second round and assessed by 44 of 47 (93.6%) panelists. The application of telemedicine for the diagnosis of common proctologic conditions (eg, hemorrhoidal disease, anal abscess and fistula, anal condylomas, and anal fissure) and functional pelvic floor disorders was generally considered inappropriate. Teleconsultation was instead deemed appropriate for the diagnosis and management of pilonidal disease. Conclusion This e-consensus revealed the boundaries of telemedicine in Italy. Standardization of infrastructures, logistics, and legality remain to be better elucidated.
Introduction eConsult has recently been introduced into Defence Primary Healthcare to allow Service Personnel of the British Armed Forces and their dependants improved access to healthcare. This review sought the views of primary care clinicians using eConsult. Method An 18-item survey was constructed after an initial scoping survey. This was then distributed to primary care clinicians in Defence Primary Healthcare to assess the broader applicability of the themes identified. Data synthesis of this alongside free-text responses from respondents was undertaken to explore advantages and disadvantages of eConsult. Results Four themes were identified: accessibility, effects on working practices, impact on the dynamics of the consultation and training/administrative support. eConsult did not save time for clinicians but was generally more convenient for patients. eConsult was often used in conjunction with telephone and face-to-face follow-up, forming a ‘blended consultation’. Accessibility was improved, but cultural factors may affect some patients engaging. Conclusions eConsult improves accessibility for patients but does not reduce workload. It should be used alongside conventional access methods, not instead of. It was found to be useful for straightforward clinical and administrative problems but less useful for more complex cases unless part of a ‘blended consultation’. Future use could be modified to provide greater data gathering for occupational health and chronic disease monitoring and should be monitored to ensure it is inclusive of all demographic groups.
The current coronavirus (COVID-19) pandemic is again reminding us of the importance of using telehealth to deliver care, especially as means of reducing the risk of cross-contamination caused by close contact. For telehealth to be effective as part of an emergency response it first needs to become a routinely used part of our health system. Hence, it is time to step back and ask why telehealth is not mainstreamed. In this article, we highlight key requirements for this to occur. Strategies to ensure that telehealth is used regularly in acute, post-acute and emergency situations, alongside conventional service delivery methods, include flexible funding arrangements, training and accrediting our health workforce. Telehealth uptake also requires a significant change in management effort and the redesign of existing models of care. Implementing telehealth proactively rather than reactively is more likely to generate greater benefits in the long-term, and help with the everyday (and emergency) challenges in healthcare.