Content uploaded by Alejandro Dominguez-Rodriguez
Author content
All content in this area was uploaded by Alejandro Dominguez-Rodriguez on Apr 11, 2023
Content may be subject to copyright.
Original Paper
The Efficacy and Usability of an Unguided Web-Based Grief
Intervention for Adults Who Lost a Loved One During the
COVID-19 Pandemic: Randomized Controlled Trial
Alejandro Dominguez-Rodriguez1,2, PhD; Sergio Sanz-Gomez2,3, PhD; Leivy Patricia González Ramírez4, PhD;
Paulina Erika Herdoiza-Arroyo5, PhD; Lorena Edith Trevino Garcia, MSc; Anabel de la Rosa-Gómez6, PhD; Joel
Omar González-Cantero7, PhD; Valeria Macias-Aguinaga, BSc; Melina Miaja4, PhD
1Department of Psychology, Health and Technology, University of Twente, Enschede, Netherlands
2Health Sciences Area, Valencian International University, Valencia, Spain
3Department of Psychiatry, Universidad de Sevilla, Seville, Spain
4School of Medicine and Health Sciences, Tecnologico de Monterrey, Guadalajara, Mexico
5School of Psychology, Universidad Internacional del Ecuador, Quito, Ecuador
6Faculty of Higher Studies Iztacala, National Autonomous University of Mexico, State of Mexico, Mexico
7Department of Behavioral Sciences, Centro Universitario de los Valles, Universidad de Guadalajara, Guadalajara, Mexico
Corresponding Author:
Paulina Erika Herdoiza-Arroyo, PhD
School of Psychology
Universidad Internacional del Ecuador
Av. Simón Bolívar and Av. Jorge Fernández
Quito, 170102
Ecuador
Phone: 593 22985 600
Email: pherdoiza@uide.edu.ec
Abstract
Background: The death of a loved one was a challenge many people faced during the COVID-19 pandemic within the context
of extraordinary circumstances and great uncertainty. Grief is an unavoidable part of life, and for most people, feelings of grief
decrease naturally over time. However, for some people, grieving can become a particularly painful process with clinical symptoms
that may require professional help to resolve. To provide psychological support to people who had lost a loved one during the
COVID-19 pandemic, an unguided web-based psychological intervention was developed.
Objective: The main objective of this study was to evaluate the efficacy of the web-based treatment, Grief COVID (Duelo
COVID in Spanish; ITLAB), in reducing clinical symptoms of complicated grief, depression, posttraumatic stress, hopelessness,
anxiety, and suicidal risk in adults. The secondary aim was to validate the usability of the self-applied intervention system.
Methods: We used a randomized controlled trial with an intervention group (IG) and a waitlist control group (CG). The groups
were assessed 3 times (before beginning the intervention, upon completing the intervention, and 3 months after the intervention).
The intervention was delivered on the web in an asynchronous format through the Duelo COVID web page. Participants created
an account that could be used on their computers, smartphones, or tablets. The evaluation process was automated as part of the
intervention.
Results: A total of 114 participants were randomly assigned to the IG or CG and met criteria for inclusion in the study (n=45,
39.5% completed the intervention and n=69, 60.5% completed the waitlist period). Most participants (103/114, 90.4%) were
women. The results indicated that the treatment significantly reduced baseline clinical symptoms in the IG for all variables (P<.001
to P=.006), with larger effect sizes for depression, hopelessness, grief, anxiety, and risk of suicide (all effect sizes ≥0.5). The
follow-up evaluation showed that symptom reduction was maintained at 3 months after the intervention. The results from the CG
showed that participants experienced significantly decreased levels of hopelessness after completing the time on the waitlist
(P<.001), but their suicidal risk scores increased. Regarding the usability of the self-applied intervention system, the results
indicated a high level of satisfaction with the Grief COVID.
J Med Internet Res 2023 | vol. 25 | e43839 | p. 1https://www.jmir.org/2023/1/e43839 (page number not for citation purposes)
Dominguez-Rodriguez et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
Conclusions: The self-applied web-based intervention Grief COVID was effective in reducing symptoms of anxiety, depression,
hopelessness, risk of suicide risk, posttraumatic stress disorder, and complicated grief disorder. Grief COVID was evaluated by
the participants, who reported that the system was easy to use. These results affirm the importance of developing additional
web-based psychological tools to help reduce clinical symptoms in people experiencing grief because of the loss of a loved one
during a pandemic.
Trial Registration: ClinicalTrials.gov NCT04638842; https://clinicaltrials.gov/ct2/show/NCT04638842
(J Med Internet Res 2023;25:e43839) doi: 10.2196/43839
KEYWORDS
web-based intervention; usability; complicated grief; hopelessness; suicidal risk; depression; anxiety; posttraumatic stress; mobile
phone
Introduction
Overview
According to the World Health Organization [1], as of July
2022, there had been 6,343,783 deaths worldwide caused by
the COVID-19 [1]. In the American continent, 2,754,186 deaths
were reported (Pan American Health Organization) [2],
including 325,976 deaths reported in Mexico [3]. In addition
to the high mortality rates during the pandemic, other factors,
such as the risk of contagions, social isolation, and economic
insecurity, precipitated mental health problems, including
anxiety, stress, depression, posttraumatic stress symptoms [4-7],
and an increased probability of developing complicated grief
disorder (CGD) [8]. According to Wallace et al [9], the
“COVID-19 pandemic has disrupted usual experiences of grief,
and modifications of approaches to support grief are needed.”
There is scarce evidence at this time about mental health
interventions during the COVID-19 pandemic or about
interventions specifically related to grief [10]. Web-based
interventions can enhance care coverage while maintaining the
same quality of care and efficacy. Interventions delivered on
the web allow greater accessibility, flexibility, availability,
scalability, and reliability, constituting a viable alternative to
face-to-face mental health services [11].
Background
Grief is a natural reaction to loss, often caused by the death of
a loved one. Grief helps individuals adapt to the new reality
[10]. Grief is an inevitable and painful experience in life with
affective, cognitive, behavioral, and spiritual manifestations
[4,12]. Some of the most common symptoms include physical
distress, sadness, anxiety, confusion, longing, rumination about
the past, and fear about the future. People might also experience
regret for something lost, remorse for something that occurred
in the past, or sorrow for a mishap [13]. Manifestations and
duration of grief vary widely, but symptoms are generally
expected to improve within the first 2 to 4 months. Culture can
affect the grieving period, mediating the relationship between
grief and time to heal [4,13]. Other factors that can affect a
person’s experience of grief are individual characteristics and
the type of relationship they had with the deceased person,
making the process more challenging to overcome [8].
There is an essential difference between grief and mourning.
Grief refers to the reaction to loss, whereas mourning refers to
the adaptation process of loss [14]. Mourning is an important
protective factor against pathological grief [12]. CGD, also
called persistent grief or persistent complex bereavement
disorder, is characterized by persistent (lasting >6 months) and
intense grief symptoms. This disorder can impede the return to
everyday life and cause significant distress or impairment in
functioning [8,10,15,16]. Hopelessness, rumination, alienation,
depression, anxiety, yearning, guilt, anger, denial, blame, shame,
or posttraumatic stress symptoms may be present when
experiencing CGD [10,12,17,18].
CGD is more likely to occur when the bereaved person does
not have social support [15], does not go through the normal
grieving process, and cannot return to their everyday life and
when symptoms of grief persist. Furthermore, the process of
mourning may be interrupted for various reasons, for example,
if the individual was not able to attend the deceased person’s
funeral, if they do not entirely accept or they deny the loss, if
the death was unexpected or violent, and if the individual
experienced multiple losses at once or over a short period of
time. In addition, people with a history of mental health
disorders (eg, mood disorders or addiction) are more likely to
develop CGD [4,12].
Before the COVID-19 pandemic, Lundorff et al [19] reported
a CGD prevalence of 10% in bereaved adults in a systematic
review. As losses owing to COVID-19 meet all the risk factors
for traumatic grief [20,21], an increase in the prevalence of
CGD was expected. Tang and Xiang [17] reported a CGD
prevalence of 37.8% in 476 participants mourning losses owing
to COVID-19, with data collected from September 1, 2020, to
October 3, 2020. In turn, secondary and concurrent factors can
make grief more difficult to overcome at this time, such as
unexpectedly high mortality rates worldwide, isolation measures
to prevent infections, intrahousehold tensions, worries about
possible contagion and the availability of health care, the
absence of farewell rituals, or the stigma surrounding COVID-19
patients and their families. These circumstances have generated
instability in many domains, including social networks and
relationships, finances, work, autonomy, safety, social roles,
plans, recreation, and personal freedom, among others
[4,8-10,13,22-25]. In the face of these deaths and secondary
losses, the need for interventions to treat grief is evident [4].
Laranjeira et al [10] reviewed grief support programs and
bereavement interventions in response to the COVID-19
pandemic. Most interventions and programs were implemented
in a clinical setting by health professionals. The interventions
J Med Internet Res 2023 | vol. 25 | e43839 | p. 2https://www.jmir.org/2023/1/e43839 (page number not for citation purposes)
Dominguez-Rodriguez et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
included support via telephone, psychoeducation, counseling,
psychological support, and specialized mental health care
cognitive behavioral therapy (CBT) interventions. Proposals
for grief interventions during the COVID-19 pandemic
[14,26-29] are scarce, and in some cases, the results on their
effectiveness are yet to be published. A small number of studies
have provided results with a limited sample size of 4 participants
[29].
There are initial studies that have been applied to people who
are experiencing grief. For example, Reitsma et al [16] designed
a web-based intervention based on CBT to reduce CGD,
posttraumatic stress, and symptoms of depression for adults
who had lost a loved one during the COVID-19 pandemic at
least 3 months before participating in the study. The treatment
consisted of 8 sessions delivered over a maximum of 12 weeks.
The intervention included psychoeducation, exposure, and
cognitive-restructuring assignments. Subsequently, in the second
part of the intervention, trained psychologists guided the
participants via email. The results of this study have recently
been published [27], involving 65 Dutch adults experiencing
bereavement during the pandemic. The study showed the
effectiveness of this web-based treatment in reducing disturbed
grief, posttraumatic stress, and depression symptoms.
In another study, Tang et al [28] developed a treatment to help
Chinese people who were experiencing CGD. The treatment
focused on meaning reconstruction and CBT for complicated
grief through imaginal exposure, cognitive restructuring, and
behavioral activation. This intervention included 8 to 10 sessions
and was preceded by training with professional counselors.
However, these results have not yet been reported.
Another study by Borghi et al [26] reported their experience
with a phone-based early psychological intervention conducted
by the clinical psychology unit of an Italian hospital. The
psychologists worked with individuals via phone who had lost
a loved one to COVID-19. The intervention involved 3 phases:
the management of practical procedures related to death, the
primary preventive phone-based psychological intervention,
and a community-based psychotherapeutic intervention for those
who were determined to be at risk for prolonged or excessive
grief. The last group was offered individual psychotherapeutic
grief sessions via video calls. Menichetti et al [30] published a
qualitative study on this intervention. The study included 246
families and examined their experiences and needs regarding
the grieving process and intervention and the role that the
psychologists played through the phone calls. A thematic
analysis was performed using transcripts of the calls,
semistructured interviews with the psychologists involved, and
observation of psychologists’ peer group discussions. The
findings suggest that psychologists can fulfill both
social-institutional and psychological-human functions to
support individuals in dealing with reactions to the loss of a
family member during the COVID-19 pandemic.
Finally, Tur et al [29] used a multiple-baseline single-case
experimental design to evaluate a web-based CBT program
(GROw) for adults with CGD. The program involved behavioral
activation therapy and mindfulness. Only 6 participants were
recruited; of these, 2 did not complete the intervention. The
results of the study were inconclusive; only half of the
participants who finished the treatment demonstrated a clinically
significant change in the bereavement measures.
Theoretical Framework for the Intervention
The intervention Grief COVID (ITLAB) draws on the principles
of Worden’s [31] Four Tasks of Grieving. Worden [31] suggests
that there are 4 tasks that one must accomplish for equilibrium
to be reestablished. The first task is to accept the reality of the
loss (intellectual acceptance and emotional acceptance). The
second task is to work through the pain of grief, including
various emotions such as sadness, fear, loneliness, despair,
hopelessness, anger, guilt, blame, shame, relief, and other
emotions associated with the loss. The third task is to adjust to
living in the environment where the deceased person is no longer
there. Finally, the fourth task is to establish an appropriate,
ongoing emotional connection with the person who has died,
and this also permits one to continue living.
In this regard, Ireland implemented a National Bereavement
Support line in response to the COVID-19 pandemic. The
service was based on Psychological First Aid principles adapted
to integrate bereavement education based on 2 models:
Worden’s Task Approach and Stroebe and Schut’s Dual Process
Model. The findings from the evaluation of the service suggest
that Psychological First Aid is an effective first-line intervention
in response to the COVID-19 pandemic. The program provides
a supportive, compassionate listening service; educational
resources; and referrals to community services [32].
To the authors’best knowledge, in Latin America, no web-based
interventions have been developed before our study to support
individuals going through a grieving process. Clinical symptoms
and variables of well-being were assessed as proposed in the
Grief COVID protocol article [11]. The results have been
organized into 2 manuscripts that include different variables.
One article has been submitted and is under review for
publication, and the second constitutes the present manuscript.
Aims
This study aimed to present the efficacy of Grief COVID (Duelo
COVID in Spanish) in decreasing the risk of developing CGD
and in reducing the symptoms of anxiety, depression,
hopelessness, and posttraumatic stress disorder (PTSD) and the
risk of suicide. The secondary objective was to validate the
usability of the self-applied intervention system.
Hypotheses
Primary Hypothesis
Participants in the self-applied multicomponent web-based
psychological intervention would report better indicators related
to grief and show a reduction in symptoms of anxiety,
depression, PTSD, and hopelessness and the risk of suicide
compared with the waitlist group. These changes would be
maintained for 3 months after treatment completion.
Secondary Hypothesis
The self-applied multicomponent web-based psychological
intervention for the prevention of CGD would show a good
level of acceptance or satisfaction or usability measures.
J Med Internet Res 2023 | vol. 25 | e43839 | p. 3https://www.jmir.org/2023/1/e43839 (page number not for citation purposes)
Dominguez-Rodriguez et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
Methods
Study Design
The researchers aimed to conduct a randomized controlled trial
(RCT), and this study followed the CONSORT (Consolidated
Standards of Reporting Trials) and SPIRIT (Standard Protocol
Items: Recommendations for Interventional Trials) guidelines.
The CONSORT-EHEALTH (Consolidated Standards of
Reporting Trials of Electronic and Mobile Health Applications
and Online Telehealth) checklist V1.6.2 is provided (Multimedia
Appendix 1). The trial registration number of the study is
ClinicalTrials.gov NCT04638842. To access the intervention,
participants were required to create an account on the Duelo
COVID web page [33], which was accessible on computers,
smartphones, and tablets. The evaluation process was automated
as part of the intervention. The participants received an email
confirming their accounts and granting them access to the web
page with the informed consent form and intervention materials.
Once the participants had completed the initial assessment,
those who met the inclusion criteria (and none of the exclusion
criteria) were randomly assigned to either the intervention group
(IG) or the control group (CG). The participants assigned to the
CG did not receive the intervention immediately; instead, they
were placed on a waitlist for 36 days. After completing this
period, the participants were granted access to the intervention.
A permuted blocks algorithm was performed using the Study
Randomizer software (Phase Locked Software) [34]. Participants
assigned to the IG received an email notifying them that they
were able to access the treatment, whereas participants who
were assigned to the CG received an email informing them that
they would be placed on a waitlist before being able to access
the treatment. After 36 days, the participants in the CG received
an email notifying them that they would now have access to the
treatment after responding to another assessment. More details
can be found in a study by Dominguez-Rodriguez et al [11].
Participants
Participants were recruited primarily through social media
networks and enrolled in the Grief COVID platform between
December 22, 2020, and November 17, 2021. Participants were
included if they met the following criteria:≥18 years of age;
internet access; a valid email address; basic digital skills; fluency
in Spanish; experience of the loss of a loved one within the last
6 months; and reported symptoms of depression, anxiety, or
stress (ie, total scores of the scales >0). Participants were
excluded if, at the time of the study, they met one or more of
the following criteria: they had a diagnosis of a psychotic
disorder; >6 months had passed since the loss of their loved
one; they received psychological and pharmacological treatment
at the time of the study; and they had a moderate to a high score
on the suicide risk scale, a recent suicide attempt (<3 months
before the assessment), or a diagnosis of PTSD. Contact
information for external psychological services was provided
on the platform, giving preference to the services that were free
of charge.
Intervention
Grief COVID is a free, web-based, self-applied, and
multicomponent intervention consisting of 12 modules
developed by psychologists with broad experience in clinical
and research settings and most of them with a PhD degree. It
was created using a user experience design and was incorporated
with techniques based on CBT [35], behavioral activation
therapy [36], mindfulness [37], and positive psychology [38].
The intervention consists of 12 sessions, delivered over a period
of 36 days. Each session was delivered in 2 formats, video or
text, and the participants could choose either format. Upon
completing each module, the participants were presented with
a 5-question quiz. Participants were required to answer at least
60% (3/5) of the questions correctly to progress to the next
module. Once a passing score was achieved, a message appeared
informing the user that after a 3-day period, the next module
would be available. Participants were recommended to progress
through 2 modules per week or 1 every 3 days. A detailed
description of the intervention is available in the protocol article
[11]. The web-based intervention can be accessed on the website
[33]. The main web page displays the institutions and
universities that made this intervention possible. Sample
screenshots of the web-based intervention are presented in
Multimedia Appendix 2.
Measures
The primary outcomes were related to grief, depression, PTSD,
hopelessness, anxiety, and suicide risk. The secondary outcome
was linked to the participant’s perception of the platform’s
usability.
The following instruments were used:
1. Inventory of Complicated Grief [39]: This is composed of
19 items, with a 5 Likert-type scale ranging from 0 to 4,
where 0=never, 1=rarely, 2=sometimes, 3=often, and
4=always. The items assess the frequency of each type of
symptom (emotional, cognitive, or behavioral). The scores
fluctuate between 0 and 76 points. Scores >25 are indicative
of complicated grief. The properties of the scale adaptation
to Spanish have good internal consistency results (Cronbach
α=.88). The version of Limonero García et al [40] was used
in this study.
2. Center for Epidemiologic Studies Depression Scale [41]:
This evaluates depression levels in the past 2 weeks. It
contains 20 questions and 4 possible answers: rarely or
never (<1 day), sometimes or rarely (1-2 days), occasionally
or a good part of the time (3-4 days), and most of the time
(5-7 days). This instrument has been validated in the
Mexican population (Cronbach α>.90) [42].
3. Posttraumatic Stress Disorder Symptom Scale (PSS) [43]:
This is a 17-item scale. The interviewer uses a 4-point scale:
0=not at all, 1=a little bit, 2=somewhat, and 3=very much.
The maximum possible score is 51 (severely affected), and
the minimum possible score is 0 (total absence of
symptoms). The diagnosis of PTSD is made when 1
symptom of re-experience, 3 symptoms of avoidance, and
2 symptoms of activation are observed [43]. The validated
version in Spanish was used in this study [44].
4. Beck Hopelessness Scale [45]: This comprises 20 items
with a true or false response option. The scores range from
0 to 20, with higher scores indicating a higher level of
hopelessness. It is a widely validated and used scale. For
J Med Internet Res 2023 | vol. 25 | e43839 | p. 4https://www.jmir.org/2023/1/e43839 (page number not for citation purposes)
Dominguez-Rodriguez et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
this study, the version validated in the Mexican population
was applied [46].
5. Generalized Anxiety Disorder-7 [47]: This is a brief scale
consisting of 7 items designed to measure the severity of
symptoms of generalized anxiety disorder during the last
2 weeks. A Likert format is used, with scores from 0 to 3,
and the maximum total score is 21. A score between 0 and
4 indicates that anxiety is not perceived, and a score
between 15 and 21 indicates severe perceived anxiety. The
Spanish version by García-Campayo et al [48] was used in
this study.
6. Plutchik Suicide Risk Scale (PSRS) [49]: This assesses the
risk of suicide through questions posed dichotomously (yes
or no), and a history of suicide attempts, suicidal ideation,
and suicide plans are considered. This scale establishes a
cutoff point of >6 that differentiates people at risk of suicide
from those who are not. It has good reliability (Cronbach
α=.74) and has been used in previous studies of the Mexican
population [50].
7. System Usability Scale (SUS) [51]: This validates the
usability of a system. It is composed of 10 items, which are
answered on a 5-point Likert-type scale concerning the
degree to which the respondent agrees with each statement
(1=strongly disagree to 5=strongly agree). To obtain the
global score of this scale, all the values obtained must be
added and multiplied by 2.5, resulting in a number between
0 and 100, which indicates the global value of this scale.
Sociodemographic data such as age, gender, country,
educational level, and employment status were requested.
Statistical Analyses
Data analyses were performed using SPSS (version 26; IBM
Corp). The normality of the data was examined using the
Kolmogorov-Smirnov test. None of the variables assessed in
this study followed normal distribution. Therefore,
nonparametric tests were used.
Sociodemographic characteristics of the sample between groups
(IG vs CG) were explored using the Mann-Whitney Utest (for
participants’ age) and Pearson χ2test (for comparisons by
gender, country, working status, and educational level).
Within-subject comparisons (pretest, posttest, and follow-up)
were calculated using the Wilcoxon signed-rank test. For
statistically significant changes, Hedges gwas used to measure
the effect sizes. Finally, the participants who completed the
intervention filled out a brief questionnaire to share their
opinions about the usability of the treatment. For each question,
the mean, SD, and range were calculated.
The final sample size obtained for this study was greater than
that necessary for the a priori power analysis performed. This
was performed using G*Power software (Universität Düsseldorf)
with the following parameters: mixed between-within-groups
ANOVA tests, Cronbach α=.05, and Cohen d=0.8 [11].
However, owing to the high dropout rate, the valid sample for
the estimation of treatment effectiveness at follow-up was
compromised. Thus, a post hoc sensitivity analysis was
performed for the 18 participants at follow-up: Wilcoxon
signed-rank test (matched pairs), Cronbach α=.05 (bilateral),
and effect size=0.93.
Ethics Approval
The Research Ethics Committee of the Autonomous University
of Ciudad Juárez, Mexico, approved the study (approval ID:
CEI-2020-2-226). Participation was voluntary, and all the
participants provided written informed consent to participate
in this study and for their data to be used for research purposes.
The study protocol is available in a study by
Dominguez-Rodriguez et al [11].
Results
Characteristics of the Participants and Dropout Rates
A total of 15,387 participants were assessed for eligibility. Of
the 15,387 participants, 14,278 (92.79%) were excluded from
the study. The remaining 7.21% (1109/15,387) of participants
were randomly assigned to the IG (n=882, 79.53%) and CG
(n=227, 20.47%). Ultimately, 10.28% (114/1109) of participants
completed the conditions (n=45, 39.5% completed the
intervention, and n=69, 60.5% completed the waitlist period;
Figure 1). The trial ended when the proposed sample size was
exceeded, and the follow-up period was also completed.
J Med Internet Res 2023 | vol. 25 | e43839 | p. 5https://www.jmir.org/2023/1/e43839 (page number not for citation purposes)
Dominguez-Rodriguez et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
Figure 1. CONSORT (Consolidated Standards of Reporting Trials) flow diagram of the study design.
Most participants were from Mexico (110/114, 96.5%). In all,
90.4% (103/114) were women, 8.8% (10/114) were men, and
0.9% (1/114) was a nonbinary person. Two-thirds of the
participants (77/114, 67.5%) reported being actively working.
Educational attainment was high, with 61.4% (70/114) of
participants reporting university degrees, 21.1% (24/114)
reporting a master’s degree, 14% (16/114) reporting a high
school diploma, 1.8% (2/114) reporting a PhD, and the
remaining 2 (1.8%) reporting other educational levels.
A total of 39.5% (45/114) of participants received treatment,
and 60.5% (69/114) of participants were assigned to the waitlist
CG. The sociodemographic characteristics of both groups were
similar, with the exception of age (Multimedia Appendix 3).
A large number of participants (treatment condition: 576/882,
65.3%; CG condition: 49/69, 71%) dropped out of the
intervention before beginning, without completing any of the
modules. A considerable dropout rate was observed among the
participants in module 1 (112/882, 12.7% of participants from
the IG). More details are provided in Table 1.
J Med Internet Res 2023 | vol. 25 | e43839 | p. 6https://www.jmir.org/2023/1/e43839 (page number not for citation purposes)
Dominguez-Rodriguez et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
Table 1. Detailed dropout rates by participants and conditions.
Control participants (n=69), n (%)Treatment participants (n=882), n (%)Last module accessed
49 (71)576 (65.3)Assigned to the intervention but did not access any module
3 (4.3)112 (12.7)1
3 (4.3)57 (6.5)2
4 (5.8)18 (2)3
1 (1.4)21 (2.4)4
0 (0)11 (1.2)5
2 (2.9)9 (1)6
0 (0)10 (1.1)7
0 (0)4 (0.5)8
0 (0)6 (0.7)9
0 (0)9 (1)10
1 (1.4)4 (0.5)11
6 (8.7)45 (5.1)12
The mean age of the treatment group was 45.7 (SD 10.5) years,
with participants’ages ranging from 22 to 62 years. In contrast,
the mean age for the CG was 34.4 (SD 10.4) years, ranging
from 21 to 58 years.
Comparison of Clinical Symptoms Between Groups
at the Pretest
Table 2 depicts the sample clinical characteristics at the pretest
assessment. No statistically significant differences were found
between the IG and CG, with the exception of a higher score
for PTSD symptoms in the CG.
Table 2. Dependent variables at the pretest measured by study condition.
Pvaluea
Median (IQR)Participants, n (%)Instrument
TreatmentControlTreatment (n=45)Control (n=69)
.1512 (11.5-17)22 (18-32)45 (100)69 (100)
ICGb
.4817 (10-24)15 (11-22)41 (91)63 (91)
CESD-Rc
.003e
9 (6-13)13 (9-15)45 (100)69 (100)
PSSd
.333 (1-5)3 (2-6)45 (100)69 (100)
BHSf
.997 (4-8)6 (4-10)45 (100)69 (100)
GAD-7g
.853 (1-6)3 (2-5)45 (100)69 (100)
PSRSh
aPvalue for the Mann-Whitney Utest.
bICG: Inventory of Complicated Grief.
cCESD-R: Center for Epidemiologic Studies Depression Scale.
dPSS: Posttraumatic Stress Disorder Symptom Scale.
eSignificant Pvalue.
fBHS: Beck Hopelessness Scale.
gGAD-7: Generalized Anxiety Disorder-7.
hPSRS: Plutchik Suicide Risk Scale.
The Effects of the Intervention Versus the Waitlist on
Clinical Symptoms: Comparisons Between Groups
We found pronounced differences between the IG and CG on
all the measures examined (Table 3) after the participants had
completed the intervention or the waitlist period. Moreover, the
gap between the PSS scores between the groups further widened.
J Med Internet Res 2023 | vol. 25 | e43839 | p. 7https://www.jmir.org/2023/1/e43839 (page number not for citation purposes)
Dominguez-Rodriguez et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
Table 3. Differences between intervention group and control group comparing treatment versus waiting list.
Pvaluea
Median (IQR)Participants, n (%)Instrument
TreatmentControlTreatment (n=45)Control (n=69)
<.001 c
10 (5-14)20 (13-31)43 (96)51 (74)
ICGb
<.001 c
6 (4-10)14 (11-17)45 (100)65 (94)
CESD-Rd
<.001 c
5 (3-9)11 (7-17)43 (96)51 (74)
PSSe
.005 c
0 (0-1)1 (0-3)43 (96)51 (74)
BHSf
<.001 c
2 (1-5)7 (4-11)43 (96)51 (74)
GAD-7g
<.001 c
2 (1-3)4 (2-5)44 (98)54 (78)
PSRSh
aPvalue for the Mann-Whitney Utest.
bICG: Inventory of Complicated Grief.
cSignificant Pvalues.
dCESD-R: Center for Epidemiologic Studies Depression Scale.
ePSS: Posttraumatic Stress Disorder Symptom Scale.
fBHS: Beck Hopelessness Scale.
gGAD-7: Generalized Anxiety Disorder-7.
hPSRS: Plutchik Suicide Risk Scale.
The Efficacy of the Intervention: Changes Within
Subjects
The IG showed clinical improvement in all the measures
assessed (Table 4). Psychological treatment had larger effect
sizes on depression scores (1.3), hopelessness (0.9), grief (0.7),
anxiety (0.7), and suicide risk (0.5). To a lesser extent, it also
affected posttraumatic stress symptoms (0.3). It is worth
highlighting that whereas the IG showed a medium-sized
decrease in the PSRS score (ie, a reduction of symptoms), the
CG exhibited a statistically significant increase in the suicide
scale score after the waitlist period (P=.004), albeit small in
size. The CG also reported a decrease in hopelessness scores,
with a medium-sized effect (0.6).
J Med Internet Res 2023 | vol. 25 | e43839 | p. 8https://www.jmir.org/2023/1/e43839 (page number not for citation purposes)
Dominguez-Rodriguez et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
Table 4. Changes in within-subjects measures by study condition.
Pvaluea
Hedges g(95% CI)Posttest measure (intervention vs waiting
list)
Pretest
Treatment
(n=45)
Control
(n=69)
Treat-
ment
(n=45)
Control
(n=69)
Median (IQR)Participants, n (%)Median (IQR)Participants, n (%)
Treat-
ment
(n=45)
Control
(n=69)
Treat-
ment
(n=45)
Control
(n=69)
Treat-
ment
(n=45)
Control
(n=69)
Treat-
ment
(n=45)
Control
(n=69)
<.001 d
.390.7 (0.4
to 1.1)
N/Ac
10 (5-17)20 (13-
31)
43 (96)51 (74)12 (11.5-
17)
22 (18-
32)
45 (100)69 (100)
ICGb
<.001d
.101.3 (0.9
to 1.8)
N/A6 (4-10)14 (11-
17)
45 (100)65 (94)17 (10-
24)
15 (11-
22)
41 (91)63 (91)
CESD-Re
.006 d
.550.4 (0.1
to 0.8)
N/A5 (3-9)11 (7-
17)
43 (96)51 (74)9 (6-13)13 (9-
15)
45 (100)69 (100)
PSSf
<.001 d
<.001 d
0.9 (0.5
to 1.4)
0.6 (0.3
to 0.9)
0 (0-1)1 (0-3)43 (96)51 (74)3 (1-5)3 (2-6)45 (100)69 (100)
BHSg
<.001 d
.620.7 (0.4
to 1.2)
N/A2 (1-5)7 (4-11)43 (96)51 (74)7 (4-8)6 (4-10)45 (100)69 (100)
GAD-7h
.004 d
.004 d
0.5 (0.2
to 0.8)
−0.3
(−0.6 to
−0.1)
2 (1-3)4 (2-5)44 (98)54 (78)3 (1-6)3 (2-5)45 (100)69 (100)
PSRSi
aPvalue for Wilcoxon signed-rank test.
bICG: Inventory of Complicated Grief.
cN/A: not applicable; Hedges gvalues are reported for comparisons with significant Pvalues.
dSignificant Pvalues.
eCESD-R: Center for Epidemiologic Studies Depression Scale.
fPSS: Posttraumatic Stress Disorder Symptom Scale.
gBHS: Beck Hopelessness Scale.
hGAD-7: Generalized Anxiety Disorder-7.
iPSRS: Plutchik Suicide Risk Scale.
Changes Maintained at Follow-up: The Efficacy After
3 Months
Regarding the 3-month follow-up, all achievements obtained
at the posttreatment measure by the IG were sustained (Table
5). Beck Hopelessness Scale, PSS, Generalized Anxiety
Disorder-7, and PSRS scores maintained the overall values that
reached posttest measures. Center for Epidemiologic Studies
Depression Scale and Inventory of Complicated Grief scores
decreased further after 3 months. Although some participants
were randomly assigned to the waitlist CG and were able to
access the intervention after completing the waiting period, a
reduced number (6/69, 9%) of the CG participants completed
the entire intervention.
J Med Internet Res 2023 | vol. 25 | e43839 | p. 9https://www.jmir.org/2023/1/e43839 (page number not for citation purposes)
Dominguez-Rodriguez et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
Table 5. Changes after treatment until follow-up period for the intervention group.
Pvalueb
3-month follow-up,
median (IQR)
Pvaluea
After treatment,
median (IQR)
Before treatment,
median (IQR)
Participants
(n=18), n (%)
Instrument
<.001 d
8 (5-10)
.001 d
12 (7-18)22.5 (15-29)18 (100)
ICGc
.001 d
2 (5-8.5)
.001 d
6.5 (5-10)16 (12-20)18 (100)
CESD-Re
.003 d
4.5 (2-9)
.008 d
5 (3-8)9 (7-11)18 (100)
BHSf
<.001 d
0.5 (0-1)
<.001 d
0 (0-1)4.5 (3-7)18 (100)
PSSg
.002 d
4 (1-6)
.001 d
2.5 (2-4)5 (4-8)18 (100)
GAD-7h
.001 d
1 (1-3)
.001 d
1.5 (1-2)4 (3-6)18 (100)
PSRSi
aPvalue for Wilcoxon signed-rank test between pretreatment and posttreatment measures.
bPvalue for Wilcoxon signed-rank tests between pretreatment and 3-month follow-up measures.
cICG: Inventory of Complicated Grief.
dSignificant Pvalues.
eCESD-R: Center for Epidemiologic Studies Depression Scale.
fBHS: Beck Hopelessness Scale.
gPSS: Posttraumatic Stress Disorder Symptom Scale.
hGAD-7: Generalized Anxiety Disorder-7.
iPSRS: Plutchik Suicide Risk Scale.
The Usability of Grief COVID Intervention
All 100% (45/45) of participants who completed the treatment
filled out the SUS. Overall, reviews were quite positive, with
a quarter of participants (11/45, 24%) selecting the highest score
(ie, reporting good usability) in all items and two-thirds of
participants (30/45, 67%) giving the highest rating to 8 of 10
items (Table 6).
Table 6. Results of System Usability Scale (n=45).
Ranks, median (IQR; range)
5.0 (4.0-5.0; 1.0-5.0)1. “I think that I would like to use this system frequently”
1.0 (1.0-1.0; 1.0-5.0)
2. “I found the system unnecessarily complex”a
5.0 (5.0-5.0; 1.0-5.0)3. “I thought the system was easy to use”
5.0 (5.0-5.0; 1.0-5.0)
4. “I think that I would need the support of a technical person to be able to use this system”a
5.0 (4.0-5.0; 1.0-5.0)5. “I found that various functions in this system were well integrated”
1.0 (1.0-1.0; 1.0-5.0)
6. “I thought there was too much inconsistency in this system”a
5.0 (5.0-5.0; 1.0-5.0)7. “I imagine that most people would learn to use this system very quickly”
1.0 (1.0-2.0; 1.0-5.0)
8. “I found the system very cumbersome to use”a
5.0 (5.0-5.0; 3.0-5.0)9. “I felt very confident in using the system”
1.0 (1.0-1.0; 1.0-5.0)
10. “I needed to learn a lot of things before I could get going with this system”a
aItems are negatively worded (ie, lower scores show greater usability).
Discussion
Principal Findings
The objective of this study was to present the efficacy of Grief
COVID in decreasing the risk of developing CGD and in
reducing symptoms of anxiety, depression, hopelessness, PTSD,
and risk of suicide.
The findings of this study showed a reduction in depression,
hopelessness, grief, anxiety, and suicide risk and a lesser effect
on posttraumatic stress scores in the IG. The results were
maintained at the 3-month postintervention follow-up. These
results reflect a statistically significant difference compared
with the CG scores, which showed an increase in suicide risk
after the waitlist period.
Proposals for grief intervention during the COVID-19 pandemic
are scarce. Tur et al [29] performed an intervention called “The
GROw program,” which evaluated depression, symptoms of
grief, and typical grief beliefs, along with daily measures of
symptom frequency and intensity using the Emotional Monitor
App. Of 6 participants, 4 completed the intervention; of them,
J Med Internet Res 2023 | vol. 25 | e43839 | p. 10https://www.jmir.org/2023/1/e43839 (page number not for citation purposes)
Dominguez-Rodriguez et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
only 2 obtained a clinically significant change in the
bereavement measures. Second, Reitsma et al [27] performed
an RCT to examine the short-term effects of an unguided
web-based grief-specific CBT in treating persistent complex
bereavement disorder, PTSD, and depression symptoms. A total
of 65 Dutch adults participated, all of whom had experienced
the death of a loved one at least 3 months before the start of the
study during the COVID-19 pandemic. Similar to our results,
their findings revealed significant effects for reductions in
disturbed grief symptoms and moderate between-group effects
for PTSD and depression symptoms.
One of the few RCTs that sought to alleviate the psychological
distress caused by COVID-19 is a study by Brog et al [52] in
Germany. They developed a web-based self-help intervention
based on CBT and psychoeducation. Although their study did
not show positive effects on the reduction of symptoms of
depression, anxiety, and stress, the intervention was found to
be suitable for preventive purposes to help individuals cope
with psychological distress. The results showed increased
emotion regulation and resilience in participants [52]. In most
cases, publications on the effectiveness of grief intervention
protocols in the context of the COVID-19 pandemic are yet to
be published [26,28].
The Grief COVID intervention was designed using techniques
that draw on CBT, behavioral activation therapy, mindfulness,
and positive psychology. Most studies to date on grief
intervention during the COVID-19 pandemic used CBT based
on its known effectiveness in treating grief [16,28,29]. These
studies also incorporated behavioral activation, and a study
conducted by Tur et al [29] included mindfulness activities as
well. No studies were found that treated grief using the
principles of positive psychology during the COVID-19
pandemic.
Important issues to consider stem from the unique circumstances
that people have experienced during the COVID-19 pandemic.
For example, face-to-face contact was restricted, and a large
part of society was affected by the economic impact of the
pandemic. In this context, web-based interventions have a
promising potential for several reasons. First, treating people
in this way can reduce the time required to access support
services. Second, the cost of web-based treatments may be more
affordable for much of the population [10]. A review by Harrop
et al [53] describes the recommendations for grief intervention
during a disaster. The literature highlights the need for increased
psychoeducation regarding grief and available support services
for bereaved individuals, as well as screening for the risk of
CGD, and the provision of professional mental health support
for those with a high risk of CGD. These tenets were integrated
into our intervention; all participants completed an initial
screening on the platform (results of the pre-evaluation
questionnaires can be consulted in the study by
Dominguez-Rodriguez et al [5]), which then generated a free
psychoeducational resource for bereavement during the
pandemic and provided additional contact information for
relevant mental health services and professionals. The secondary
objective of this study was to validate the usability of the
self-applied intervention system. Most participants who finished
the intervention (41/45, 91%) reported a high level of
satisfaction with the platform. The positive ratings of our
intervention system may be attributed, in part, to the
user-centered design that accounted for the specific needs and
characteristics of the intended users [54]. Laranjeira et al [10]
suggested an urgent need for evidence to evaluate the feasibility,
effectiveness, and acceptability of web-based interventions in
conventional bereavement situations, as well as disaster contexts.
Our study contributes to the evidence that begins to lessen this
knowledge gap.
The results of this study have several implications for health
care providers and local governments. First, web-based
interventions have been shown to be effective in reducing
psychological distress, including the symptoms of grief,
depression, anxiety, and posttraumatic stress [27,29,52]. Second,
this treatment modality represents a viable option for people
who may have difficulty accessing specialized face-to-face care
because of a lack of proximity to services where they live or
other reasons. In addition, the costs associated with web-based
interventions are lower for public institutions when compared
with individual face-to-face interventions, as web-based
treatments require only the initial economic investment in and
maintenance of the platform [10].
This study does have some limitations, the first of which is
related to the dropout rate. Only 5.1% (45/882) of the IG
participants completed the intervention. It is not uncommon for
internet-based and self-administered interventions to have high
dropout rates, particularly among participants assigned to the
waitlist group. This may be due to reasons such as a preference
for face-to-face treatment or lack of internet access or a digital
device on which to access the platform. Another consideration
is that 12 modules may be too long for a preventive intervention
of this type. However, similar intervention proposals have
ranged from 8 to 12 sessions [16,28]. To reduce the incidence
of dropouts, further studies should send weekly notifications to
users to encourage them to continue with the intervention. In
addition, a survey might be sent to participants who decide not
to complete the entire program to better understand their reasons
and improve upon these points in future studies. Specifically,
all the participants on the CG condition completed the
pre-evaluation questionnaire and then proceeded to complete a
waiting period of 36 days. Once they finished the waiting period,
they were requested to repeat the evaluation to assess their
symptomatology and confirm that they still did not fulfill any
exclusion criteria. Once this was achieved, they were granted
access to the intervention. However, most participants (63/69,
91%) assigned to this group did not finish the intervention, as
only just 6 participants completed the intervention and one of
them answered the follow-up measure. Therefore, we did not
report postevaluation data from the CG. Future studies should
consider ways to encourage adherence to treatment for those
assigned to the waitlist, and this could help reduce the dropout
rate. Another limitation of this study is that there were
substantially more women than men in both the CG and the IG.
This phenomenon has been observed in other similar studies
[27,29]. An additional limitation of this study is that to date,
few studies regarding web-based interventions to prevent CGD
exist, and there is little evidence to suggest that an unguided
intervention can be as effective as a guided intervention in
J Med Internet Res 2023 | vol. 25 | e43839 | p. 11https://www.jmir.org/2023/1/e43839 (page number not for citation purposes)
Dominguez-Rodriguez et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
treating grief. However, there is existing evidence that suggests
that for the treatment of mild symptoms of depression [55] or
anxiety [56], unguided web-based interventions have similar
outcomes to guided interventions, particularly in the medium
term (3- and 6-month follow-ups). Nevertheless, further research
is needed to understand the mechanisms of change and variables
associated with web-based treatment outcomes.
Another limitation of the study is that because the intervention
was web-based, unguided, and preventive in nature, we did not
consider it ethical to include individuals who reported substantial
suicidality (ie, suicidal intent or plans). This limited external
validity, particularly by decreasing the proportion of severely
depressed participants in the sample. A further limitation is that
to evaluate the results of the intervention, only the participants
who completed the intervention could access the
postmeasurement assessment. Evaluating participants only at
the posttreatment point, and not evaluating them during the
intervention, limits our knowledge regarding the treatment’s
efficacy at different stages of the intervention. Future studies
should include assessment measures at several points throughout
the intervention to provide a more complete picture of the
treatment’s efficacy at different stages. In this regard, we have
conducted 2 studies (Dominguez-Rodriguez, A, unpublished
data, March 2023) that help us to better understand the reasons
some participants continued with the intervention, whereas some
others dropped out. In the first study, the participants who
dropped out of the intervention were invited to respond to the
SUS, along with ad hoc instruments designed to allow us to
analyze the participants’ reasons for abandoning the
intervention. The study was conducted using 3 psychological
intervention platforms that were available during the COVID-19
pandemic: Salud Mental COVID-19 [57], Duelo COVID-19
[33], and Personal COVID-19 [58]. In addition, we conducted
a qualitative study with the Grief COVID participants to gather
information about their perceptions of the intervention. The
study included participants who completed the intervention as
well as those who dropped out. In addition, in this qualitative
study, we further explored the important harms or unintended
effects in each group while observing none.
The results of these studies will be published in the later articles
and will help us understand the underlying causes of the dropout
rates (Table 1).
Another limitation is that one of the exclusion criteria was that
the participants have a diagnosis of a psychotic disorder.
However, no specific instrument was included to assess this
criterion, apart from asking the participants if they were taking
any form of medication, which could indicate if they were
consuming an antipsychotic drug. Future studies similar to ours
might consider including assessment measures for psychosis as
a part of the intervention. For example, Rüegg et al [59]
implemented a self-applied intervention similar to Grief COVID
and did include these types of assessments.
Regarding the strengths of this study, to the best of our
knowledge, this is the first study to publish the results of an
intervention aimed at preventing CGD in Latin America.
Furthermore, this study is one of a small number of studies that
examined grief in the context of the COVID-19 pandemic. In
addition, the sample size of our study is sufficiently large to
show statistically significant results, and the study included data
from 3 time points. The platform was created using principles
of responsive design, which permitted the intervention to be
accessed on devices of different screen sizes, including cell
phones, tablets, and computers [11]. This is one of the few RCTs
of web-based interventions developed and delivered in Latin
America, where such interventions, although greatly needed,
are still scarce [60]. Finally, Grief COVID is the only web-based
intervention implemented for bereaved people in the Latin
American region during the COVID-19 pandemic.
Conclusions
The self-applied web-based intervention Grief COVID was
efficient in reducing the symptomatology of anxiety, depression,
hopelessness, suicide risk, PTSD, and CGD. Grief COVID was
evaluated by the participants who answered the system usability
questionnaire as easy to use. The results of this study are highly
relevant to professionals in other countries who wish to design,
develop, and deliver interventions similar to Grief COVID to
provide grief-related support to the general population and
decrease the risk of developing CGD and associated clinical
correlates.
Acknowledgments
The authors want to thank graphic designers and animators, Javier Velazquez, Edgar Arenas, Karine Tovanche, Alexia Apodaca,
Cinthia Holguín, Mariana Martínez, Pedro Gutierrez, and Mitzi Valencia, who created the videos and made more attractive
contents for this intervention. The authors would also like to thank Joabián Alvarez Silva, the developer for this intervention, and
Dr Flor Rocío Ramírez Martínez for obtaining the funding to conduct the project. The authors also thank Dr Paulina Arenas
Landgrave, Sofía Martínez Luna, and Dr María Jesús Hernández Jiménez for their collaboration in the development of the
theoretical contents of the intervention. The Autonomous University of Ciudad Juárez provided funds for the development of the
Grief COVID platform to provide support to the Mexican population that lost someone to COVID-19 or to any other reasons
during the last 6 months. The authors would also like to thank Tecnológico de Monterrey for their support of the manuscript fees.
The funding body had no role in the study design, writing of the manuscript, or the decision to submit the paper to a specific
journal.
Data Availability
The data sets generated and analyzed during this study are available from the corresponding author on reasonable request.
J Med Internet Res 2023 | vol. 25 | e43839 | p. 12https://www.jmir.org/2023/1/e43839 (page number not for citation purposes)
Dominguez-Rodriguez et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
Authors' Contributions
AD-R conceived the original idea and supervised all project steps. AD-R, SS-G, and PEH-A accessed and verified the underlying
data and took responsibility for the integrity of the data and accuracy of the corresponding analysis. AD-R and AdlR-G designed
the study and original protocol. AD-R, SS-G, PEH-A, LPGR, JOG-C, VM-A, LETG, AdlR-G, and MM wrote the manuscript.
All the authors contributed to the manuscript and approved the submitted version. LETG and VM-A are currently not affiliated
with any institution, but are Independent Researchers.
Conflicts of Interest
None declared.
Multimedia Appendix 1
CONSORT-eHEALTH checklist (V 1.6.1).
[PDF File (Adobe PDF File), 55397 KB-Multimedia Appendix 1]
Multimedia Appendix 2
Sample screenshots of the web-based intervention.
[PDF File (Adobe PDF File), 625 KB-Multimedia Appendix 2]
Multimedia Appendix 3
Sociodemographic characteristics of participants.
[PDF File (Adobe PDF File), 75 KB-Multimedia Appendix 3]
References
1. WHO coronavirus (COVID-19) dashboard. World Health Organization. URL: https://covid19.who.int/ [accessed 2022-07-07]
2. COVID-19 - PAHO/WHO response report 78. Pan American Health Organization. Washington, DC, USA: PAHO/WHO;
2022. URL: https://iris.paho.org/handle/10665.2/56137 [accessed 2022-06-17]
3. COVID-19 Mexico Información general. Secretaria de Salud. 2022. URL: https://datos.covid-19.conacyt.mx/ [accessed
2022-07-07]
4. Dashti S, Fathi Najafi T, Mohammadzadeh F, Rezaei Kalat A, Bahri N. Knowledge level of health care providers about
complicated grief during the COVID-19 pandemic. Iran J Psychiatry 2022 Apr;17(2):154-161 [FREE Full text] [doi:
10.18502/ijps.v17i2.8905] [Medline: 36262764]
5. Dominguez-Rodriguez A, Herdoiza-Arroyo PE, Martínez Arriaga RJ, Bautista Valerio E, Mateu Mollá J, de la Rosa-Gómez
A, et al. Prevalence of anxiety symptoms and associated clinical and sociodemographic factors in Mexican adults seeking
psychological support for grief during the COVID-19 pandemic: a cross-sectional study. Front Psychiatry 2022 Mar
14;13:749236 [FREE Full text] [doi: 10.3389/fpsyt.2022.749236] [Medline: 35370841]
6. Martínez Arriaga RJ, González Ramírez LP, de la Roca-Chiapas JM, Hernández-González M. Psychological distress of
COVID-19 pandemic and associated psychosocial factors among Mexican students: an exploratory study. Psychol Sch
2021 Sep;58(9):1844-1857 [FREE Full text] [doi: 10.1002/pits.22570] [Medline: 34538936]
7. González Ramírez LP, Martínez Arriaga RJ, Hernández-Gonzalez MA, De la Roca-Chiapas JM. Psychological distress
and signs of post-traumatic stress in response to the COVID-19 health emergency in a Mexican sample. Psychol Res Behav
Manag 2020 Jul 22;13:589-597 [FREE Full text] [doi: 10.2147/PRBM.S259563] [Medline: 32801956]
8. Goveas JS, Shear MK. Grief and the COVID-19 pandemic in older adults. Am J Geriatr Psychiatry 2020
Oct;28(10):1119-1125 [FREE Full text] [doi: 10.1016/j.jagp.2020.06.021] [Medline: 32709542]
9. Wallace CL, Wladkowski SP, Gibson A, White P. Grief during the COVID-19 pandemic: considerations for palliative care
providers. J Pain Symptom Manage 2020 Jul;60(1):e70-e76 [FREE Full text] [doi: 10.1016/j.jpainsymman.2020.04.012]
[Medline: 32298748]
10. Laranjeira C, Moura D, Salci MA, Carreira L, Covre E, Jaques A, et al. A scoping review of interventions for family
bereavement care during the COVID-19 pandemic. Behav Sci (Basel) 2022 May 19;12(5):155 [FREE Full text] [doi:
10.3390/bs12050155] [Medline: 35621452]
11. Dominguez-Rodriguez A, Martínez-Luna SC, Hernández Jiménez MJ, De La Rosa-Gómez A, Arenas-Landgrave P, Esquivel
Santoveña EE, et al. A self-applied multi-component psychological online intervention based on UX, for the prevention of
complicated grief disorder in the Mexican population during the COVID-19 outbreak: protocol of a randomized clinical
trial. Front Psychol 2021 Mar 29;12:644782 [FREE Full text] [doi: 10.3389/fpsyg.2021.644782] [Medline: 33854466]
12. Varshney P, Prasad G, Chandra PS, Desai G. Grief in the COVID-19 times: are we looking at complicated grief in the
future? Indian J Psychol Med 2021 Jan;43(1):70-73 [FREE Full text] [doi: 10.1177/0253717620985424] [Medline: 34349310]
J Med Internet Res 2023 | vol. 25 | e43839 | p. 13https://www.jmir.org/2023/1/e43839 (page number not for citation purposes)
Dominguez-Rodriguez et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
13. Adiukwu F, Kamalzadeh L, Pinto da Costa M, Ransing R, de Filippis R, Pereira-Sanchez V, et al. The grief experience
during the COVID-19 pandemic across different cultures. Ann Gen Psychiatry 2022 Jun 14;21(1):18 [FREE Full text] [doi:
10.1186/s12991-022-00397-z] [Medline: 35701763]
14. Solomon RM, Hensley BJ. EMDR therapy treatment of grief and mourning in times of COVID-19 (Coronavirus). J EMDR
Prac Res 2020 Jul 29;14(3):162-174. [doi: 10.1891/emdr-d-20-00031]
15. Mortazavi SS, Assari S, Alimohamadi A, Rafiee M, Shati M. Fear, loss, social isolation, and incomplete grief due to
COVID-19: a recipe for a psychiatric pandemic. Basic Clin Neurosci 2020;11(2):225-232 [FREE Full text] [doi:
10.32598/bcn.11.covid19.2549.1] [Medline: 32855782]
16. Reitsma L, Boelen PA, de Keijser J, Lenferink LI. Online treatment of persistent complex bereavement disorder, posttraumatic
stress disorder, and depression symptoms in people who lost loved ones during the COVID-19 pandemic: study protocol
for a randomized controlled trial and a controlled trial. Eur J Psychotraumatol 2021 Nov 24;12(1):1987687 [FREE Full
text] [doi: 10.1080/20008198.2021.1987687] [Medline: 34868479]
17. Tang S, Xiang Z. Who suffered most after deaths due to COVID-19? Prevalence and correlates of prolonged grief disorder
in COVID-19 related bereaved adults. Global Health 2021 Feb 11;17(1):19 [FREE Full text] [doi:
10.1186/s12992-021-00669-5] [Medline: 33573673]
18. Torrens-Burton A, Goss S, Sutton E, Barawi K, Longo M, Seddon K, et al. 'It was brutal. It still is': a qualitative analysis
of the challenges of bereavement during the COVID-19 pandemic reported in two national surveys. Palliat Care Soc Pract
2022 Apr 19;16:26323524221092456 [FREE Full text] [doi: 10.1177/26323524221092456] [Medline: 35462622]
19. Lundorff M, Holmgren H, Zachariae R, Farver-Vestergaard I, O'Connor M. Prevalence of prolonged grief disorder in adult
bereavement: a systematic review and meta-analysis. J Affect Disord 2017 Apr 01;212:138-149. [doi:
10.1016/j.jad.2017.01.030] [Medline: 28167398]
20. Lobb EA, Kristjanson LJ, Aoun SM, Monterosso L, Halkett GK, Davies A. Predictors of complicated grief: a systematic
review of empirical studies. Death Stud 2010 Sep;34(8):673-698. [doi: 10.1080/07481187.2010.496686] [Medline: 24482845]
21. Payàs Puigarnau A. El mensaje de las lágrimas: Una guía para superar la pérdida de un ser querido. Barcelona, Spain:
Ediciones Paidós; 2014.
22. Boelen PA, Eisma MC, Smid GE, Keijser JD, Lenferink LI. Remotely delivered cognitive behavior therapy for disturbed
grief during the COVID-19 crisis: challenges and opportunities. J Loss Trauma 2021;26(3):211-219. [doi:
10.1080/15325024.2020.1793547]
23. Breen LJ. Harnessing social support for bereavement now and beyond the COVID-19 pandemic. Palliat Care Soc Pract
2021 Feb 17;15:2632352420988009 [FREE Full text] [doi: 10.1177/2632352420988009] [Medline: 34104884]
24. Hamid W, Jahangir MS. Dying, death and mourning amid COVID-19 pandemic in Kashmir: a qualitative study. Omega
(Westport) 2022 Aug;85(3):690-715. [doi: 10.1177/0030222820953708] [Medline: 32862778]
25. Zhai Y, Du X. Loss and grief amidst COVID-19: a path to adaptation and resilience. Brain Behav Immun 2020 Jul;87:80-81
[FREE Full text] [doi: 10.1016/j.bbi.2020.04.053] [Medline: 32335197]
26. Borghi L, Menichetti J, Vegni E, Early Bereavement Psychological Intervention working group. A phone-based early
psychological intervention for supporting bereaved families in the time of COVID-19. Front Public Health 2021 Feb
2;9:625691 [FREE Full text] [doi: 10.3389/fpubh.2021.625691] [Medline: 33604325]
27. Reitsma L, Boelen PA, de Keijser J, Lenferink LI. Self-guided online treatment of disturbed grief, posttraumatic stress,
and depression in adults bereaved during the COVID-19 pandemic: a randomized controlled trial. Behav Res Ther 2023
Apr;163:104286 [FREE Full text] [doi: 10.1016/j.brat.2023.104286] [Medline: 36906949]
28. Tang R, Xie T, Jiao K, Xu X, Zou X, Qian W, et al. Grief reactions and grief counseling among bereaved Chinese individuals
during COVID-19 pandemic: study protocol for a randomized controlled trial combined with a longitudinal study. Int J
Environ Res Public Health 2021 Aug 27;18(17):9061 [FREE Full text] [doi: 10.3390/ijerph18179061] [Medline: 34501650]
29. Tur C, Campos D, Suso-Ribera C, Kazlauskas E, Castilla D, Zaragoza I, et al. An internet-delivered cognitive-behavioral
therapy (iCBT) for prolonged grief disorder (PGD) in adults: a multiple-baseline single-case experimental design study.
Internet Interv 2022 Sep;29:100558 [FREE Full text] [doi: 10.1016/j.invent.2022.100558] [Medline: 35865996]
30. Menichetti Delor JP, Borghi L, Cao di San Marco E, Fossati I, Vegni E. Phone follow up to families of COVID-19 patients
who died at the hospital: families' grief reactions and clinical psychologists' roles. Int J Psychol 2021 Aug;56(4):498-511
[FREE Full text] [doi: 10.1002/ijop.12742] [Medline: 33511652]
31. Worden JW. Grief Counseling and Grief Therapy: A Handbook for the Mental Health Practitioner. 3rd edition. Cham,
Switzerland: Springer; 2002.
32. Bates U, Brennan J, Keegan O. Implementation and evaluation of a National Bereavement Support line in response to the
COVID-19 pandemic in Ireland. J Public Health (Oxf) (forthcoming) 2022 Oct 30:fdac119 [FREE Full text] [doi:
10.1093/pubmed/fdac119] [Medline: 36310507]
33. Dominguez-Rodriguez A, Hernández-Jiménez MJ, Martínez Luna SC, de la Rosa A, Landgrave PA, Esquivel Santoveña
EE, et al. Duelo COVID-19. 2020. URL: https://www.duelocovid.com/ [accessed 2022-06-17]
34. Study Randomizer. URL: https://app.studyrandomizer.com/ [accessed 2020-12-01]
J Med Internet Res 2023 | vol. 25 | e43839 | p. 14https://www.jmir.org/2023/1/e43839 (page number not for citation purposes)
Dominguez-Rodriguez et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
35. Kersting A, Dölemeyer R, Steinig J, Walter F, Kroker K, Baust K, et al. Brief internet-based intervention reduces
posttraumatic stress and prolonged grief in parents after the loss of a child during pregnancy: a randomized controlled trial.
Psychother Psychosom 2013;82(6):372-381. [doi: 10.1159/000348713] [Medline: 24061387]
36. Osiris Pazarán RM, Guerrero Uribe UG, Herrera Melo A, Ascencio Huertas L, Rodríguez Lugo ME, Aguilar Salas I, et al.
Manual operativo del curso para el acompañamiento del duelo en situaciones especiales durante la pandemia de la COVID-19
en México [Operational Manual of the Course for the Accompaniment of Grief in Special Situations During the COVID-19
Pandemic in Mexico]. Secretaría de Salud. 2020. URL: http://inprf.gob.mx/ensenanzanew/archivos/2020/manual_duelo.
pdf [accessed 2022-07-07]
37. Goldberg SB, Tucker RP, Greene PA, Simpson TL, Kearney DJ, Davidson RJ. Is mindfulness research methodology
improving over time? A systematic review. PLoS One 2017 Oct 31;12(10):e0187298 [FREE Full text] [doi:
10.1371/journal.pone.0187298] [Medline: 29088283]
38. Carr A, Cullen K, Keeney C, Canning C, Mooney O, Chinseallaigh E, et al. Effectiveness of positive psychology interventions:
a systematic review and meta-analysis. J Posit Psychol 2021;16(6):749-769. [doi: 10.1080/17439760.2020.1818807]
39. Prigerson HG, Bierhals AJ, Kasl SV, Reynolds 3rd CF, Shear MK, Newsom JT, et al. Complicated grief as a disorder
distinct from bereavement-related depression and anxiety: a replication study. Am J Psychiatry 1996 Nov;153(11):1484-1486.
[doi: 10.1176/ajp.153.11.1484] [Medline: 8890686]
40. Limonero Garcia JT, Lacasta Reverte M, García García J, Maté Méndez J, Prigerson HG. Adaptación al castellano del
inventario de duelo complicado. Med Pal 2009;16(5):291-297 [FREE Full text]
41. Eaton WW, Smith C, Ybarra M, Muntaner C, Tien A. Center for epidemiologic studies depression scale: review and revision
(CESD and CESD-R). In: Maruish ME, editor. The use of Psychological Testing for Treatment Planning and Outcomes
Assessment: Instruments for Adults. Mahwah, NJ, USA: Lawrence Erlbaum Associates Publishers; 2004:363-377.
42. González-Forteza C, Torres CS, Jimenez A, Hernández IP, González-González A, Juarez F, et al. Reliability and validity
of the depression scale CES-D in high school and college students from Mexico City: results from a census. Salud Ment
(Mex) 2011 Jan;34(1):53-59.
43. Foa EB, Cashman L, Jaycox L, Perry K. The validation of a self-report measure of posttraumatic stress disorder: the
posttraumatic diagnostic scale. Psychol Assess 1997 Dec;9(4):445-451. [doi: 10.1037/1040-3590.9.4.445]
44. Novy DM, Stanley MA, Averill P, Daza P. Psychometric comparability of English- and Spanish-language measures of
anxiety and related affective symptoms. Psychol Assess 2001 Sep;13(3):347-355. [doi: 10.1037//1040-3590.13.3.347]
[Medline: 11556271]
45. Beck AT, Weissman A, Lester D, Trexler L. The measurement of pessimism: the hopelessness scale. J Consult Clin Psychol
1974 Dec;42(6):861-865. [doi: 10.1037/h0037562] [Medline: 4436473]
46. Córdova Osnaya M, Rosales Pérez JC. Confiabilidad y validez de constructo de la Escala de Ideación Suicida de Beck en
estudiantes mexicanos de educación media superior. Alternativas Psicol 2012;16(26):16-25 [FREE Full text]
47. Spitzer RL, Kroenke K, Williams JB, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7.
Arch Intern Med 2006 May 22;166(10):1092-1097. [doi: 10.1001/archinte.166.10.1092] [Medline: 16717171]
48. García-Campayo J, Zamorano E, Ruiz MA, Pardo A, Pérez-Páramo M, López-Gómez V, et al. Cultural adaptation into
Spanish of the generalized anxiety disorder-7 (GAD-7) scale as a screening tool. Health Qual Life Outcomes 2010 Jan
20;8:8 [FREE Full text] [doi: 10.1186/1477-7525-8-8] [Medline: 20089179]
49. Plutchik R, Van Praag HM. Suicide risk: amplifiers and attenuators. J Offender Rehabil 1994;21(3-4):173-186. [doi:
10.1300/j076v21n03_11]
50. Alderete-Aguilar C, Cruz-Maycott R, Candela-Iglesias M, Rodríguez Estrada E, Reyes-Terán G. Assessment of depression,
anxiety, hopelessness and suicidal risk in HIV+ inpatients. Salud Ment (Mex) 2017 Jan 31;40(1):23-27. [doi:
10.17711/sm.0185-3325.2017.004]
51. Brooke J. SUS: a 'quick and dirty' usability scale. In: Jordan PW, Thomas B, McClelland IL, Weerdmeester B, editors.
Usability Evaluation in Industry. London, UK: CRC Press; 1996:189-194.
52. Brog NA, Hegy JK, Berger T, Znoj H. Effects of an internet-based self-help intervention for psychological distress due to
COVID-19: results of a randomized controlled trial. Internet Interv 2022 Mar;27:100492 [FREE Full text] [doi:
10.1016/j.invent.2021.100492] [Medline: 34956841]
53. Harrop E, Mann M, Semedo L, Chao D, Selman LE, Byrne A. What elements of a systems' approach to bereavement are
most effective in times of mass bereavement? A narrative systematic review with lessons for COVID-19. Palliat Med 2020
Oct;34(9):1165-1181 [FREE Full text] [doi: 10.1177/0269216320946273] [Medline: 32736489]
54. Dominguez-Rodriguez A, De La Rosa-Gómez A. A perspective on how user-centered design could improve the impact of
self-applied psychological interventions in low- or middle-income countries in Latin America. Front Digit Health 2022 Jun
2;4:866155 [FREE Full text] [doi: 10.3389/fdgth.2022.866155] [Medline: 35721795]
55. Karyotaki E, Efthimiou O, Miguel C, Bermpohl FM, Furukawa TA, Cuijpers P, Individual Patient Data Meta-Analyses for
Depression (IPDMA-DE) Collaboration, et al. Internet-based cognitive behavioral therapy for depression: a systematic
review and individual patient data network meta-analysis. JAMA Psychiatry 2021 Apr 01;78(4):361-371 [FREE Full text]
[doi: 10.1001/jamapsychiatry.2020.4364] [Medline: 33471111]
J Med Internet Res 2023 | vol. 25 | e43839 | p. 15https://www.jmir.org/2023/1/e43839 (page number not for citation purposes)
Dominguez-Rodriguez et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
56. Cuijpers P, Donker T, van Straten A, Li J, Andersson G. Is guided self-help as effective as face-to-face psychotherapy for
depression and anxiety disorders? A systematic review and meta-analysis of comparative outcome studies. Psychol Med
2010 Dec;40(12):1943-1957. [doi: 10.1017/S0033291710000772] [Medline: 20406528]
57. Dominguez-Rodriguez A, de la Rosa-Gómez A, Arenas-Landgrave P, Farfallini L, Montañez Alvarado P, Del Villar OA,
et al. Salud Mental COVID-19. 2020. URL: https://www.saludmentalcovid.com/ [accessed 2022-06-17]
58. Dominguez Rodriguez A, Martínez Arriaga RJ, de la Rosa A, Bautista Valerio E, Herdoiza Arroyo P, Alvarez Silva A, et
al. Personal Salud COVID-19. 2021. URL: https://www.personalcovid.com/
59. Rüegg N, Moritz S, Berger T, Lüdtke T, Westermann S. An internet-based intervention for people with psychosis (EviBaS):
study protocol for a randomized controlled trial. BMC Psychiatry 2018 Apr 13;18(1):102 [FREE Full text] [doi:
10.1186/s12888-018-1644-8] [Medline: 29653532]
60. Jiménez-Molina Á, Franco P, Martínez V, Martínez P, Rojas G, Araya R. Internet-based interventions for the prevention
and treatment of mental disorders in Latin America: a scoping review. Front Psychiatry 2019 Sep 13;10:664 [FREE Full
text] [doi: 10.3389/fpsyt.2019.00664] [Medline: 31572242]
Abbreviations
CBT: cognitive behavioral therapy
CG: control group
CGD: complicated grief disorder
CONSORT: Consolidated Standards of Reporting Trials
CONSORT-EHEALTH: Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications
and Online Telehealth
IG: intervention group
PSRS: Plutchik Suicide Risk Scale
PSS: Posttraumatic Stress Disorder Symptom Scale
PTSD: posttraumatic stress disorder
RCT: randomized controlled trial
SPIRIT: Standard Protocol Items: Recommendations for Interventional Trials
SUS: System Usability Scale
Edited by A Mavragani; submitted 26.10.22; peer-reviewed by C Laranjeira, P Franco; comments to author 29.11.22; revised version
received 24.12.22; accepted 27.02.23; published 06.04.23
Please cite as:
Dominguez-Rodriguez A, Sanz-Gomez S, González Ramírez LP, Herdoiza-Arroyo PE, Trevino Garcia LE, de la Rosa-Gómez A,
González-Cantero JO, Macias-Aguinaga V, Miaja M
The Efficacy and Usability of an Unguided Web-Based Grief Intervention for Adults Who Lost a Loved One During the COVID-19
Pandemic: Randomized Controlled Trial
J Med Internet Res 2023;25:e43839
URL: https://www.jmir.org/2023/1/e43839
doi: 10.2196/43839
PMID: 36877800
©Alejandro Dominguez-Rodriguez, Sergio Sanz-Gomez, Leivy Patricia González Ramírez, Paulina Erika Herdoiza-Arroyo,
Lorena Edith Trevino Garcia, Anabel de la Rosa-Gómez, Joel Omar González-Cantero, Valeria Macias-Aguinaga, Melina Miaja.
Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 06.04.2023. This is an open-access
article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/),
which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the
Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication
on https://www.jmir.org/, as well as this copyright and license information must be included.
J Med Internet Res 2023 | vol. 25 | e43839 | p. 16https://www.jmir.org/2023/1/e43839 (page number not for citation purposes)
Dominguez-Rodriguez et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
