Available via license: CC BY 4.0
Content may be subject to copyright.
Original Paper
One-Week Suicide Risk Prediction Using Real-Time Smartphone
Monitoring: Prospective Cohort Study
Maria Luisa Barrigon1,2*, MD, PhD; Lorena Romero-Medrano3,4*, MS; Pablo Moreno-Muñoz3,5, MS; Alejandro
Porras-Segovia1, MD, PhD; Jorge Lopez-Castroman3,6,7, MD, PhD; Philippe Courtet7,8, MD, PhD; Antonio
Artés-Rodríguez3,4,9,10, PhD; Enrique Baca-Garcia1,4,6,9,11,12,13,14,15, MD, PhD
1Department of Psychiatry, Jimenez Diaz Foundation University Hospital, Madrid, Spain
2Institute of Psychiatry and Mental Health, Hospital General Universitario Gregorio Marañón, Madrid, Spain
3Department of Signal Theory and Communications, Universidad Carlos III de Madrid, Madrid, Spain
4Evidence-Based Behavior (eB2), Madrid, Spain
5Cognitive Systems Section, Technical University of Denmark, Lyngby, Denmark
6Department of Psychiatry, Centre Hospitalier Universitaire Nîmes, Nîmes, France
7Institut de Génomique Fonctionnelle, CNRS-INSERM, University of Montpellier, Montpellier, France
8Department of Emergency Psychiatry and Acute Care, Centre Hospitalier Universitaire, Montpellier, France
9Centro de Investigación Biomédica en Red de Salud Mental (CIBERSAM), Carlos III Institute of Health, Madrid, Spain
10Instituto de Investigacion Sanitaria Gregorio Marañón, Madrid, Spain
11Department of Psychiatry, Autonomous University of Madrid, Madrid, Spain
12Department of Psychiatry, Rey Juan Carlos University Hospital, Móstoles, Madrid, Spain
13Department of Psychiatry, General Hospital of Villalba, Madrid, Spain
14Department of Psychiatry, Infanta Elena University Hospital, Valdemoro, Madrid, Spain
15Department of Psychology, Universidad Catolica del Maule, Talca, Chile
*these authors contributed equally
Corresponding Author:
Enrique Baca-Garcia, MD, PhD
Department of Psychiatry
Jimenez Diaz Foundation University Hospital
Av Reyes Católicos, 2
Madrid, 28040
Spain
Phone: 34 91 541 72 67
Fax: 34 91 542 35 36
Email: ebacgar2@yahoo.es
Abstract
Background: Suicide is a major global public health issue that is becoming increasingly common despite preventive efforts.
Though current methods for predicting suicide risk are not sufficiently accurate, technological advances provide invaluable tools
with which we may evolve toward a personalized, predictive approach.
Objective: We aim to predict the short-term (1-week) risk of suicide by identifying changes in behavioral patterns characterized
through real-time smartphone monitoring in a cohort of patients with suicidal ideation.
Methods: We recruited 225 patients between February 2018 and March 2020 with a history of suicidal thoughts and behavior
as part of the multicenter SmartCrisis study. Throughout 6 months of follow-up, we collected information on the risk of suicide
or mental health crises. All participants underwent voluntary passive monitoring using data generated by their own smartphones,
including distance walked and steps taken, time spent at home, and app usage. The algorithm constructs daily activity profiles
for each patient according to these data and detects changes in the distribution of these profiles over time. Such changes are
considered critical periods, and their relationship with suicide-risk events was tested.
J Med Internet Res 2023 | vol. 25 | e43719 | p. 1https://www.jmir.org/2023/1/e43719 (page number not for citation purposes)
Barrigon et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
Results: During follow-up, 18 (8%) participants attempted suicide, and 14 (6.2%) presented to the emergency department for
psychiatric care. The behavioral changes identified by the algorithm predicted suicide risk in a time frame of 1 week with an area
under the curve of 0.78, indicating good accuracy.
Conclusions: We describe an innovative method to identify mental health crises based on passively collected information from
patients’ smartphones. This technology could be applied to homogeneous groups of patients to identify different types of crises.
(J Med Internet Res 2023;25:e43719) doi: 10.2196/43719
KEYWORDS
e-health; m-health; Ecological Mometary Asssessment; risk prediction; sensor monitoring; suicidal; suicide attempt; suicide
Introduction
Each year, suicide is the cause of about 1.4% of all deaths
worldwide, totaling approximately 800,000 lives lost, which
means that more people die from suicide than from war and
homicide combined [1]. In 2020, the first full year of the
COVID-19 pandemic, suicide was responsible for nearly as
many years of potential life lost as the disease [2], making death
by suicide a major public health issue worldwide and one that
is worsening in spite of preventive efforts [3].
Interventions aimed at reducing the risk of suicide rely on the
effective identification of high-risk patients. There are many
known risk factors underlying suicide mortality, though their
ability to identify people at risk for suicide is scarcely better
than chance [4]. Furthermore, traditionally known factors such
as younger age, depression, and history of childhood trauma
are useful in predicting suicide in the long term (months or
years), though not over a period of weeks or days [5-7].
Therefore, prevention of suicide urgently requires improved
short-term prediction methods [8].
Nowadays, the gold standard method of assessing suicide risk
is the clinical interview, which, though clearly more effective
than questionnaires [9,10], is not without its limitations.
Specifically, data obtained in the course of a clinical assessment
are cross-sectional and, as such, are useful only for a brief period
[11]. Additionally, categorizations of suicide risk based on
clinical findings frequently result in a high false positive rate
(FPR) while overlooking many deaths owing to suicide [12].
In response to this, machine learning approaches have been
designed in recent years which hold promise as an improved
method of predicting mental health crises in general [13] and
suicide risk in particular [14]. Regarding mental health crises,
a recent work [13] predicted critical events within 1 month by
using a machine learning model that analyzed data from
electronic health records, achieving an area under the curve
(AUC) of 0.797.
There is a pressing need to move away from methods of suicide
prediction based on risk stratification in favor of approaches
that use individual risk measures [15]. Though suicide is a public
health problem of the highest order, an individual approach is
required to decrease suicide, and to do so, a precision health
perspective may be useful [16]. Technological advances widely
available for consumer use provide an ideal setting to advance
toward preventive medicine by harnessing the personalized data
they generate. As technology becomes increasingly embedded
in our lives, people’s digital devices (eg, smartphones and
wearables) produce a massive amount of data with possible
clinical relevance. This information, together with patient and
caregiver self-reported data, is referred to as “patient‐generated
health data” [17] and has proven to be useful in fields such as
oncology [18], ambulatory cardiac monitoring [19], and
neuropsychiatric illnesses [20]. Recently, the term “digital
phenotyping” was coined, referring to descriptions of behavior
based on people’s interaction with their smartphones (eg, sensor
information, keyboard interaction, voice, and speech data) [21].
The data that make up a person’s digital phenotype may be
either active (requiring input from the user, collected through
Ecological Momentary Assessment [EMA]) [22] or passive
(gathered by sensors incorporated in the device, requiring no
action by the user) [23].
In the field of suicide prevention, only active data have been
widely used to depict the suicidal digital phenotype [24] and
combining active data with information from electronic health
records has been demonstrated to be useful in predicting suicide
[25]. In contrast, although its feasibility has been demonstrated
in acute mental health setting [26], no studies to date have
proven the predictive potential of passive data. Given the low
rates of compliance with EMA among patients with suicidal
ideation [24], there is great interest in such automatically
generated data in the field of suicide prevention and research.
In this study, we aimed to test the effectiveness of a
smartphone-based system for continuous monitoring of patients
with suicidal ideation. We used passively collected data from
embedded smartphone sensors to detect behavioral changes
before a clinical risk event in patients participating in the
Spanish branch of the SmartCrisis Study [27], designed to study
suicide risk factors in a cohort of patients with a history of
suicide behavior or suicidal ideation. We hypothesized that our
system would be able to detect changes in a window of 1 week
before a risk event.
Methods
Overview
The SmartCrisis study [27] was conducted from February 2018
to March 2020 in Madrid (Spain) and Montpellier and Nimes
(France). For this analysis, only data from Spain were used.
Study Design and Sample
Participants were outpatients with any psychiatric diagnosis
undergoing follow-up in the program for secondary suicide
prevention at the Fundacion Jimenez Diaz Mental Health
Department.
J Med Internet Res 2023 | vol. 25 | e43719 | p. 2https://www.jmir.org/2023/1/e43719 (page number not for citation purposes)
Barrigon et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
Inclusion criteria were age 18 years or older, a history of suicide
behavior or suicidal ideation according to the Columbia Suicide
Severity Rating Scale [28], ability to understand and sign the
informed consent form, and ownership of a smartphone
connected to a Wi-Fi network at least once a week. Patients
were not compensated for their participation.
All participants downloaded the Evidence-Based Behavior (eB2)
app to their smartphones. Developed by the Department of
Signal Theory and Communications at Carlos III University in
Madrid, the app collects data from smartphone sensors [29].
Measures
Sociodemographic variables, including age, sex, marital status,
and employment status, were collected. Psychiatric diagnoses
were clinical, based on information contained in the electronic
health record according to the criteria contained in the 10th
edition of the International Classification of Diseases [30].
Participants were monitored for 6 months using their own
smartphones and assessed at baseline, 6 months, and at the end
of the study using a range of questionnaires. We also collected
information on clinical status throughout the year using
electronic health records, which enabled us to identify suicide
attempts or visits to the emergency department requiring
psychiatric assessment, which were our proxies for suicide risk
(ie, risk events). We decided to broaden the scope of suicide
risk status using this proxy given the relative infrequency of
suicide attempts, as the rate of nonfatal attempts in the year
following a previous attempt is only 16% to 17% [31,32], and
given the relationship between emergency department visits
and suicide [33]. For every participant, we considered the first
event a risk event, and after that, the monitoring was stopped.
The eB2 app collects, merges, and preprocesses the following
raw data: actigraphy, location tracking (ie, global positioning
system [GPS]), device usage, and activity measured by Google
Fit. These raw data generated by smartphone sensors were used
to build half-hourly summaries of the location, steps taken, and
distance walked, as well as app usage for the day. The output
data representation was used to model patients’ daily routines,
which we defined as daily profiles. A distribution of these
profiles over a series of days constituted a behavioral pattern;
changes between these behavioral patterns represented a
potential crisis to be correlated with suicide risk.
Ethical Considerations
The study was approved by the Fundacion Jimemez Diaz
Research Ethics Committee (PIC 99/2017_FJD) and carried out
in compliance with the tenets of the Declaration of Helsinki
[34]. All patients gave written informed consent to participate
after a complete description of the study, they were not
compensated for their participation. Concerning data protection
and confidentiality, each patient’s identification was ensured
by a username and password. The data gathered by the eB2 app
were anonymized if it were sensitive data, then translated into
a unique data schema, and finally transmitted through Wi-Fi to
the eB2 backend server where it were stored. The transmission
was done through a RESTful application programming interface,
which had been developed using the JAVASpring framework.
This application programming interface is secure sockets layer
protected, and, to restrict access to the patients’information, a
token-based access policy was implemented following the
OAuth2 standard.
Data Analysis
The system for data collection (eB2) is passive and unobtrusive.
It is downloaded to participants’ own smartphones under the
supervision of research assistants and works in the background,
with no active user collaboration required. For this particular
study, the app was configured to generate 4 types of variables
from the raw smartphone data: distance traveled, time spent at
home, steps taken, and use of any app. Each of these daily
activity variables was processed from raw data as a
48-dimensional vector, with each component representing 30
minutes of activity. The “distance” variable is continuous and
reflects the distance traveled by the patient, calculated as the
difference between location traces. The “time spent at home”
variable is considered binary: a value of 1 was assigned if the
patient was at home at any time during a particular 30-minute
slot and 0 if they were not. For each patient, the eB2 app
identified participant’s home based on the processing of location
traces. In particular, a patient’s home was considered as the
place where he or she spends more time during nights (along
time). The “steps” variable is continuous and contains the
number of steps walked over a particular time slot. The “app
usage” variable is binary, and a value of 1 was assigned if the
patient used any smartphone app during the period and 0 for no
app use. We used a logarithmic scale for the continuous
variables of distance and steps.
We created a model to obtain intuitive categorical variables
representing patients’ profiles as reflected by data on mobility,
physical activity, and app use (Figure 1). For instance, one value
for the categorical variable could indicate a “high-activity”
profile, with greater physical activity and more time spent
outdoors, while another may indicate “low-activity,” with less
physical activity and an increase in smartphone usage. We
created a personalized model for each patient. We used an
unsupervised model for this task, specifically a Heterogeneous
Mixture Model with 10 components. The number of components
defines the maximum number of possible profiles (10 in this
case), where the final number of profiles for each patient is
selected by means of a model selection method, the Bayesian
information criterion. The details of the method can be found
in the work by Moreno-Muñoz et al [35]. To simplify data
management by machine learning models, the initial set of
mobile data, which was high-dimensional (48 dimensions per
variable) and composed of different data types (ie, binary and
continuous), information corresponding to each day was
represented by a 1-dimensional categorical variable indicating
the daily profile.
J Med Internet Res 2023 | vol. 25 | e43719 | p. 3https://www.jmir.org/2023/1/e43719 (page number not for citation purposes)
Barrigon et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
Figure 1. Daily observations after preprocessing and profile clustering assignment from a patient over 120 days. The x-axis is shared across the 5
graphics and represents time (in days). Four upper rows: distance traveled (logarithmic scale), time spent at home, steps taken (logarithmic scale), and
app usage, respectively. For each figure, the y-axis indicates the value of the variable at each half-hour slot on the day. Fifth row: probability distribution
over 10 possible daily profiles obtained for each day. The y-axis indicates the likelihood that a given day is described by each profile.
In a second stage, we aimed to identify abrupt transitions in the
generative distribution of daily profile sequences for individual
patients. We defined each fixed distribution of profiles over a
series of days as a behavioral pattern. An example of an abrupt
transition would be when a patient with normally constant
behavior consisting of 5 days per week of a “high-activity”
profile alternating with 2 days of “low activity” shows a sudden
inversion in this proportion (ie, 2 days of high activity and 5
days of physical inactivity). We consider these situations
as change points, and we refer to them as behavioral changes,
which we aimed to detect.
The change-point detection model used is based on the
hierarchical extension of the Bayesian online change-point
detection algorithm [36]. We assumed that the main sequence
of daily profiles could be divided into nonoverlapping behavioral
patterns separated by behavioral changes. We also assumed that
the profiles were distributed unevenly within different behaviors,
though the parameters of each distribution are unknown. The
goal was to collect both the unknown parameters and the
locations of behavioral changes. Since we cannot observe the
true sequence of daily profiles representing a behavior, we must
use the sequence of probability profiling vectors. The
change-point detection method used was presented in our
previous work [36] and uses samples of the posterior
distributions to fully characterize the probability of each profile
for each day. Using this method, a discrete variable is introduced
into the model that counts the exact number of days since the
last change-point occurred. The aim was to continuously infer
the probability distribution of this variable given the data,
obtaining a measure of uncertainty of the last location of
behavioral change given the sequence of daily profiles up to
that moment. Moreover, the proposed model is able to naturally
integrate missing observations. Missing data are handled
differently by the model when observations are partially missing
(one or some features are totally or partially missing within a
day) or totally missing (every feature is missing that day). The
first case is handled in the daily profiling step through the
Heterogeneous Mixture Model, following the approach detailed
in [36,37]. The second situation is tackled in the second stage,
when the change-point detection model is applied through the
sequence of daily profile representations. In this case, we
consider a Bayesian approach that is based on marginalization
J Med Internet Res 2023 | vol. 25 | e43719 | p. 4https://www.jmir.org/2023/1/e43719 (page number not for citation purposes)
Barrigon et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
of missing observations, as also detailed in [36,37]. Treating
missing data through algorithms instead of heuristic approaches
provides robustness and allows to reduce the false alarm rate.
In a final stage, we defined a strategy to detect behavioral
changes based on the cumulative probability that a change
occurred over the previous days. Specifically, we consider that
a behavioral change has been detected at day “d” if the
cumulative probability that a change occurred over the previous
7 days exceeds a probability threshold that needs to be chosen
a priori and we refer to as stability threshold. This point is in
fact one of the model’s advantages because its choice provides
flexibility to the detection mechanism, allowing the detection
sensitivity to be adapted to the context of caregivers and patients.
The lower the probability threshold, the greater the sensitivity
for detection.
The entire process is represented graphically in Figure 2.
In particular, the receiver operating characteristic curve has
been generated over a range of values of this stability threshold.
Specifically, we have applied the method for 50 values, equally
distributed within the (0,1) interval, and for every patient in the
sample. For each stability threshold, the FPR and true positive
rate (TPR) have been calculated considering the predictions
obtained for every patient, day, and risk event, resulting in as
many data points as days we have in the whole cohort of
patients. On a specific day where the method predicts an event
that actually occurred, we consider that data point a true positive;
on the other hand, if the method predicts an event that did not
occur (usually called a false alarm), we consider that data point
a false positive. Using a fixed stability threshold, these data
points are used to compute the FPR and TPR, which are then
represented as a point on the receiver operating characteristic
curve.
The cross points on Figure 3 represent these 50 pairs (FPR and
TPR) for the 50 stability thresholds tested and the associated
interpolation curve.
Figure 2. Diagram illustrating the collection, preprocessing, segmentation, and detection stages of the mobile health data collection system performed
by the eB2 app.
Figure 3. (A) Receiver operating characteristic curve of our detection method for predicting suicide-risk events in a 1-week window. This metric has
been obtained over the stability threshold, defined as the probability threshold that considers whether a behavioral change occurred within the previous
7 days or not. We have applied our method for 50 values of the stability threshold and every patient in the sample. For each stability threshold, the false
positive rate (FPR) and true positive rate (TPR) have been calculated considering the result obtained for every patient and suicide risk event. The cross
points on the figure represent these 50 pairs (FPR and TPR) and the associated interpolation curve. (B) Model sensitivity and specificity for different
values of the stability threshold. (C) TPR and FPR for different values of the stability threshold.
J Med Internet Res 2023 | vol. 25 | e43719 | p. 5https://www.jmir.org/2023/1/e43719 (page number not for citation purposes)
Barrigon et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
Results
The sample consisted of 225 participants, 84 (37.3%) male and
141 (62.7%) female, with a mean age of 43.24 (SD 14.13) years.
Among the total sample, most participants were diagnosed with
mood or anxiety disorders. Sample characteristics are shown
in Table 1.
Participants were monitored by the eB2 app during 139.55 (SD
57.92) days, and at the end of the 6-month follow-up, 117 (52%)
patients continued uploading data (Figure 4).
A total of 18 (8%) suicide attempts were recorded during the
follow-up period, and there were 14 (6.2%) emergency
department visits for psychiatric care; together, these totaled
32 (14.2%) suicide risk events.
The behavioral changes detected by the model described above
predicted 1-week suicide risk with an AUC of 0.78 (Figure 3).
Table 1. Sample characteristics.
ValuesCharacteristics
Gender, n (%)
84 (37.3)Male
141 (62.7)Female
43.24 (14.13)Age (years), mean (SD)
Marital status, n (%)
77 (34.2)Married or unmarried couple
90 (40)Single
52 (23.1)Separated or divorced
6 (2.7)Widowed
Employment status, n (%)
101 (44.9)Employed or student
50 (22.2)Unemployed
12 (5.3)Retired
44 (19.6)Temporary leave
18 (8)Permanent leave
Living with others, n (%)
171 (76)Yes
43 (19.1)No
Psychiatric diagnosis, n (%)
104 (46.2)Mood disorders
169 (75.1)Anxiety disorders
109 (48.4)Personality disorders
24 (10.7)Drug abuse
10 (4.4)Eating disorders
1 (0.4)Psychotic disorders
Suicidal history, n (%)
94 (41.8)Suicide attempt, previous year
85 (37.8)Suicide attempt, more than 1 year previously
46 (20.4)Lifetime suicidal ideation
J Med Internet Res 2023 | vol. 25 | e43719 | p. 6https://www.jmir.org/2023/1/e43719 (page number not for citation purposes)
Barrigon et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
Figure 4. Survival curve of retention for the passive monitoring app over the 6-month follow-up period.
Discussion
Using smartphone-based monitoring, we followed up a cohort
of patients at risk of suicide, identifying their behavioral profiles
and changes to these profiles which predicted a period of clinical
risk. Our results indicate that this innovative method based on
passively collected information from patients’ smartphones is
capable of detecting risk situations with high precision
(AUC=0.78) over a 1-week period.
To date, many efforts have been made to classify the digital
phenotype of patients with suicidal ideation, and a number of
studies using smartphone-based EMA have reported high
variability of suicidal thoughts [5,38-40] and have observed
negative affect [41-46], environmental triggers [46,47], and
altered sleep [48] preceding suicidal thoughts, though the ability
of this methodology to predict risk of suicide has been less
comprehensively explored. As an example, a recent study found
that real-time changes in suicidal thoughts during psychiatric
hospitalization predicted suicide attempts in the month after
discharge with good accuracy (AUC=0.89) [25]. Passively
collected data have been used in the mental health field to
characterize stress and anxiety, depression, bipolar disorder,
schizophrenia, and posttraumatic stress disorder [20], but no
research has specifically studied suicidal behavior. In this study,
we identified behavioral changes that alerted us to the risk of
suicide or a mental health crisis over the following week.
Our results represent a breakthrough in personalized medicine
to treat suicidal behavior. Our tool determines an individual
patient’s risk status by comparing their present and past
behavioral patterns and could be used to define suicide risk and
design specific preventive interventions in line with
prevention-oriented formulations [49]. Previous works have
found that the ability to identify risk of suicide attempts after
an emergency department visit improved with a machine
learning prediction model that combines different sources of
information: patient’s self-reports, information in the electronic
health record, and clinician reports (1-month model: AUC=0.76;
6-month model: AUC=0.77) [50]. We theorize that if these
sources were merged with passive data, a more accurate system
for predicting suicide risk will be the result. This is of special
relevance given evidence from an important meta-analysis
reporting that 95% of patients classified as high-risk do not die
by suicide, while half of all suicides occur in patients who were
classified as low-risk [7].
We obtained a personalized short-term method of risk detection
that can be adjusted for cost-effective intervention programs
adapted to different health care systems. In low-resource
systems, it may be optimal to select a threshold that preserves
J Med Internet Res 2023 | vol. 25 | e43719 | p. 7https://www.jmir.org/2023/1/e43719 (page number not for citation purposes)
Barrigon et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
moderately high sensitivity and specificity; as a result,
behavioral changes identified will likely correspond to clinical
situations of suicide risk, and minor changes will not be
misidentified. Therefore, the tool will not underdetect risk events
and will not activate false alarms. In well-resourced health
systems, where costlier suicide protocols can be implemented,
a threshold can be selected to maximize sensitivity while
accepting moderate to low specificity. Furthermore, given the
accuracy obtained with our predictive model, adding a
personalized intervention would be cost-effective as per recently
proposed requirements [51].
Our model is remarkably robust in its handling of both total and
partially missing observations. Both situations are naturally
integrated into the model by assuming a Bayesian approach
[36,37]. This is of special importance for clinicians who treat
patients with an unclear risk of suicide, as clinical intuition
might resemble missing data in at-risk situations.
Our app for smartphone-based monitoring in an outpatient
sample of patients with a history of suicide illustrates the
potential of passive monitoring techniques. We applied our tool
to a large, clinically representative real-world sample with a
range of diagnoses, following these individuals for an extended
time period in a tax-funded health care system representative
of most European contexts.
When research with passive data collection is made, ethical
issues arise, and preserving privacy and confidentiality while
simultaneously ensuring safety if suicidal risk is detected is a
challenge [52]. Regarding this concern, we have to point out
that in our specifical sample, monitoring is well accepted [53].
Furthermore, our ethics protocol complies with European
standards.
In this study, some limitations should be acknowledged. First,
in order to obtain a sufficient number of critical events, we
decided to take into account not only suicide attempts but also
any emergency room visits involving psychiatric care, as these
have been found to significantly influence suicide risk [33].
Second, a number of patients were not under follow-up in our
facilities after 6 months, which is inherent to our real-world
design [53]. Third, we analyzed only a subset of data generated
by smartphone-based monitoring of location traces, Google Fit
activities, actigraphy, and app use logs. We focused on this set
because of the quality of these data sources and the availability
of the 4 sources for a large number of patients, thereby allowing
us to increase the size of our sample. Future research could be
enhanced by including data on light exposure, sleep, specific
activities (eg, time spent playing sports), particular app usage
(eg, messaging apps), and number of phone unlocks or calls,
among others. However, these sources are either not available
for every operating system or require a higher number of
permissions from the user. Last, some patients willing to enroll
in the study could not participate as they did not have a mobile
phone or had a noncompatible device; this highlights the digital
gap that researchers and clinicians must always account for
when considering the inclusion of new technologies in health
care. In fact, resource investment in digital protocols, which
may be more cost-efficient in the long term, can optimize the
use of traditional resources and narrow the digital gap.
Overall, the results of this cohort study indicate that an
unsupervised machine learning approach to data obtained by
passive real-time smartphone-based monitoring of patients with
suicidal ideation is useful in predicting suicide risk. These data,
when combined with actively collected data from patients and
caregivers (ie, EMA), data from electronic health records, and
clinical assessment, can improve suicide risk detection.
Moreover, this technology may be transferable to other
psychiatric conditions in which crisis prediction is needed.
Acknowledgments
This work has been partly supported by the Spanish government (AEI/MCI/ISCIII; under grants ISCIII PI20/01555,
PID2021-123182OB-I00, PID2021-125159NB-I00, and TED2021-131823B-I00; by the Comunidad de Madrid under grants
IND2018/TIC-9649, IND2022/TIC- 23550, and S2022/BMD-7216 AGES-CM 3CM; by the AFSP [LSRG-1-005-16]; and by
Fundacio La Marato de TV3 202226-31).
Data Availability
Data are available on demand.
Conflicts of Interest
PC has been a consultant to or has received honoraria or grants from Janssen Cilag, Pfizer, Exeltis, and Ethypharm Digital Therapy.
EBG has been a consultant to or has received honoraria or grants from Janssen Cilag, Lundbeck, Otsuka, Pfizer, Servier, and
Sanofi. He is a cofounder of Evidence-Based Behavior (eB2). AAR is a cofounder of eB2.
References
1. Knipe D, Padmanathan P, Newton-Howes G, Chan LF, Kapur N. Suicide and self-harm. Lancet 2022;399(10338):1903-1916
[doi: 10.1016/S0140-6736(22)00173-8] [Medline: 35512727]
2. Porras-Segovia A, Baca-García E, Courtet P, Wainberg M, Oquendo MA. If suicide were COVID-19: a neglected cause
of premature death. J Clin Psychiatry 2021;82(2):20l13702 [FREE Full text] [doi: 10.4088/JCP.20l13702] [Medline:
33988922]
J Med Internet Res 2023 | vol. 25 | e43719 | p. 8https://www.jmir.org/2023/1/e43719 (page number not for citation purposes)
Barrigon et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
3. Zalsman G, Hawton K, Wasserman D, van Heeringen K, Arensman E, Sarchiapone M, et al. Suicide prevention strategies
revisited: 10-year systematic review. Lancet Psychiatry 2016;3(7):646-659 [doi: 10.1016/S2215-0366(16)30030-X] [Medline:
27289303]
4. Franklin JC, Ribeiro JD, Fox KR, Bentley KH, Kleiman EM, Huang X, et al. Risk factors for suicidal thoughts and behaviors:
a meta-analysis of 50 years of research. Psychol Bull 2017;143(2):187-232 [doi: 10.1037/bul0000084] [Medline: 27841450]
5. Kleiman EM, Turner BJ, Fedor S, Beale EE, Huffman JC, Nock MK. Examination of real-time fluctuations in suicidal
ideation and its risk factors: results from two ecological momentary assessment studies. J Abnorm Psychol
2017;126(6):726-738 [doi: 10.1037/abn0000273] [Medline: 28481571]
6. Hendin H, Maltsberger JT, Szanto K. The role of intense affective states in signaling a suicide crisis. J Nerv Ment Dis
2007;195(5):363-368 [doi: 10.1097/NMD.0b013e318052264d] [Medline: 17502800]
7. Large M, Galletly C, Myles N, Ryan CJ, Myles H. Known unknowns and unknown unknowns in suicide risk assessment:
evidence from meta-analyses of aleatory and epistemic uncertainty. BJPsych Bull 2017;41(3):160-163 [FREE Full text]
[doi: 10.1192/pb.bp.116.054940] [Medline: 28584653]
8. Glenn CR, Nock MK. Improving the short-term prediction of suicidal behavior. Am J Prev Med 2014;47(3 Suppl
2):S176-S180 [FREE Full text] [doi: 10.1016/j.amepre.2014.06.004] [Medline: 25145736]
9. Tran T, Luo W, Phung D, Harvey R, Berk M, Kennedy RL, et al. Risk stratification using data from electronic medical
records better predicts suicide risks than clinician assessments. BMC Psychiatry 2014;14:76 [FREE Full text] [doi:
10.1186/1471-244X-14-76] [Medline: 24628849]
10. Chan MKY, Bhatti H, Meader N, Stockton S, Evans J, O'Connor RC, et al. Predicting suicide following self-harm: systematic
review of risk factors and risk scales. Br J Psychiatry 2016;209(4):277-283 [FREE Full text] [doi: 10.1192/bjp.bp.115.170050]
[Medline: 27340111]
11. Vera-Varela C, Manrique Mirón PC, Barrigón ML, Álvarez-García R, Portillo P, Chamorro J, MEmind Study Group, et
al. Low level of agreement between self-report and clinical assessment of passive suicidal ideation. Arch Suicide Res
2022;26(4):1895-1910 [doi: 10.1080/13811118.2021.1945984] [Medline: 34223799]
12. Large MM. The role of prediction in suicide prevention. Dialogues Clin Neurosci 2018;20(3):197-205 [FREE Full text]
[doi: 10.31887/DCNS.2018.20.3/mlarge] [Medline: 30581289]
13. Garriga R, Mas J, Abraha S, Nolan J, Harrison O, Tadros G, et al. Machine learning model to predict mental health crises
from electronic health records. Nat Med 2022;28(6):1240-1248 [FREE Full text] [doi: 10.1038/s41591-022-01811-5]
[Medline: 35577964]
14. Kirtley OJ, van Mens K, Hoogendoorn M, Kapur N, de Beurs D. Translating promise into practice: a review of machine
learning in suicide research and prevention. Lancet Psychiatry 2022;9(3):243-252 [doi: 10.1016/S2215-0366(21)00254-6]
[Medline: 35183281]
15. Mulder R, Newton-Howes G, Coid JW. The futility of risk prediction in psychiatry. Br J Psychiatry 2016;209(4):271-272
[FREE Full text] [doi: 10.1192/bjp.bp.116.184960] [Medline: 27698212]
16. Gambhir SS, Ge TJ, Vermesh O, Spitler R. Toward achieving precision health. Sci Transl Med 2018;10(430):eaao3612
[FREE Full text] [doi: 10.1126/scitranslmed.aao3612] [Medline: 29491186]
17. Shapiro M, Johnston D, Wald J, Mon D. Patient-generated Health Data: White Paper Prepared for the Office of the National
Coordinator for Health it by RTI International. 2012. URL: https://www.healthit.gov/sites/default/files/
rti_pghd_whitepaper_april_2012.pdf [accessed 2023-07-15]
18. Jim HSL, Hoogland AI, Brownstein NC, Barata A, Dicker AP, Knoop H, et al. Innovations in research and clinical care
using patient-generated health data. CA Cancer J Clin 2020;70(3):182-199 [FREE Full text] [doi: 10.3322/caac.21608]
[Medline: 32311776]
19. Sana F, Isselbacher EM, Singh JP, Heist EK, Pathik B, Armoundas AA. Wearable devices for ambulatory cardiac monitoring:
JACC state-of-the-art review. J Am Coll Cardiol 2020;75(13):1582-1592 [FREE Full text] [doi: 10.1016/j.jacc.2020.01.046]
[Medline: 32241375]
20. Reinertsen E, Clifford GD. A review of physiological and behavioral monitoring with digital sensors for neuropsychiatric
illnesses. Physiol Meas 2018;39(5):05TR01 [FREE Full text] [doi: 10.1088/1361-6579/aabf64] [Medline: 29671754]
21. Insel TR. Digital phenotyping: technology for a new science of behavior. JAMA 2017;318(13):1215-1216 [doi:
10.1001/jama.2017.11295] [Medline: 28973224]
22. Shiffman S, Stone AA, Hufford MR. Ecological momentary assessment. Annu Rev Clin Psychol 2008;4:1-32 [FREE Full
text] [doi: 10.1146/annurev.clinpsy.3.022806.091415] [Medline: 18509902]
23. Torous J, Kiang MV, Lorme J, Onnela JP. New tools for new research in psychiatry: a scalable and customizable platform
to empower data driven smartphone research. JMIR Ment Health 2016;3(2):e16 [FREE Full text] [doi: 10.2196/mental.5165]
[Medline: 27150677]
24. Sedano-Capdevila A, Porras-Segovia A, Bello HJ, Baca-García E, Barrigon ML. Use of ecological momentary assessment
to study suicidal thoughts and behavior: a systematic review. Curr Psychiatry Rep 2021;23(7):41 [doi:
10.1007/s11920-021-01255-7] [Medline: 34003405]
J Med Internet Res 2023 | vol. 25 | e43719 | p. 9https://www.jmir.org/2023/1/e43719 (page number not for citation purposes)
Barrigon et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
25. Wang SB, Coppersmith DDL, Kleiman EM, Bentley KH, Millner AJ, Fortgang R, et al. A pilot study using frequent
inpatient assessments of suicidal thinking to predict short-term postdischarge suicidal behavior. JAMA Netw Open
2021;4(3):e210591 [FREE Full text] [doi: 10.1001/jamanetworkopen.2021.0591] [Medline: 33687442]
26. Haines-Delmont A, Chahal G, Bruen AJ, Wall A, Khan CT, Sadashiv R, et al. Testing suicide risk prediction algorithms
using phone measurements with patients in acute mental health settings: feasibility study. JMIR Mhealth Uhealth
2020;8(6):e15901 [FREE Full text] [doi: 10.2196/15901] [Medline: 32442152]
27. Berrouiguet S, Barrigón ML, Castroman JL, Courtet P, Artés-Rodríguez A, Baca-García E. Combining mobile-health
(mHealth) and Artificial Intelligence (AI) methods to avoid suicide attempts: the smartcrises study protocol. BMC Psychiatry
2019;19(1):277 [FREE Full text] [doi: 10.1186/s12888-019-2260-y] [Medline: 31493783]
28. Posner K, Brown GK, Stanley B, Brent DA, Yershova KV, Oquendo MA, et al. The Columbia-suicide severity rating scale:
initial validity and internal consistency findings from three multisite studies with adolescents and adults. Am J Psychiatry
2011;168(12):1266-1277 [FREE Full text] [doi: 10.1176/appi.ajp.2011.10111704] [Medline: 22193671]
29. Berrouiguet S, Ramírez D, Barrigón ML, Moreno-Muñoz P, Camacho RC, Baca-García E, et al. Combining continuous
smartphone native sensors data capture and unsupervised data mining techniques for behavioral changes detection: a case
series of the evidence-based behavior (eB2) study. JMIR Mhealth Uhealth 2018;6(12):e197 [FREE Full text] [doi:
10.2196/mhealth.9472] [Medline: 30530465]
30. World Health Organization. Tenth Revision of the International Classification of Diseases and Related Health Problems
(ICD-10). Geneva, Switzerland: WHO; 1992.
31. Owens D, Horrocks J, House A. Fatal and non-fatal repetition of self-harm. Systematic review. Br J Psychiatry
2002;181:193-199 [FREE Full text] [doi: 10.1192/bjp.181.3.193] [Medline: 12204922]
32. Liu BP, Lunde KB, Jia CX, Qin P. The short-term rate of non-fatal and fatal repetition of deliberate self-harm: a systematic
review and meta-analysis of longitudinal studies. J Affect Disord 2020;273:597-603 [FREE Full text] [doi:
10.1016/j.jad.2020.05.072] [Medline: 32560959]
33. O'Neill S, Graham B, Ennis E. Emergency department and hospital care prior to suicide: a population based case control
study. J Affect Disord 2019;249:366-370 [doi: 10.1016/j.jad.2019.02.052] [Medline: 30807938]
34. World Medical Association. World Medical Association Declaration of Helsinki: ethical principles for medical research
involving human subjects. JAMA 2013;310(20):2191-2194 [FREE Full text] [doi: 10.1001/jama.2013.281053] [Medline:
24141714]
35. Moreno-Muñoz P, Romero-Medrano L, Moreno Á, Herrera-López J, Baca-García E, Artés-Rodríguez A. Passive detection
of behavioral shifts for suicide attempt prevention. ArXiv Preprint posted online on 14 Nov 2020 [FREE Full text]
36. Romero-Medrano L, Moreno-Muñoz P, Artés-Rodríguez A. Multinomial sampling of latent variables for hierarchical
change-point detection. J Sign Process Syst 2021;94(2):215-227 [FREE Full text] [doi: 10.1007/s11265-021-01705-8]
37. Ghahramani Z, Jordan M. Supervised learning from incomplete data via an EM approach. Advances in Neural Information
Processing Systems Morgan-Kaufmann. 1993. URL: https://papers.nips.cc/paper/1993/hash/f2201f5191c4e92cc5af043
eebfd0946-Abstract.html [accessed 2022-04-02]
38. Spangenberg L, Glaesmer H, Hallensleben N, Rath D, Forkmann T. (In)stability of capability for suicide in psychiatric
inpatients: longitudinal assessment using ecological momentary assessments. Suicide Life Threat Behav 2019;49(6):1560-1572
[doi: 10.1111/sltb.12547] [Medline: 30834576]
39. Kleiman EM, Turner BJ, Fedor S, Beale EE, Picard RW, Huffman JC, et al. Digital phenotyping of suicidal thoughts.
Depress Anxiety 2018;35(7):601-608 [doi: 10.1002/da.22730] [Medline: 29637663]
40. Hallensleben N, Glaesmer H, Forkmann T, Rath D, Strauss M, Kersting A, et al. Predicting suicidal ideation by interpersonal
variables, hopelessness and depression in real-time. An ecological momentary assessment study in psychiatric inpatients
with depression. Eur Psychiatry 2019;56:43-50 [FREE Full text] [doi: 10.1016/j.eurpsy.2018.11.003] [Medline: 30530103]
41. Rizk MM, Choo TH, Galfalvy H, Biggs E, Brodsky BS, Oquendo MA, et al. Variability in suicidal ideation is associated
with affective instability in suicide attempters with borderline personality disorder. Psychiatry 2019;82(2):173-178 [FREE
Full text] [doi: 10.1080/00332747.2019.1600219] [Medline: 31013205]
42. Victor SE, Scott LN, Stepp SD, Goldstein TR. I want you to want me: interpersonal stress and affective experiences as
within-person predictors of nonsuicidal self-injury and suicide urges in daily life. Suicide Life Threat Behav
2019;49(4):1157-1177 [FREE Full text] [doi: 10.1111/sltb.12513] [Medline: 30159910]
43. Humber N, Emsley R, Pratt D, Tarrier N. Anger as a predictor of psychological distress and self-harm ideation in inmates:
a structured self-assessment diary study. Psychiatry Res 2013;210(1):166-173 [FREE Full text] [doi:
10.1016/j.psychres.2013.02.011] [Medline: 23643187]
44. Links PS, Eynan R, Heisel MJ, Barr A, Korzekwa M, McMain S, et al. Affective instability and suicidal ideation and
behavior in patients with borderline personality disorder. J Pers Disord 2007;21(1):72-86 [doi: 10.1521/pedi.2007.21.1.72]
[Medline: 17373891]
45. Links PS, Eynan R, Heisel MJ, Nisenbaum R. Elements of affective instability associated with suicidal behaviour in patients
with borderline personality disorder. Can J Psychiatry 2008;53(2):112-116 [doi: 10.1177/070674370805300206] [Medline:
18357929]
J Med Internet Res 2023 | vol. 25 | e43719 | p. 10https://www.jmir.org/2023/1/e43719 (page number not for citation purposes)
Barrigon et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
46. Husky M, Swendsen J, Ionita A, Jaussent I, Genty C, Courtet P. Predictors of daily life suicidal ideation in adults recently
discharged after a serious suicide attempt: a pilot study. Psychiatry Res 2017;256:79-84 [FREE Full text] [doi:
10.1016/j.psychres.2017.06.035] [Medline: 28624676]
47. Nock MK, Prinstein MJ, Sterba SK. Revealing the form and function of self-injurious thoughts and behaviors: a real-time
ecological assessment study among adolescents and young adults. J Abnorm Psychol 2009;118(4):816-827 [FREE Full
text] [doi: 10.1037/a0016948] [Medline: 19899851]
48. Littlewood DL, Kyle SD, Carter LA, Peters S, Pratt D, Gooding P. Short sleep duration and poor sleep quality predict
next-day suicidal ideation: an ecological momentary assessment study. Psychol Med 2019;49(3):403-411 [FREE Full text]
[doi: 10.1017/S0033291718001009] [Medline: 29697037]
49. Pisani AR, Murrie DC, Silverman MM. Reformulating suicide risk formulation: from prediction to prevention. Acad
Psychiatry 2016;40(4):623-629 [FREE Full text] [doi: 10.1007/s40596-015-0434-6] [Medline: 26667005]
50. Nock MK, Millner AJ, Ross EL, Kennedy CJ, Al-Suwaidi M, Barak-Corren Y, et al. Prediction of suicide attempts using
clinician assessment, patient self-report, and electronic health records. JAMA Netw Open 2022;5(1):e2144373 [FREE Full
text] [doi: 10.1001/jamanetworkopen.2021.44373] [Medline: 35084483]
51. Ross EL, Zuromski KL, Reis BY, Nock MK, Kessler RC, Smoller JW. Accuracy requirements for cost-effective suicide
risk prediction among primary care patients in the US. JAMA Psychiatry 2021;78(6):642-650 [FREE Full text] [doi:
10.1001/jamapsychiatry.2021.0089] [Medline: 33729432]
52. Torous J, Roberts LW. The ethical use of mobile health technology in clinical psychiatry. J Nerv Ment Dis 2017;205(1):4-8
[doi: 10.1097/NMD.0000000000000596] [Medline: 28005647]
53. Porras-Segovia A, Díaz-Oliván I, Barrigón ML, Moreno M, Artés-Rodríguez A, Pérez-Rodríguez MM, et al. Real-world
feasibility and acceptability of real-time suicide risk monitoring via smartphones: a 6-month follow-up cohort. J Psychiatr
Res 2022;149:145-154 [doi: 10.1016/j.jpsychires.2022.02.026] [Medline: 35276631]
Abbreviations
AUC: area under the curve
eB2: Evidence-Based Behavior
EMA: Ecological Momentary Assessment
FPR: false positive rate
TPR: true positive rate
Edited by T de Azevedo Cardoso, A Mavragani; submitted 21.10.22; peer-reviewed by R Garriga, C Larkin, J Hinman; comments to
author 20.12.22; revised version received 03.02.23; accepted 26.06.23; published 01.09.23
Please cite as:
Barrigon ML, Romero-Medrano L, Moreno-Muñoz P, Porras-Segovia A, Lopez-Castroman J, Courtet P, Artés-Rodríguez A, Baca-Garcia
E
One-Week Suicide Risk Prediction Using Real-Time Smartphone Monitoring: Prospective Cohort Study
J Med Internet Res 2023;25:e43719
URL: https://www.jmir.org/2023/1/e43719
doi: 10.2196/43719
PMID: 37656498
©Maria Luisa Barrigon, Lorena Romero-Medrano, Pablo Moreno-Muñoz, Alejandro Porras-Segovia, Jorge Lopez-Castroman,
Philippe Courtet, Antonio Artés-Rodríguez, Enrique Baca-Garcia. Originally published in the Journal of Medical Internet Research
(https://www.jmir.org), 01.09.2023. This is an open-access article distributed under the terms of the Creative Commons Attribution
License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any
medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete
bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license
information must be included.
J Med Internet Res 2023 | vol. 25 | e43719 | p. 11https://www.jmir.org/2023/1/e43719 (page number not for citation purposes)
Barrigon et alJOURNAL OF MEDICAL INTERNET RESEARCH
XSL
•
FO
RenderX
