Available via license: CC BY-NC-ND
Content may be subject to copyright.
Contents lists available at ScienceDirect
Schizophrenia Research: Cognition
journal homepage: www.elsevier.com/locate/scog
Research Paper
Training engagement, baseline cognitive functioning, and cognitive gains
with computerized cognitive training: A cross-diagnostic study
Philip D. Harvey
a,⁎
, Alexandra M. Balzer
b
, Raymond J. Kotwicki
b
a
University of Miami Miller School of Medicine, USA
b
Skyland Trail, USA
ABSTRACT
Computerized cognitive training (CCT) interventions are increasing in their use in outpatient mental health settings. These interventions have demonstrated efficacy
for improving functional outcomes when combined with rehabilitation interventions. It has recently been suggested that patients with more cognitive impairment
have a greater therapeutic response and that reduced engagement in training can identify cases who manifest low levels of benefit from treatment. Participants were
psychiatric rehabilitation clients, with diagnoses of major depression, bipolar disorder and schizophrenia. Newly admitted cases received CCT, delivered via Brain
HQ, with cognitive functioning divided into groups on the basis of a BACS t-score of 40 or less vs. more. Training engagement was indexed by the number of training
levels achieved per day trained. Forty-nine cases trained on average for 17 days and completed a mean of 150 levels. Overall, patients improved by an average of 4.4
points (0.44 SD) in BACS t-scores (p< .001). Improvements were positively correlated with training engagement (r= 0.30, p< .05), but not with days trained
(r= 0.09) or levels earned (r=0.03) alone. Patients with higher levels of baseline cognitive performance had reduced cognitive gains (p< .003), but did not have
less training engagement (p= .97). Diagnoses did not predict cognitive gains (p= .93) or target engagement (p= .74). Poorer performance at baseline and higher
levels of training engagement accounted for >10% in independent variance in cognitive gains. The mean level of cognitive improvement far exceeded practice
effects. The index of engagement, levels achieved per training day, is easily extracted from the training records of patients, which would allow for early and
continuous monitoring of treatment engagement in CCT activities and therapist intervention as needed to improve engagement.
1. Introduction
Computerized cognitive training (CCT), often delivered with con-
current psychosocial rehabilitation as cognitive remediation therapy
(CRT) has been widely used in healthy older people and in patients with
schizophrenia. The evidence base for these two populations is quite
broad (Harvey et al., 2018) and CRT in particular has been shown to be
associated with functional gains. Meta-analyses have suggested that
CCT has cognitive benefits and when delivered as CRT also has con-
sistent functional gains (Wykes et al., 2011). The evidence base in other
psychiatric conditions is less substantial and there are many fewer CCT
and CRT studies in mood disorders, including bipolar disorder and
major depression.
Although supported by the results of meta-analyses, there have been
negative results. For instance, several trials have shown negative results
(Goff et al., 2007;Kantrowitz et al., 2016;Murthy et al., 2012;Rass
et al., 2012), even when paired with pharmacological interventions. It
has been suggested that one possibility for some of the negative results
has been failure of the training participants to actively participate in the
intervention, failing to manifest engagement with training and make
progress on the training procedures. It had been suggested that it would
be possible to increase training engagement by adding game-like
features to make the training procedures compelling (Fleming et al.,
2017;Lumsden et al., 2016). However, one study found that gamifi-
cation elements specifically lowered learning rates, perhaps by dis-
tracting users from the cognitive tasks themselves (Katz et al., 2014).
Recently it was also reported that there may be a specific threshold
for improvement on the central training task that is required to induce
transfer to untrained cognitive tasks. A previous study quantitatively
analyzed the relationship between gains in auditory processing during
training and overall cognitive gains and concluded that the final level of
auditory speed performance predicted the magnitude of cognitive gain,
and participants who did not achieve a level faster than ~85 ms did not
show generalized cognitive gains (Biagianti et al., 2016).
Another possible factor associated with training gains in CRT may
be baseline levels of cognitive performance. In a simple face-valid way,
it would appear that individuals with psychiatric conditions who do not
manifest cognitive impairments at baseline might not be candidates for
interventions aimed at cognitive enhancement. In the absence of cog-
nitive impairments, other factors might be responsible for disability.
Several studies have addressed this issue. For example, Strassnig et al.
(2018) reported that for schizophrenia patients with MCCB t-scores
>40, there was no significant correlation between MCCB performance
and independently rated indices of functional disability; whereas, for
https://doi.org/10.1016/j.scog.2019.100150
Received 19 January 2019; Received in revised form 25 April 2019; Accepted 3 May 2019
⁎
Corresponding author at: University of Miami Miller School of Medicine, 1120 NW 14th Street, Suite 1450, Miami, FL 33136, USA.
E-mail address: Philipdharvey1@cs.com (P.D. Harvey).
Schizophrenia Research: Cognition 19 (2020) 100150
Available online 13 May 2019
2215-0013/ © 2019 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/BY-NC-ND/4.0/).
T
the sample of patients with more severe cognitive deficits, disability in
everyday functioning was correlated with cognitive test performance.
In an aggregated set of data across four cognitive training trials, Detore
et al. (2019) reported that, in the entire database and across all of the
studies, patients with more severe cognitive impairments had a greater
benefit from CRT than patients with less impairment.
In this study, we examine cognitive gains associated with CCT in a
sample of patients with mood and psychotic disorders. All patients re-
ceived treatment at a psychiatric rehabilitation facility and patients
were trained with CCT with the Posit Science Brain HQ training pro-
gram. This program has shown efficacy in single site (Ahmed et al.,
2015;Fisher et al., 2009, 2010;Loewy et al., 2016;Surti et al., 2011)
and multi-site trials (Fisher et al., 2014;Keefe et al., 2012) targeting
people with schizophrenia, with some negative trials as described
above. Further, in a large-scale (n= 150), long term (26 week; 5 days
per week) randomized clinical trial (Mahncke et al., in press), active
training did not separate from placebo training. However, there was
evidence of substantial failure to engage in training in the active
treatment group. In contrast to previous successful studies (Fisher et al.,
2009;Keefe et al., 2012), performance on the index of target engage-
ment described above was at 121 ms in the active treatment group. This
is much slower than the other two studies (70 and 71 ms respectively)
and the threshold suggested by Biagianti et al. (2016) of 85 ms.
We examined cognitive gains for the sample as a whole, gains in
cases with greater and lesser cognitive impairments, and in each di-
agnostic group, as well as measuring training engagement in CCT and
its association with cognitive gains on an untrained cognitive assess-
ment measure to index near transfer. Our hypothesis was that the extent
of engagement in training would be associated with cognitive gains
from pretreatment to post-test. We also expected that patients with
more cognitive impairment would manifest greater gains. In order to
assess these variables, we performed exploratory comparative analyses
of the impact on cognitive change of training engagement and baseline
cognitive performance.
2. Method
2.1. Treatment site and participants
Participants were clients admitted to care at Skyland Trail, a non-
profit residential, partial hospitalization, and intensive out-patient
psychiatric rehabilitation facility located in Atlanta, GA. The center has
a continuum of care, where more symptomatic individuals were in-
itially placed in residential facilities and with planned transitions into
day treatment, intensive outpatient, and transitional treatment tracks as
symptomology and functionality improves. Less symptomatic in-
dividuals are directly admitted into outpatient services. The average
length of treatment for patients in all levels of care at Skyland Trail is
about 4 months. The data collected in this study were extracted from
medical records and there was no additional contact with clients and no
modifications of their care. At admission, all clients signed a general
consent form agreeing that data in their electronic medical records may
be used for research and quality improvement projects without re-
vealing their identities. As a result, these data analyses were not sub-
mitted as a research project to an institutional review board, because all
of the data collection procedures were part of ongoing standard clinical
care at the facility and these analyses were aimed at determining which
cases should be referred to CCT in the future.
These data were collected from May 2016 to December 2017,
during which time all admissions to treatment services were adminis-
tered a battery of assessments as part of the standard admissions pro-
cess. All patients received a diagnosis with a structured procedure that
has been previously published (Kotwicki and Harvey, 2013). This pro-
cedure included a structured interview with the MINI International
Neuropsychiatric Inventory (MINI; Sheehan et al., 1998). During this
time, all admissions were also tested with a neuropsychological
assessment. The distribution of all diagnoses for the consecutive ad-
missions was 23% bipolar disorder, 47% major depression, and 15%
schizophrenia, with other diagnoses less common. We did not analyze
data from patients whose primary diagnoses were substance abuse or
personality disorders. We also examined data only from cases between
the ages of 18 and 50. Cases with incomplete assessment data and those
who were referred to CCT, but refused to participate, were also ex-
cluded.
The eventual sample was 52% female and 53% young adult, ages 18
to 25 years old (Mean age = 28.6, SD = 11.6). A total of 49 cases met
the diagnostic criteria, completed baseline and post treatment assess-
ments with the BACS and completed at least one day of CCT training.
The diagnostic distribution of the cases was major depression (39%),
bipolar disorder (39%), and schizophrenia (22%). The Brief Assessment
of Cognition for Schizophrenia, paper version, (BACS; Keefe et al.,
2004) was used to measure cognitive ability. CCT was delivered to
patients via the online computer program Posit Science Brain HQ. The
participants were instructed to practice at least 3 h of training each
week, available either through the CRT group which met for 45 min
each day or as independent homework.
2.2. BACS
The following 6 tests constitute the BACS. All tests with alternative
forms were administered with form A first and B second.
List Learning (Verbal Memory): Patients are presented with 15 words
and then asked to recall as many as possible. This procedure is re-
peated 5 times. There are 8 alternate forms, of which 2 were used in
this study.
Digit Sequencing Task (Working Memory): Patients are presented with
strings of numbers of increasing length. They are asked to tell the
experimenter the numbers in order, from lowest to highest.
Token Motor Task (Motor Speed): Patients are given 100 plastic to-
kens and asked to place them into a container as quickly as possible
for 60 s.
Verbal Fluency,Category Instances (Semantic Fluency): Patients are
given 60 s to name as many words as possible within the animal
category.
Controlled Oral Word Association Test (Letter Fluency): In two separate
trials, patients are given 60 s each to generate as many words as
possible that begin with the letters F and S.
Tower of London Test (Executive Functions) Patients look at two pic-
tures simultaneously. Each shows 3 different-colored balls arranged
on 3 pegs, with the balls in a unique arrangement in each picture.
The patient is required to accurately estimate the fewest number of
times the balls in one picture would have to be moved in order to
make the arrangement of balls identical to that of the other, op-
posing picture.
Symbol Coding (Attention and Motor Speed) In this test, the numbers
1–9 are coded to symbols and drawn on a response sheet for 90 s.
A composite score using previously published procedures was the
primary outcome variable because the sample size did not allow for
analyses of the subtests. This composite is a t-score with a mean of 50
and a standard deviation of 10. Standard interpretations of this com-
posite score would suggest that scores of 40 or less reflect “possible
cognitive impairment.”
2.3. Computerized cognitive training
Participants self-administered the training after receiving instruc-
tions, with a proctor in the room to answer questions and encourage
adherence. All training used the commercially available Posit Science
Brain HQ system. Participants were asked to train at least 30 min per
session three days per week and to prioritize training on the “double
P.D. Harvey, et al. Schizophrenia Research: Cognition 19 (2020) 100150
2
decision” training task for at least half of the time. The rest of the
training was self-selected, but if the participants asked what they should
train on, they were told to prioritize brain speed, working memory, and
attention tasks. Training information was taken directly from the Brain
HQ portal and consisted of days spent training and levels achieved, with
all information collected after the participants had completed their
training. There was no attempt to directly monitor adherence with
these indices during training.
2.4. Statistical methods
Total scores on the BACS composite score at baseline and endpoint
were the dependent measures. We created an engagement score which
was the number of levels achieved per training day and a change score
which was the difference of baseline and endpoint scores on the BACS
composite. We divided the patients into subgroups on the basis of their
baseline composite BACS score (40 or less vs. higher). We used a two-
way analysis of variance (Diagnosis × cognitive status) to examine
differences in BACS change scores and target engagement. We then
correlated the BACS change scores with days trained, levels achieved,
and the ratio of the two. Finally, regression models were used to
identify independent contributions of all variables found to be corre-
lated with changes in cognitive performance.
3. Results
Table 1 presents the results of the BACS Composite at baseline and
endpoint, days trained, levels achieved, and target engagement as a
function of diagnosis and cognitive status. A paired t-test found that the
BACS scores improved significantly from baseline to endpoint, t
(48) = 3.38, p< .001 in the sample as a whole. The effect size for the
change was d = 0.44. The two-way ANOVA examining the effect of
diagnosis × cognitive status on BACS change scores found a significant
effect of baseline cognitive status, F(1,48) = 10.21, p< .005, but no
significant effect of diagnosis, F(2,47) = 0.08, p> .90 or diag-
nosis × cognitive status interactions F(2,47) = 2.24, p> .10. The two-
way ANOVA examining the effect of diagnosis × cognitive status on
training engagement scores found no significant effect of cognitive
status, F(1,48) = 0.00, p> .95, no significant effect of diagnosis, F
(2,47) = 0.31, p> .70 and no diagnosis × cognitive status interactions
F(2,47) = 1.39 p> .20.
Pearson correlations between changes in BACS performance from
baseline to endpoint, days trained, levels achieved, and the training
engagement variable are presented in Table 2. Changes in the BACS
from baseline to endpoint were significantly correlated with training
engagement, p< .05, but not with either days trained or levels
achieved. Days trained correlated with levels attained, but not with
levels per day and levels per day correlated with levels attained but not
with days trained.
A final analysis used the two variables found to predict cognitive
improvement with training, baseline cognitive status and levels
achieved per day, in a regression model. Since the two variables were
not related to each other, we used a stepwise entry procedure to predict
changes in cognition (Difference of BACS from Baseline to endpoint).
The overall analysis was significant, F(2,46) = 8.73, p< .001. Both
predictors entered with equation, with baseline cognitive status en-
tering first, t(46) = 3.42, p< .001, accounting for 18% of the variance,
followed by levels achieved per day, t(46) = 2.47, p= .02, accounting
for an additional 10% of the variance.
4. Discussion
In this study in a diverse sample of psychiatric patients receiving
rehabilitation therapy, two separate predictors of cognitive gains with
CCT were identified. Patients with higher levels of cognitive perfor-
mance made fewer training gains than patients with lower baseline
performance. Further, training engagement during training was a sig-
nificant and independent predictor of cognitive gains with training. The
amount of variance accounted for by these two predictors in treatment-
related gains in cognitive performance was quite substantial and there
were several other variables that did not predict treatment gains, in-
cluding diagnosis and the number of days spent training. The effect size
for change was training is considerably larger than expected with
practice alone (about 0.1–0.2 SD with one retest; Keefe et al., 2017),
suggesting that gains are associated with training and not reassessment
or placebo effects. Further, the systematic correlations with training
engagement, but not simple exposure to the training program as in-
dexed by days trained alone, support the idea that this is a CCT related
cognitive gain and not an artifact of a nonrandomized research design.
Both of these predictors had been identified previously in more
homogenous samples of patients (Biagianti et al., 2016;Detore et al.,
2019;Fisher et al., 2009;Keefe et al., 2012). However, these data
suggest that both engagement and baseline impairments are simulta-
neously applicable to the prediction of treatment-related cognitive
gains in CCT. Conveniently, both of these variables are quite easy to
measure, both prior to and during treatment. Further, these findings
also suggest that, at least for patients with psychiatric diagnoses and
persistent disability evidenced by rehabilitation treatment, patients
Table 1
Scores on the BACS and cognitive training process and engagement variables: Presented by diagnosis and baseline cognitive status.
Major depression Bipolar disorder Schizophrenia Overall
N = 19 N = 19 N = 11 n = 49
M SD M SD M SD M SD
BACS baseline 47.15 11.08 38.79 8.92 41.40 6.79 42.00 9.71
BACS endpoint 50.38 8.25 44.74 10.88 45.60 5.97 46.69 9.28
Training days 23.85 20.58 11.74 8.92 15.60 9.17 16.40 14.32
Levels achieved 209.54 150.51 98.95 117.28 135.10 88.21 141.79 129.15
Levels/training day 9.53 3.91 8.44 4.41 8.44 3.70 8.78 4.03
Baseline BACS <41 Baseline BACS >40
M SD M SD
BACS baseline 34.26 5.80 49.08 5.99
BACS endpoint 42.35 8.55 50.23 7.49
Training days 14.48 11.34 18.46 15.49
Levels achieved 137.91 138.50 160.31 128.04
Levels/training day 8.94 4.43 9.01 3.71
P.D. Harvey, et al. Schizophrenia Research: Cognition 19 (2020) 100150
3
with higher levels of cognitive performance do not benefit from treat-
ment even with exertion of adequate effort.
There are a couple of important clinical points from these data.
First, baseline assessment seems critical in populations where cognitive
impairments may not be ubiquitous and severe. Second, monitoring of
engagement should begin immediately after treatment starts. Third, it
should not be expected that engagement will covary with baseline
cognitive performance. Finally, as there is considerable evidence of
enhanced treatment gains associated with combined skills training and
CCT (Bowie et al., 2012), as well as combined CCT and social cognition
training (Lindenmayer et al., 2013, 2018), CCT in mental health po-
pulations should probably not be offered without other training ser-
vices. Although subjective reports of motivation have also previously
been found to relate to training gains in CCT (Saperstein and Medalia,
2015), the current study uses a direct measure of efficiency of training
gains to predict cognitive improvements with training.
The limitations of the study include the small sample size and un-
derpowered diagnostic-group comparisons. With larger samples, the
patients with MDD would likely, as expected, be found to have sig-
nificantly less cognitive impairment. We also did not examine the time
course of engagement in treatment, so we cannot tell if patients with
poor engagement can be identified immediately. In a previous study at
this site, we found that lack of treatment engagement identified im-
mediately after admission was not amenable to targeted interventions
using tangible rewards aimed at increasing engagement (Kotwicki et al.,
2017). Immediate lack of engagement also predicted worse treatment
outcomes. We did not have patients train on a single training proce-
dure, so we cannot isolate the specific training that led to gains.
However, the fact that training engagement with a heterogeneous
training procedure led to cognitive gains may actually be a positive
feature of the study, because of the ease of calculation of general
training engagement. The number of training sessions is less than some
previous studies with Brain HQ (Mahncke et al., in press;Fisher et al.,
2009), but the effect size for gains on untrained tasks was similar to
several previous studies for both Brain HQ (Fisher et al., 2010) and
other strategies, including the studies reviewed in the Detore et al.
(2019) meta-analysis. Finally, a randomized design could be more de-
finitive, but even in this open study there was considerable variance in
treatment outcomes that was systematically predicted by previously
identified predictors.
These data suggest that assessment of baseline cognitive perfor-
mance should be considered as a practice standard before engaging in
CCT and CRT interventions. Further, monitoring of treatment engage-
ment, either with task-specific indices such as those used by Fisher et al.
(2009),Keefe et al. (2012),Biagianti et al. (2016), and Mahncke et al.
(in press) or with more general indicators of training-related engage-
ment such as the current study, should commence early in treatment
and continuation decisions should be made quickly. Our previous
findings of failures of tangible rewards to improve general engagement
in rehabilitation treatment are consistent with previous suggestions that
CRT interventions are not facilitated by extrinsic rewards (Saperstein
and Medalia, 2015). Anecdotally, leveraging social interactions through
pairing CCT participants during training sessions, and providing a
therapist-led process group after training each day seemed to bolster
engagement. The results of this study suggest that intrinsic motivation
may not be adequate to induce CCT-related gains in patients whose
baseline scores are in the unimpaired range. Although these findings
replicate those of Detore et al. (2019), at least one other study has re-
ported that higher levels of performance in certain cognitive domains
leads to better gains (Lindenmayer et al., 2017). However, these pa-
tients had much more severe cognitive impairments on average than
the patients in this study, being institutionalized people with schizo-
phrenia. For example, our baseline mean cognitive performance score
was a t-score of 42 (21st percentile) and mean baseline t-score in
Lindenmayer et al. (2017) was 16 (0.1st percentile). Given the dis-
tributions of scores in that study, it appears as few of the participants
would be expected to have had a baseline t-score of 40 or more
(baseline score = 16; SD = 13; baseline plus 2 SD = 42). The reduced
benefits of training in patients with higher levels of cognitive perfor-
mance could be examined through re-analysis of existing datasets, as
this appears to be an important topic.
In conclusion, a treatment intervention that was successful overall,
leading to cognitive gains that notably exceed the expectations based
on retesting alone, was maximally effective in participants with base-
line levels of cognitive performance in the impaired range and in those
patients who exerted consistent effort across training sessions while
receiving CCT. These findings also suggest a trans-diagnostic effect of
CCT in cases whose cognitive impairments are substantial enough and
suggest that, particularly in patients with MDD whose cognitive im-
pairments may be both less common and less severe, baseline assess-
ment may be productive. Previous studies of CCT in people with MDD
have reported successes (Bowie et al., 2013), particularly in treatment
resistant MDD where cognitive impairments are likely to be more sig-
nificant.
This research was not supported by external funding. Dr. Kotwicki
and Ms. Balzer are full time employees of Skyland Trail. The CCT
software and Cognitive assessment materials were purchased from their
suppliers.
Conflict of interest
In the last three years, Dr. Harvey has received consulting fees or
travel reimbursements from Allergan, Alkermes, Akili, Biogen,
Boehringer Ingelheim, Forum Pharma, Genentech, Intra-Cellular
Therapies, Jazz Pharma, Lundbeck Pharma, Minerva Pharma, Otsuka
America (Otsuka Digital Health), Roche Parma, Sanofi Pharma,
Sunovion Pharma, Takeda Pharma, and Teva. He receives royalties
from the Brief Assessment of Cognition in Schizophrenia and the
MATRICS Consensus Battery. He has a research grant from Takeda and
from the Stanley Medical Research Foundation.
Dr. Kotwicki and Ms. Balzer are full-time employees of Skyland
Trail.
References
Ahmed, A.O., Hunter, K.M., Goodrum, N.M., Batten, N.-J., Birgenheir, D., et al., 2015. A
randomized study of cognitive remediation for forensic and mental health patients
with schizophrenia. J. Psychiatr. Res. 68, 8–18.
Table 2
Intercorrelations of change scores on the BACS and cognitive training process and engagement variables.
Training days Levels achieved Levels per day
Change in BACS scores over the training period −0.09 0.03 0.30
⁎
Training days – 0.79
⁎⁎⁎
0.00
Levels achieved – – 0.47
⁎⁎
Note.
⁎
p< .05.
⁎⁎
p< .01.
⁎⁎⁎
p< .001.
P.D. Harvey, et al. Schizophrenia Research: Cognition 19 (2020) 100150
4
Biagianti, B., Fisher, M., Neilands, T.B., Loewy, R., Vinogradov, S., 2016. Engagement
with the auditory processing system during targeted auditory cognitive training
mediates changes in cognitive outcomes in individuals with schizophrenia.
Neuropsychology 30, 998–1008.
Bowie, C.R., McGurk, S.R., Mausbach, B., Patterson, T.L., Harvey, P.D., 2012. Combined
cognitive remediation and functional skills training for schizophrenia: effects on
cognition, functional competence, and real-world behavior. Am. J. Psychiatry 169,
710–718.
Bowie, C.R., Gupta, M., Holshausen, K., Jokic, R., Best, M., Milev, R., 2013. Cognitive
remediation for treatment-resistant depression: effects on cognition and functioning
and the role of on-line homework. J. Nerv. Ment. Dis. 201, 680–685.
Detore, N., Mueser, K.T., Byrd, J., McGurk, S.R., 2019. Cognitive functioning as a pre-
dictor of response to cognitive remediation. J. Psychiatr. Res. 113, 117–124.
Fisher, M., Holland, C., Merzenich, M.M., Vinogradov, S., 2009. Using neuroplasticity-
based auditory training to improve verbal memory in schizophrenia. Am. J.
Psychiatry 166, 805–811.
Fisher, M., Holland, C., Subramaniam, K., Vinogradov, S., 2010. Neuroplasticity-based
cognitive training in schizophrenia: an interim report on the effects 6 months later.
Schizophr. Bull. 36, 869–879. https://doi.org/10.1093/schbul/sbn170.
Fisher, M., Loewy, R., Carter, C., Lee, A., Ragland, J.D., Niendam, T., Schlosser, D., Pham,
L., Miskovich, T., Vinogradov, S., 2014. Neuroplasticity-based auditory training via
laptop computer improves cognition in young individuals with recent onset schizo-
phrenia. Schizophr. Bull. 41, 250–258. https://doi.org/10.1093/schbul/sbt232.
Fleming, T.M., Bavin, L., Stasiak, K., Hermansson-Webb, E., Merry, S.N., Cheek, C., et al.,
2017. Serious games and gamification for mental health: current status and promising
directions. Front. Psych. 7.
Goff, D.C., Lamberti, J.S., Leon, A.C., Green, M.F., Miller, A.L., Patel, J., et al., 2007. A
placebo-controlled add-on trial of the ampakine, CX516, for cognitive deficits in
schizophrenia. Neuropsychopharmacology 33, 465–472.
Harvey, P.D., McGurk, S.R., Mahncke, H., Wykes, T., 2018. Controversies in computerized
cognitive training. Biol. Psychiatry Cogn. Neurosci. Neuroimaging 3, 907–915.
Kantrowitz, J.T., Sharif, Z., Medalia, A., Keefe, R.S.E., Harvey, P.D., Bruder, G., et al.,
2016. A multicenter, rater-blinded, randomized controlled study of auditory pro-
cessing–focused cognitive remediation combined with open-label lurasidone in pa-
tients with schizophrenia and schizoaffective disorder. J. Clin. Psychiatry 77,
799–806.
Katz, B., Jaeggi, S., Buschkuehl, M., Stegman, A., Shah, P., 2014. Differential effect of
motivational features on training improvements in school-based cognitive training.
Front. Hum. Neurosci. 8, 242.
Keefe, R.S.E., Goldberg, T.E., Harvey, P.D., Gold, J.M., Poe, M., Coughenour, L., 2004.
The brief assessment of cognition in schizophrenia: reliability, sensitivity, and com-
parison with a standard neurocognitive battery. Schizophr. Res. 68, 283–297.
Keefe, R.S.E., Vinogradov, S., Medalia, A., Buckley, P.F., Caroff, S.N., D'Souza, D.C., et al.,
2012. Feasibility and pilot efficacy results from the multisite cognitive remediation in
the schizophrenia trials network (CRSTN) randomized controlled trial. J. Clin.
Psychiatry 73, 1016–1022.
Keefe, R.S.E., Davis, V.G., Harvey, P.D., et al., 2017. Placebo response and practice effects
in schizophrenia cognition trials. JAMA Psychiat. 74, 807–814.
Kotwicki, R., Harvey, P.D., 2013. Systematic study of structured diagnostic procedures in
outpatient psychiatric rehabilitation: a three-year, three-cohort study of the stability
of psychiatric diagnoses. Innov. Clin. Neurosci. 10, 14–19.
Kotwicki, R., Balzer, A.M., Harvey, P.D., 2017. Measuring and facilitating client en-
gagement with financial incentives: implications for improving clinical outcomes in a
mental health setting. Community Ment. Health J. 53, 501–509.
Lindenmayer, J.P., McGurk, S.R., Khan, A., Kaushik, S., Thanju, A., Hoffman, L., et al.,
2013. Improving social cognition in schizophrenia: a pilot intervention combining
computerized social cognition training with cognitive remediation. Schizophr. Bull.
39, 507–517.
Lindenmayer, J.-P., Ozog, V.A., Khan, A., Ljuri, I., Fregenti, S., McGurk, S.R., 2017.
Predictors of response to cognitive remediation in service recipients with severe
mental illness. Psychiatric Rehab. J. 40, 61–69.
Lindenmayer, J.P., Khan, A., McGurk, S.R., Kulsa, M.K.C., Ljuri, I., Ozog, V., et al., 2018.
Does social cognition training augment response to computer-assisted cognitive re-
mediation for schizophrenia? Schizophr. Res. 201, 180–186.
Loewy, R., Fisher, M., Schlosser, D.A., Biagianti, B., Stuart, B., Mathalon, D.H.,
Vinogradov, S., 2016. Intensive auditory cognitive training improves verbal memory
in adolescents and young adults at clinical high risk for psychosis. Schizophr. Bull.
42, S118–S126.
Lumsden, J., Edwards, E.A., Lawrence, N.S., Coyle, D., Munafò, M.R., 2016. Gamification
of cognitive assessment and cognitive training: a systematic review of applications
and efficacy. JMIR Serious Games 4.
Mahncke, H.W., Kim, S.J., Rose, A., et al., 2019. Evaluation of a plasticity-based cognitive
training program in schizophrenia: results from the eCaesar trial. Schizophr. Res (in
press).
Murthy, N.V., Mahncke, H., Wexler, B.E., Maruff, P., Inamdar, A., Zucchetto, M., et al.,
2012. Computerized cognitive remediation training for schizophrenia: an open label,
multi-site, multinational methodology study. Schizophr. Res. 139, 87–91.
Rass, O., Forsyth, J.K., Bolbecker, A.R., Hetrick, W.P., Breier, A., Lysaker, P.H., O'Donnell,
B.F., 2012. Computer-assisted cognitive remediation for schizophrenia: a randomized
single-blind pilot study. Schizophr. Res. 139, 92–98.
Saperstein, A.M., Medalia, A., 2015. The role of motivation in cognitive remediation for
people with schizophrenia. In: Simpson, E.H., Balsam, P.D. (Eds.), Behavioral
Neuroscience of Motivation, Current Topics in Behavioral Neurosciences. Springer
International Publishing, pp. 533–546.
Sheehan, D.V., Lecrubier, Y., Harnett-Sheehan, K., et al., 1998. The mini international
neuropsychiatric interview (M.I.N.I.): the development and validation of a structured
diagnostic psychiatric interview. J. Clin. Psychiatr. 59, 22–33.
Strassnig, M.T., Bowie, C.R., Pinkham, A.E., et al., 2018. Which levels of cognitive im-
pairments and negative symptoms are related to functional deficits in schizophrenia?
J. Psychiatr. Res. 104, 124–129.
Surti, T.S., Corbera, S., Bell, M.D., Wexler, B.E., 2011. Successful computer-based visual
training specifically predicts visual memory enhancement over verbal memory im-
provement in schizophrenia. Schizophr. Res. 132, 131–134.
Wykes, T., Huddy, V., Cellard, C., McGurk, S.R., Czobor, P., 2011. A meta-analysis of
cognitive remediation for schizophrenia: methodology and effect sizes. Am. J.
Psychiatry 168, 472–485.
P.D. Harvey, et al. Schizophrenia Research: Cognition 19 (2020) 100150
5
