ArticlePDF Available

Abstract

The provocative paper by Ioannidis (2005) claiming that "most research findings are false" re-ignited longstanding concerns (see Meehl, 1967) that findings in the behavioral sciences are unlikely to be replicated. Then, a landmark paper by Nosek et al. (2015a) substantiated this conjecture, showing that, study reproducibility in psychology hovers at 40%. With the unfortunate failure of clinical trials in brain injury and other neurological disorders, it may be time to reconsider approaches not only in clinical interventions, but also how we establish their efficacy. A scientific community galvanized by a history of failed clinical trials and motivated by this "crisis" may be at critical cross-roads for change engendering a culture of transparent, open science where the primary goal is to test and not support hypotheses about specific interventions. The outcome of this scientific introspection could be a paradigm shift that accelerates our science bringing investigators closer to important advancements in rehabilitation medicine. In this commentary we offer a brief summary of how open science, study pre-registration and reorganization of scientific incentive structure could advance the clinical sciences.
Contents lists available at ScienceDirect
International Journal of Psychophysiology
journal homepage: www.elsevier.com/locate/ijpsycho
What the replication crisis means for intervention science
ABSTRACT
The provocative paper by Ioannidis (2005) claiming that “most research findings are false” re-ignited longstanding concerns (see Meehl, 1967) that findings in the
behavioral sciences are unlikely to be replicated. Then, a landmark paper by Nosek et al. (2015a) substantiated this conjecture, showing that, study reproducibility in
psychology hovers at 40%. With the unfortunate failure of clinical trials in brain injury and other neurological disorders, it may be time to reconsider approaches not
only in clinical interventions, but also how we establish their efficacy. A scientific community galvanized by a history of failed clinical trials and motivated by this
“crisis” may be at critical cross-roads for change engendering a culture of transparent, open science where the primary goal is to test and not support hypotheses
about specific interventions. The outcome of this scientific introspection could be a paradigm shift that accelerates our science bringing investigators closer to
important advancements in rehabilitation medicine. In this commentary we offer a brief summary of how open science, study pre-registration and reorganization of
scientific incentive structure could advance the clinical sciences.
Can the “Crisis” evoke a Paradigm Shift in Intervention Science?
The provocative paper by Ioannidis (2005) claiming that “most re-
search findings are false” re-ignited longstanding concerns (see Meehl,
1967) that findings in the behavioral sciences are unlikely to be re-
plicated. Then, a landmark paper by Nosek et al. (2015a) substantiated
this conjecture, showing that reproducibility hovers at 40%. While one
might argue about the terminology and causes used to describe the
situation facing behavioral scientists (“replication crisis”; cf. Maxwell
et al., 2015), there is growing consensus that there is room for im-
provement in approach and methods used and these concerns have
spared few areas of research in the health sciences (Benjamin et al.,
2018;Friesike et al., 2015;Goodman et al., 2016;Munafò et al., 2017;
Nosek et al., 2015b).
Central to concerns in the replication crisis is a demand for larger
data sets for maximizing statistical power while simultaneously ques-
tioning the incentive structure for publishing only statistically sig-
nificant findings (Cohen, 2016) and adhering to the philosophically
flawed null-hypothesis significance testing (NHST) (see Henkel, 2017;
Schneider, 2015). One might argue that the goals of rehabilitation
medicine and, cognitive remediation specifically, sit at an unenviable
intersection occupied by studies with small(ish) sample sizes aiming to
detect often subtle effects buried in the noise of inter and intra subject
variability (Park and Ingles, 2001;Rohling et al., 2009;Sitzer et al.,
2006;Wykes et al., 2011). To make matters worse, the goal to de-
monstrate statistically significant effects in interventions must be
achieved while surpassing increasingly stringent statistical thresholds
that have been proposed to handle replication problems based in NHST
(Benjamin et al., 2018).
With the unfortunate failure of clinical trials in brain injury and
other neurological disorders, it may be time to reconsider approaches
not only in our interventions, but also how we establish their efficacy.
In the case of pharmacologic interventions in acute traumatic brain
injury (TBI) in particular, the staggering 100% rate of failure (see Stein,
2015) has left the community to ponder how so many promising
interventions could survive early studies, only to falter so impressively
during phase III clinical trials (see Menon & Maas, 2014). To under-
stand where things have gone wrong with both behavioral and phar-
macologic interventions, one place to look would be the structures in
place guiding how we set-out to study the phenomenon in the first
place.
The “replication crisis” not only highlights the limitations of tradi-
tional statistical approaches and the circumscribed requirements for
scientific publication, but it leads to questions about the culture of
science. The culture of medical science includes an incentive structure
that requires innovative approaches, novel findings, and validation
through statistical significance via NHST. If NHST fails, researchers
commonly test many post-hoc hypotheses in order to fit the data (i.e., p-
hacking; Head et al., 2015). Unfortunately, this culture does not pro-
mote efficient science or the open study of clinical research because
researchers are not incentivized to publish or share results with the
scientific community when interventions fail. It is essential for the
scientific community to be aware of both successes and failures of well-
designed clinical interventions, making null findings a vital part of the
scientific landscape and ultimately expediting research.
There is also an important need to understand how group data from
an intervention study can inform us about the efficacy of any inter-
vention in the individual. Drawing from ergodic theory, Molenaar
(2004) predicted that cases where statistical estimates based on group
data would rarely reflect processes within individuals. In fact, empirical
studies show that individual and group estimates do diverge con-
siderably (Fisher et al., 2018;Seghier and Price, 2018). In the worst
case, we would need a different mechanistic model for each person to
treat that person's cognitive deficit or disease process. In other words, it
is unclear how to understand and treat cognitive dysfunction with
group data without knowing how group-level inferences map into in-
dividual processes over time. Thus, directly measuring within-subject
variability is a central feature to precision medicine to determine which
failures to replicate are driven by a lack of person-level analysis. In
rehabilitation medicine, reproducibility is at least partially linked to
https://doi.org/10.1016/j.ijpsycho.2019.05.006
International Journal of Psychophysiology xxx (xxxx) xxx–xxx
0167-8760/ © 2019 Published by Elsevier B.V.
how well group-level data represent individual responses to treatment.
Therefore, the person-level is the appropriate level at which interven-
tions should be conceptualized and studied and our group-level claims
should be rooted in models of processes that are validated within per-
sons. These processes could be tested for validity with convergent be-
havioral, neuroimaging, and other biological measures to identify me-
chanisms of action that could target precision interventions. In
principle, starting with the individual as the basic unit of study could
reduce the time needed to discover mechanisms of disorders and
change in clinical samples.
What to do?
So where do these challenges leave the intervention scientist? While
the problems with NHST have been described for decades (see Cohen,
1994;Rozeboom, 1960), the advent of the “crisis” may bring new
perspective and a collective imperative to change how we collect,
analyze, and disseminate clinical research findings. A general solution
to many of the concerns expressed here is to advocate for transparent
and open science. One vital step in an open science landscape is study
pre-registration, where interventionists using established methods for
even small sample or single-subject designs (e.g., multiple baseline,
reversal designs) can register the study goals and the results can be
accessed by the scientific community regardless of the study outcome.
Because all findings are available to the scientific community, open
science allows us to eliminate ineffective interventions and aids in
identifying interventions that work through replication. Moreover,
providing access to the outcomes of all studies can allow investigators
to view treatment effects and their variances as a continuum. This could
be one way to mitigate the community's reliance on arbitrary NHST
criteria and the tendency to artificially declare some findings significant
and others not (Wasserstein et al., 2019). As others have begun to note
more urgently, scientists should accept a statistical and epistemic
worldview that embraces uncertainty at its foundation (McShane et al.,
2019).
Perhaps unsurprisingly – and encouragingly, recent analysis of pre-
registered studies revealed a sharp rise in the publication of null find-
ings and replication studies (Warren, 2018). Because of this, pre-
registration will also reduce the tendency for interventions to appear
successful largely because they have been propped-up by well-meaning,
but naturally biased, researchers who have been incentivized to defend
interventions as opposed to critically testing their efficacy. Open sci-
ence and study preregistration may also help to standardize methods,
which are currently lacking in some areas of the clinical neurosciences
including functional brain imaging (see Esteban et al., 2019;Hallquist
and Hillary, 2018). Moreover, data sharing fostered in open science
holds additional opportunities to test the reliability of interventions and
their generalizability between research labs. Open science initiatives
can ultimately lead to data repositories that permit estimates for po-
pulation, sample level, and person level effect sizes. By extension, data
sharing provides estimates of patient response distributions that can be
used as “priors” for testing a range of hypotheses (including the null)
within a Bayesian framework which is becoming increasingly accessible
for statistical analyses (e.g., Bayes factor estimates) (Hoijtink et al.,
2019). This effort should consider the extent to which aggregate data
represent any process within the individuals within groups to quantify
potential process variability, clarify mechanisms, and tailor treatments.
Finally, for the goals presented here to be realized it requires a change
in scientific culture so that researchers are awarded and promoted
based upon their dedication to support open science, data sharing, and
study replication, (Gernsbacher, 2018).
One goal of this Special Issue to address current challenges in re-
habilitation medicine. It is an important time for clinical interven-
tionists to have this conversation. The concerns outlined above with
regard to NHST and scientific incentive structure are certainly not new
and are not the sole reason for difficulties advancing rehabilitation
medicine. However, a scientific community galvanized by a history of
failed clinical trials and motivated by this “crisis” may be at critical
cross-roads for change engendering a culture of transparent, open sci-
ence where the primary goal is to test and not support hypotheses about
specific interventions. The outcome of this scientific introspection
might be a paradigm shift that accelerates our science bringing in-
vestigators closer to important advances in rehabilitation medicine.
References
Benjamin, D.J., Berger, J.O., Johannesson, M., Nosek, B.A., Wagenmakers, E.J., Berk, R.,
... Cesarini, D., 2018. Redefine statistical significance. Nat. Hum. Behav. 2 (1), 6.
Cohen, J., 1994. The earth is round (p < .05). Am. Psychol. 49, 997–1003.
Cohen, J. (2016). The earth is round (p <.05). In What if there were no significance tests?
(pp. 69-82). Routledge.
Esteban, O., Markiewicz, C.J., Blair, R.W., Moodie, C.A., Isik, A.I., Erramuzpe, A., Kent,
J.D., Goncalves, M., DuPre, E., Snyder, M., Oya, H., Ghosh, S.S., Wright, J., Durnez,
J., Poldrack, R.A., Gorgolewski, K.J., 2019. fMRIPrep: a robust preprocessing pipeline
for functional MRI. Nat. Methods 16 (1), 111–116. https://doi.org/10.1038/s41592-
018-0235-4.
Fisher, A.J., Medaglia, J.D., Jeronimus, B.F., 2018. Lack of group-to-individual general-
izability is a threat to human subjects research. Proc. Natl. Acad. Sci. U. S. A. 115
(27), E6106–E6115. https://doi.org/10.1073/pnas.1711978115. Epub 2018 Jun 18.
PubMed PMID: 29915059; PubMed Central PMCID: PMC6142277.
Friesike, S., Widenmayer, B., Gassmann, O., Schildhauer, T., 2015. Opening science: to-
wards an agenda of open science in academia and industry. J. Technol. Transf. 40 (4),
581–601.
Gernsbacher, M.A., 2018. Writing empirical articles: transparency, reproducibility,
clarity, and memorability. Advanced Methods and Practice in Psychological Science 1
(3), 403–414.
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. (2016). What does research reproducibility
mean?. Science translational medicine,8(341), 341ps12-341ps12.
Hallquist, M.N., Hillary, F.G., 2018. Graph theory approaches to functional network or-
ganization in brain disorders: a critique for a brave new small-world. Network
Neuroscience 3 (1), 1–26.
Head, M.L., Holman, L., Lanfear, R., Kahn, A.T., Jennions, M.D., 2015. The extent and
consequences of p-hacking in science. PLoS Biol. 13 (3), e1002106.
Henkel, R.E., 2017. The Significance Test Controversy: A Reader. Routledge.
Hoijtink, H., Mulder, J., van Lissa, C., Gu, X., 2019. A tutorial on testing hypotheses using
the Bayes factor. Psychol. Methods. https://doi.org/10.1037/met0000201.
Ioannidis, J.P., 2005. Why most published research findings are false. PLoS Med. 2 (8),
e124 Epub 2005 Aug 30. PubMed PMID: 16060722; PubMed Central PMCID:
PMC1182327.
Maxwell, S.E., Lau, M.Y., Howard, G.S., 2015. Is psychology suffering from a replication
crisis? What does “failure to replicate” really mean? Am. Psychol. 70 (6), 487.
McShane, B. B., Gal, D., Gelman, A., Robert, C., & Tackett, J. L. (2019). Abandon sta-
tistical significance. The American Statistician,73(sup1), 235-245.
Meehl, P.E., 1967. Theory-Testing in Psychology and Physics: A Methodological Paradox.
Philos. Sci. 34 (2), 103–115.
Molenaar, P.C., 2004. A manifesto on psychology as idiographic science: bringing the
person back into scientific psychology, this time forever. Measurement 2 (4),
201–218.
Munafò, M.R., Nosek, B.A., Bishop, D.V., Button, K.S., Chambers, C.D., du Sert, N.P., ...
Ioannidis, J.P., 2017. A manifesto for reproducible science. Nat. Hum. Behav. 1 (1),
0021.
Nosek, B.A. et al. (2015a). Estimating the reproducibility of psychological science.
Science,349(6251), aac4716.
Nosek, B.A., Alter, G., Banks, G.C., Borsboom, D., Bowman, S.D., Breckler, S.J., ...
Contestabile, M., 2015b. Promoting an open research culture. Science 348 (6242),
1422–1425.
Park, N.W., Ingles, J.L., 2001. Effectiveness of attention rehabilitation after an acquired
brain injury: a meta-analysis. Neuropsychology 15 (2), 199.
Rohling, M. L., Faust, M. E., Beverly, B., & Demakis, G. (2009). Effectiveness of cognitive
rehabilitation following acquired brain injury: a meta-analytic re-examination of
Cicerone et al.'s (2000, 2005) systematic reviews. Neuropsychology, 23(1), 20.
Rozeboom, W.W., 1960. The fallacy of the null-hypothesis significance test. Psychol. Bull.
57 (5), 416.
Schneider, J.W., 2015. Null hypothesis significance tests. A mix-up of two different the-
ories: the basis for widespread confusion and numerous misinterpretations.
Scientometrics 102 (1), 411–432.
International Journal of Psychophysiology xxx (xxxx) xxx–xxx
2
Seghier, M.L., Price, C.J., 2018. Interpreting and Utilising Intersubject Variability in Brain
Function. Trends Cogn. Sci. 22 (6), 517–530. https://doi.org/10.1016/j.tics.2018.03.
003.
Sitzer, D.I., Twamley, E.W., Jeste, D.V., 2006. Cognitive training in Alzheimer's disease: a
meta-analysis of the literature. Acta Psychiatr. Scand. 114 (2), 75–90.
Stein, D.G., 2015. Embracing failure: What the Phase III progesterone studies can teach
about TBI clinical trials. Brain Inj. 29 (11), 1259–1272 Epub 2015 Aug 14. Review.
PubMed PMID: 26274493; PubMed Central PMCID: PMC4667711.
Warren, M., 2018. First analysis of ‘pre-registered’ studies shows sharp rise in null find-
ings. Oct. https://doi.org/10.1038/d41586-018-07118-1.
Wasserstein, R.L., Schirm, A.L., & Lazar, N.A. (2019) Moving to a world beyond “p < 0.
05”. The American Statistician,73(sup1), 1-19.
Wykes, T., Huddy, V., Cellard, C., McGurk, S.R., Czobor, P., 2011. A meta-analysis of
cognitive remediation for schizophrenia: methodology and effect sizes. Am. J.
Psychiatr. 168 (5), 472–485.
Frank G. Hillary
a,b,
, John D. Medaglia
c,d,e
a
Penn State Department of Psychology, Penn State University, United States
of America
b
Social, Life, and Engineering Imaging Center (SLEIC), United States of
America
c
Department of Psychology, Drexel University, United States of America
d
Department of Neurology, Drexel University, United States of America
e
Department of Neurology, Perelman School of Medicine, University of
Pennsylvania, United States of America
E-mail address: fhillary@psu.edu (F.G. Hillary).
Corresponding author at: 313 Bruce V. Moore Bldg., University Park, PA 16801, United States of America.
International Journal of Psychophysiology xxx (xxxx) xxx–xxx
3
... This is a bit of a knock in a field that's coming to favour the quantitative over the qualitative. But we must remember not only that quantitative analysis can be flawed (regarding the replication crisis and disciplines distorted by publication bias for instance [48]), but that it often relies on theory and argument that becomes implicit. We take for granted how much theoretical scaffolding lies behind the tools we trust. ...
... In recent years, several alternative measurement devices have been proposed -most notably wearable devices [41]. If researchers utilize wearable devices to estimate individual light exposure, they should report the accuracy of the devices and acknowledge their limitations [42]. ...
Conference Paper
Light at night (LAN) enables humans to extend their lifestyle and exploration, cultivates economic growth, and increases the perception of safety. On the other hand, LAN has been connected to adverse health outcomes, such as circadian disruption, mood effects, and increased breast cancer incidence risk in humans. Studies investigating health outcomes of LAN utilize a variety of methods sometimes resulting in conflicting outcomes. We conducted a systematic review focusing on the experimental methods and health outcomes of LAN studies. While most studies found a negative impact of LAN on human health, lighting conditions were not adequately reported or controlled in many cases. Recommendations are provided for future research studies investigating LAN effects on human health outcomes.
... When external validation is performed, the original prediction models are often abandoned in favour of new models [1,6]: the models were not predictive at all, but retrodictive or postdictive [67,82]. This links to the wider problem of reproducibility and replicability in science [1,[127][128][129][130]. At a minimum, we should successfully reproduce our results twice before delivering more definitive statements [14]; but preferably, independent verifications of the 7 A discussion of predictive models based on machine learning and other methods is outside the scope of this work, even if some of those methods may start from regression models (although often performing external validation). ...
... When recent disappointing psychological reproducibility results emerged from a major large scale study, researchers began vigorously debating whether the field of psychology was facing a "replication crisis" (Gilbert et al. 2016;Hillary and Medaglia 2019;Open Science Collaboration 2015). While longstanding concerns regarding the accurate reporting and replication of findings in psychology date back decades ago to the work of Meehl (1967), failing to replicate almost 60% of the selected past influential psychological studies was undeniably most alarming (Giordano and Waller 2020). ...
Article
Full-text available
This study applies equivalence testing methods on a representative sample of published empirical structural equation modeling studies in the psychological sciences and assesses the extent to which reported fit results are replicated when compared to those obtained via traditional methods. Results of 382 models from a sample of 242 articles published in 5 top ranked journals from the general field of Developmental Psychology were reported. The results indicated that a sizeable number of models designated in the original studies as ‘good’ were in fact not reproducible when examined through an equivalence testing lens. These questionable models displayed substantial discrepancy with the data and should not have qualified for further consideration because they were simply not plausible models. Implications of the results and suggestions for best modeling practices within the psychological sciences are discussed.
... Also pertinent in this discussion is the concept of "expectation bias" [36,37]. Succinctly phrased, time-stressed clinicians have considerable difficulty "seeing" patterns of patient behavior that are unreported or that they do not expect. ...
Article
In December 2019, the US Centers for Disease Control and Prevention (CDC) announced their intention to review and revise their 2016 Guideline on Prescription of Opioids to Adults. As part of this revision, CDC solicited nominations for an advisory “Opioid Workgroup” to report to the Board of Scientific Counselors of the National Center for Injury Prevention and Control. This paper offers a critical review of concerns identified in the final Workgroup report of July 2021, as contrasted against the revised and expanded guidelines published 16 months later in November 2022. The author finds that although the Workgroup was tasked to identify substantive issues, its input to the CDC was largely marginalized or ignored in the revised guidelines. The workgroup also failed to reach consensus on central issues of methodology, which should disqualify the CDC guidelines as a de factor standard of clinical practice. Arguably, the CDC should be removed from all further participation in development of public policy for the treatment of severe pain.
... Here, we chose explicit modeling of this heterogeneity in the Bayesian random effect meta-analysis, using a sensitivity analysis for the heterogeneity prior. Improving the replicability of clinical research [63] will require a major effort in harmonizing study and analysis designs and in using analyses capable of modeling heterogeneity itself. Therefore, the inclusion of multiple cohorts and the use of meta-analyses Fig. 5. Forest plots of amyloid and volume effects on cognitive outcomes. ...
Article
Full-text available
Background Cognitive decline is a key outcome of clinical studies in Alzheimer’s disease (AD). Objective To determine effects of global amyloid load as well as hippocampus and basal forebrain volumes on longitudinal rates and practice effects from repeated testing of domain specific cognitive change in the AD spectrum, considering non-linear effects and heterogeneity across cohorts. Methods We included 1,514 cases from three cohorts, ADNI, AIBL, and DELCODE, spanning the range from cognitively normal people to people with subjective cognitive decline and mild cognitive impairment (MCI). We used generalized Bayesian mixed effects analysis of linear and polynomial models of amyloid and volume effects in time. Robustness of effects across cohorts was determined using Bayesian random effects meta-analysis. Results We found a consistent effect of amyloid and hippocampus volume, but not of basal forebrain volume, on rates of memory change across the three cohorts in the meta-analysis. Effects for amyloid and volumetric markers on executive function were more heterogeneous. We found practice effects in memory and executive performance in amyloid negative cognitively normal controls and MCI cases, but only to a smaller degree in amyloid positive controls and not at all in amyloid positive MCI cases. Conclusions We found heterogeneity between cohorts, particularly in effects on executive functions. Initial increases in cognitive performance in amyloid negative, but not in amyloid positive MCI cases and controls may reflect practice effects from repeated testing that are lost with higher levels of cerebral amyloid.
... In the current paper, we take advantage of the open science paradigm that has followed the replication crisis discourse (Hillary & Medaglia, 2020) and use data from the Treatment of Adolescents with Depression Study (TADS, The TADS Team, 2003) to replicate the findings of Rognli et al., (2021). TADS was a large, four-arm multisite clinical trial of treatments for adolescent depression, conducted in the United States between 1998 and 2004. ...
Article
Full-text available
More knowledge is needed about how family factors influence depression features in adolescents. We aimed to replicate a previous small sample study showing that higher discrepancy between adolescents and caregivers in report of conflict predicted more hopelessness in adolescents with major depression. The current sample comprised 417 adolescent and maternal caregiver dyads. The adolescents were aged between 12 and 17 years (55.4% female, 44.6% male). The sample was drawn from the Treatment of Adolescents with Depression Study (TADS). Analyses were conducted using latent difference score modelling, item response theory, and linear regression in a Bayesian framework. Higher adolescent-perceived conflict relative to caregiver-perceived conflict predicted more adolescent hopelessness. This replicated the original findings in a much larger sample. Adolescent gender did not influence the associations. Addressing divergence in adolescent and parent perceptions of conflict may be relevant to ameliorate hopelessness in treatment of adolescents with depression.
... 55 But the operation strategies were unclear so that far more limited the possibilities of replications and robust interpretations of which underlying mechanism is effective, 84 which is one of the most longstanding concerns in the intervention field. 85 This again pointed out a necessity for a precision reporting of intervention details, as to gaining scientific credibility and facilitating replications in the emerging scientific areas. ...
Article
Full-text available
Unlabelled: Non-suicidal self-injury behavior (NSSI) is a serious public health concern that requires immediate attention. Despite the high prevalence of NSSI among the Chinese population, there is a significant gap in research on the comprehensive picture of this field. Therefore, a scoping review was conducted to investigate the prevalence, methods, risk factors, and preventive intervention programs related to NSSI in China. The review found that the estimated lifetime prevalence of NSSI among Chinese youth population is alarmingly high at 24.7% (N = 1,088,433). Common methods of NSSI include scratching, hitting, and biting. Additionally, the review synthesized 249 risk factors based on the biopsychosocial-ecological framework, highlighting the urgent need for intervention. However, only 12 empirical studies focus on NSSI prevention or intervention programs were included. These findings underscore the necessity for more clinical practices and larger studies to identify effective interventions and ultimately alleviate the burden of NSSI on the Chinese population. Funding: This review was supported by Humanity and Social Science Youth foundation of Ministry of Education (22YJCZH018), Science and Technology Innovation 2030 (STI2030-Major Projects:2021ZD0200702), National Natural Science Foundation of China (81825009), and Shuimu Tsinghua Scholar. No funding agencies were involved in the data collection, data analysis, and writing of this paper.
... While the main features of this exoskeleton are similar to others available on the market or tested in similar studies, there are slight differences that justify the present study. We also provide evidence on the impact of delivering walking training with this kind of technology in incomplete SCI patients, contributing to answering the fundamental question related to the outcomes of robotic walking rehabilitation conveyed with exoskeletons [28,29]. Therefore our objectives were: (1) to assess the safety and adherence to the robotic-assisted walking training as provided with the HANK exoskeleton by individuals with incomplete SCI, and (2) to assess the changes in walking function after a training program with HANK exoskeleton compared to traditional walking therapy in patients with non chronical condition. ...
Article
Full-text available
Background: In recent years, ambulatory lower limb exoskeletons are being gradually introduced into the clinical practice to complement walking rehabilitation programs. However, the clinical evidence of the outcomes attained with these devices is still limited and nonconclusive. Furthermore, the user-to-robot adaptation mechanisms responsible for functional improvement are still not adequately unveiled. This study aimed to (1) assess the safety and feasibility of using the HANK exoskeleton for walking rehabilitation, and (2) investigate the effects on walking function after a training program with it. Methods: A randomized controlled trial was conducted including a cohort of 23 patients with less than 1 year since injury, neurological level of injury (C2-L4) and severity (American Spinal Cord Injury Association Impairment Scale [AIS] C or D). The intervention was comprised of 15 one-hour gait training sessions with lower limb exoskeleton HANK. Safety was assessed through monitoring of adverse events, and pain and fatigue through a Visual Analogue Scale. LEMS, WISCI-II, and SCIM-III scales were assessed, along with the 10MWT, 6MWT, and the TUG walking tests (see text for acronyms). Results: No major adverse events were reported. Participants in the intervention group (IG) reported 1.8 cm (SD 1.0) for pain and 3.8 (SD 1.7) for fatigue using the VAS. Statistically significant differences were observed for the WISCI-II for both the "group" factor (F = 16.75, p < 0.001) and "group-time" interactions (F = 8.87; p < 0.01). A post-hoc analysis revealed a statistically significant increase of 3.54 points (SD 2.65, p < 0.0001) after intervention for the IG but not in the CG (0.7 points, SD 1.49, p = 0.285). No statistical differences were observed between groups for the remaining variables. Conclusions: The use of HANK exoskeleton in clinical settings is safe and well-tolerated by the patients. Patients receiving treatment with the exoskeleton improved their walking independence as measured by the WISCI-II after the treatment.
Preprint
Full-text available
The thermal grill, in which innocuous warm and cool stimuli are interlaced, can produce a paradoxical burning pain sensation—the thermal grill illusion (TGI). While the mechanisms underlying TGI remain unclear, prominent theories point to spinal dorsal horn integration of innocuous thermal inputs to elicit pain. It remains unknown whether the TGI activates peripheral nociceptors, or solely thermosensitive afferents and is integrated within the spinal cord. Different types of sensitization have established mechanisms and can inform TGI mechanisms: if the TGI elicits (1) primary hyperalgesia, peripheral nociceptors are activated; (2) secondary hyperalgesia in the absence of primary hyperalgesia, spinal integration is required; and (3) brush allodynia, wide-dynamic range neurons are involved in mediating the TGI. Here, we determine whether the TGI elicits primary hyperalgesia, secondary hyperalgesia or brush allodynia. Fifty-two participants underwent individually calibrated phasic thermal grill stimulation. We found that the TGI elicited primary hyperalgesia, but only in participants with component temperatures in the noxious range (<19 °C and >41 °C). The TGI also elicited secondary hyperalgesia, even in participants with strictly innocuous thermal inputs. No participants developed brush allodynia. We observed sex differences in primary hyperalgesia: only males exhibited thermal grill-induced primary hyperalgesia. These findings suggest that the TGI is integrated in the spinal dorsal horn, likely mediated by heat-pinch-cold (HPC) neurons, and, to some degree, by primary nociceptive afferents in males. This study shows that the TGI may have sex-dependent mechanisms and determines that HPC cells are involved in the illusory sensation of pain from innocuous thermal inputs.
Article
Full-text available
Learning about hypothesis evaluation using the Bayes factor could enhance psychological research. The Bayes factor quantifies the support in the data for two competing hypotheses. These may be the traditional null and alternative hypotheses, but these may also be informative hypotheses like m1 > m2 > m3 and (m1 − m2) > (m2 − m3) where m1, m2, and m3 denote the means in three experimental groups. Bayesian hypotheses evaluation offers options such as quantifying evidence in favor of the null-hypothesis, simultaneous evaluation of multiple hypotheses, and Bayesian updating, that is, recomputation of the Bayes factor after additional data have been collected. In this tutorial it is elaborated how researchers can use the Bayes factor for the analysis of their own data. The focus is completely applied and each topic discussed is illustrated using Bayes factors for the evaluation of hypotheses in the context of an ANOVA model, obtained using the R package bain. Readers can execute all the analyses presented while reading this tutorial if they download bain and the R-codes used from the bain website. It will be elaborated in a completely nontechnical manner: what the Bayes factor is, how it can be obtained, how Bayes factors should be interpreted, and what can be done with Bayes factors. After reading this tutorial and executing the associated code, researchers will be able to use their own data for the evaluation of hypotheses by means of the Bayes factor, not only in the context of ANOVA models, but also in the context of other statistical models.
Article
Full-text available
Preprocessing of functional magnetic resonance imaging (fMRI) involves numerous steps to clean and standardize the data before statistical analysis. Generally, researchers create ad hoc preprocessing workflows for each dataset, building upon a large inventory of available tools. The complexity of these workflows has snowballed with rapid advances in acquisition and processing. We introduce fMRIPrep, an analysis-agnostic tool that addresses the challenge of robust and reproducible preprocessing for fMRI data. fMRIPrep automatically adapts a best-in-breed workflow to the idiosyncrasies of virtually any dataset, ensuring high-quality preprocessing without manual intervention. By introducing visual assessment checkpoints into an iterative integration framework for software testing, we show that fMRIPrep robustly produces high-quality results on a diverse fMRI data collection. Additionally, fMRIPrep introduces less uncontrolled spatial smoothness than observed with commonly used preprocessing tools. fMRIPrep equips neuroscientists with an easy-to-use and transparent preprocessing workflow, which can help ensure the validity of inference and the interpretability of results. © 2018, The Author(s), under exclusive licence to Springer Nature America, Inc.
Article
Full-text available
This article provides recommendations for writing empirical journal articles that enable transparency, reproducibility, clarity, and memorability. Recommendations for transparency include preregistering methods, hypotheses, and analyses; submitting registered reports; distinguishing confirmation from exploration; and showing your warts. Recommendations for reproducibility include documenting methods and results fully and cohesively, by taking advantage of open-science tools, and citing sources responsibly. Recommendations for clarity include writing short paragraphs, composed of short sentences; writing comprehensive abstracts; and seeking feedback from a naive audience. Recommendations for memorability include writing narratively; embracing the hourglass shape of empirical articles; beginning articles with a hook; and synthesizing, rather than Mad Libbing, previous literature.
Article
Full-text available
Significance The current study quantified the degree to which group data are able to describe individual participants. We utilized intensive repeated-measures data—data that have been collected many times, across many individuals—to compare the distributions of bivariate correlations calculated within subjects vs. those calculated between subjects. Because the vast majority of social and medical science research aggregates across subjects, we aimed to assess how closely such aggregations reflect their constituent individuals. We provide evidence that conclusions drawn from aggregated data may be worryingly imprecise. Specifically, the variance in individuals is up to four times larger than in groups. These data call for a focus on idiography and open science that may substantially alter best-practice guidelines in the medical and behavioral sciences.
Article
Full-text available
Over the past two decades, resting-state functional connectivity (RSFC) methods have provided new insights into the network organization of the human brain. Studies of brain disorders such as Alzheimer’s disease or depression have adapted tools from graph theory to characterize differences between healthy and patient populations. Here, we conducted a review of clinical network neuroscience, summarizing methodological details from 106 RSFC studies. Although this approach is prevalent and promising, our review identified four challenges. First, the composition of networks varied remarkably in terms of region parcellation and edge definition, which are fundamental to graph analyses. Second, many studies equated the number of connections across graphs, but this is conceptually problematic in clinical populations and may induce spurious group differences. Third, few graph metrics were reported in common, precluding meta-analyses. Fourth, some studies tested hypotheses at one level of the graph without a clear neurobiological rationale or considering how findings at one level (e.g., global topology) are contextualized by another (e.g., modular structure). Based on these themes, we conducted network simulations to demonstrate the impact of specific methodological decisions on case-control comparisons. Finally, we offer suggestions for promoting convergence across clinical studies in order to facilitate progress in this important field.
Article
Full-text available
We consider between-subject variance in brain function as data rather than noise. We describe variability as a natural output of a noisy plastic system (the brain) where each subject embodies a particular parameterisation of that system. In this context, variability becomes an opportunity to: (i) better characterise typical versus atypical brain functions; (ii) reveal the different cognitive strategies and processing networks that can sustain similar tasks; and (iii) predict recovery capacity after brain damage by taking into account both damaged and spared processing pathways. This has many ramifications for understanding individual learning preferences and explaining the wide differences in human abilities and disabilities. Understanding variability boosts the translational potential of neuroimaging findings, in particular in clinical and educational neuroscience.
Article
Full-text available
In science publishing and many areas of research, the status quo is a lexicographic decision rule in which any result is first required to have a p-value that surpasses the 0.05 threshold and only then is consideration--often scant--given to such factors as prior and related evidence, plausibility of mechanism, study design and data quality, real world costs and benefits, novelty of finding, and other factors that vary by research domain. There have been recent proposals to change the p-value threshold, but instead we recommend abandoning the null hypothesis significance testing paradigm entirely, leaving p-values as just one of many pieces of information with no privileged role in scientific publication and decision making. We argue that this radical approach is both practical and sensible.
Article
Full-text available
We propose to change the default P-value threshold for statistical significance for claims of new discoveries from 0.05 to 0.005.