Heterogeneity Is a Hallmark of Traumatic Brain Injury, Not a Limitation: A New Perspective on Study Design in Rehabilitation Research

Abstract

Purpose
In both basic science and intervention research in traumatic brain injury (TBI), heterogeneity in the patient population is frequently cited as a limitation and is often interpreted as a factor reducing certainty in the generalizability of research findings and as a source of conflicting findings across studies. Historically, much of TBI research in rehabilitation and cognition has relied upon case–control studies, with small to modest sample sizes. In this context, heterogeneity is indeed a significant limitation. Here, however, we argue that heterogeneity in patient profiles is a hallmark characteristic of TBI and therefore cannot be avoided or ignored. We argue that this inherent heterogeneity must be acknowledged and accounted for prior to study design. Fortunately, advances in statistical methods and computing power allow researchers to leverage heterogeneity, rather than be constrained by it.

Method
In this article, we review sources of heterogeneity that contribute to challenges in TBI research, highlight methodological advances in statistical analysis and in other fields with high degrees of heterogeneity (e.g., psychiatry) that may be fruitfully applied to decomposing heterogeneity in TBI, and offer an example from our research group incorporating this approach.

Conclusion
Only by adopting new methodological approaches can we advance the science of rehabilitation following TBI in ways that will impact clinical practice and inform decision making, allowing us to understand and respond to the range of individual differences that are a hallmark in this population.

Full text

1"
2"
3"
4"
Heterogeneity is a Hallmark of Traumatic Brain Injury, Not a Limitation: 5"
A New Perspective on Study Design in Rehabilitation Research 6"
7"
8"
Natalie V. Covington1, 2 & Melissa C. Duff2 9"
10"
11"
1. Department of Speech-Language-Hearing Sciences, University of Minnesota, Minneapolis, 12"
MN 13"
2. Department of Hearing and Speech Sciences, Vanderbilt University Medical Center, 14"
Nashville, TN 15"
16"
Funding Statement: ASHFoundation Clinical Research Grant to NVC and NIDILRR grant 17"
90SFGE0012 to MCD. 18"
Conflict of Interest Statement: The authors report no relevant conflicts of interest. 19"
20"
21"
Corresponding Author: 22"
Natalie V. Covington, PhD CCC-SLP 23"
University of Minnesota 24"
115 Shevlin Hall 25"
164 Pillsbury Drive Southeast, Minneapolis, MN 55455 26"
Email: nvcoving@umn.edu 27"
ORCID: https://orcid.org/0000-0002-0016-2654 28"

Abstract: 29"
30"
Purpose: In both basic science and intervention research in traumatic brain injury (TBI), 31"
heterogeneity in the patient population is frequently cited as a limitation, and is often interpreted 32"
as a factor reducing certainty in the generalizability of research findings and as a source of 33"
conflicting findings across studies. Historically, much of TBI research in rehabilitation and 34"
cognition has relied upon case-control studies, with small to modest sample sizes. In this context, 35"
heterogeneity is indeed a significant limitation. Here, however, we argue that heterogeneity in 36"
patient profiles is a hallmark characteristic of TBI, and therefore cannot be avoided or ignored. 37"
We argue that this inherent heterogeneity must be acknowledged and accounted for prior to study 38"
design. Fortunately, advances in statistical methods and computing power allow researchers to 39"
leverage heterogeneity, rather than be constrained by it. 40"
41"
Method: In this Viewpoint, we review sources of heterogeneity that contribute to challenges in 42"
TBI research, highlight methodological advances in statistical analysis and in other fields with 43"
high degrees of heterogeneity (e.g. psychiatry) that may be fruitfully applied to decomposing 44"
heterogeneity in TBI, and offer an example from our research group incorporating this approach. 45"
46"
Conclusions: Only by adopting new methodological approaches can we advance the science of 47"
rehabilitation following TBI in ways that will impact clinical practice and inform decision 48"
making, allowing us to understand and respond to the range of individual differences that are a 49"
hallmark in this population. 50"
51"

Traumatic brain injury (TBI) is a major cause of acquired disability, with an estimated 52"
3.2-5 million Americans living with a TBI-related long-term disability (Coronado et al., 2011; 53"
Zaloshnja, Miller, Langlois, & Selassie, 2008). Critically, the cognitive and communicative 54"
consequences of TBI are some of the last to resolve, if they resolve at all. While patients with 55"
TBI often make significant physical gains in the first months following injury, cognitive and 56"
communication difficulties are more resistant to change (Pagulayan, Temkin, Machamer, & 57"
Dikmen, 2006). Impairments in cognition and communication persist in large proportions of 58"
patients with TBI, with approximately 60% of patients reporting memory impairments even 10 59"
years post-injury (Ponsford et al., 2014). Given the substantial risks cognitive and 60"
communicative impairments pose to regaining gainful employment and a high quality of life, 61"
maximizing the benefits of cognitive rehabilitation for each individual patient is imperative. 62"
Overall, participation in behavioral interventions appears to improve functional outcomes 63"
for patients with TBI, compared to those who do not receive such treatment (Cicerone et al., 64"
2008; Geurtsen, van Heugten, Martina, & Geurts, 2010; Goranson, Graves, Allison, & La 65"
Freniere, 2003; Konigs, Beurskens, Snoep, Scherder, & Oosterlaan, 2018). While rehabilitation 66"
services have been shown to be ultimately cost-saving, they are still very resource intensive 67"
(Andelic et al., 2014; Cooney & Carroll, 2016; Griesbach, Kreber, Harrington, & Ashley, 2015; 68"
Stolwyk, Gooden, Kim, & Cadilhac, 2019). Therefore, it is critical that rehabilitation maximizes 69"
gains for each patient. However, it has been suggested that heterogeneity within the population 70"
of individuals with TBI may be hampering effective intervention at the individual patient level 71"
(Cicerone et al., 2011; Dahdah et al., 2016; Goranson et al., 2003; Hart et al., 2014; Lu, Gary, 72"
Neimeier, Ward, & Lapane, 2012). In addition, systematic reviews and meta-analyses frequently 73"
find that known heterogeneity in the TBI sample is not adequately described, constraining efforts 74"

to aggregate findings across studies (Bigler et al., 2013; Elliott & Parente, 2014; Kennedy et al., 75"
2008; O’Neil-Pirozzi, Kennedy, & Sohlberg, 2016; Rohling, Faust, Beverly, & Demakis, 2009). 76"
Furthermore, over a 30-year period, none of the promising clinical interventions in animal 77"
models have translated to successful clinical treatments in human randomized controlled trials 78"
(RCT) (Lu et al., 2012). Lu and colleagues cite heterogeneity in the population of individuals 79"
with TBI as a primary barrier inhibiting progress in intervention research. 80"
The ideas and arguments in the current paper reflect our evolving thinking about 81"
heterogeneity’s role in TBI rehabilitation research, spurred by conversations, discussions, and 82"
concerns raised by attendees at the 2020 International Cognitive Communication Disorders 83"
Conference (ICCDC). TBI researchers have long recognized that heterogeneity in the patient 84"
population at large impedes the generalizability of their results to new patients. Clinically, 85"
rehabilitation professionals, including speech-language pathologists, treat individuals with TBI. 86"
However, our current research model often treats individuals with TBI as groups: how does the 87"
(fictional) average individual with TBI respond to a particular treatment. In order to advance the 88"
field of rehabilitation in TBI, and to make more precise decisions about particular treatment 89"
strategies for particular patients, we need to take a new approach. Addressing heterogeneity in 90"
the population head-on, by employing new methods of data collection, aggregation, and analysis 91"
will allow us to begin to make discoveries that have an impact on individual patients and their 92"
needs, by understanding and accounting for individual differences and heterogeneity in a 93"
population that is marked by these characteristics. 94"
Indeed, regarding medical management in TBI, Maas and colleagues write, “The question 95"
is if attempts to limit heterogeneity are appropriate, or alternatively, that the existing 96"
heterogeneity in patient populations, management approaches, and outcomes, may be used to 97"

advantage by exploring these differences and analyzing the underlying causes for a given 98"
outcome or individual patient response to a selected therapy or intervention” (Maas et al., 2012; 99"
emphasis added). This is a question that should also be at the forefront of rehabilitation research, 100"
as well: given that heterogeneity is a hallmark of TBI, how can we best leverage this inherent 101"
variability in ways that improve clinical decision-making? Would we be better served by seeking 102"
to sample across heterogeneity, rather than attempting to “control” for inherent heterogeneity by 103"
using stringent exclusionary criteria that result in samples that are not reflective of the larger 104"
population? 105"
Here, we argue that heterogeneity is hallmark in TBI: a feature, not a bug. However, the 106"
research methods commonly employed in our field (including by our research group) have been 107"
insufficient for addressing this hallmark heterogeneity. The primary goal of rehabilitative efforts 108"
following TBI is to deliver appropriate intervention to the individual patient: there is an inherent 109"
tension between commonly employed research designs that focus on group-level response to 110"
intervention and the clinical reality of treating one patient at a time. In this paper, we argue that 111"
the field of rehabilitation science in TBI must take a new methodological approach if we are to 112"
effectively decompose heterogeneity in TBI and begin to understand the sources and causes of 113"
inter-individual variability in clinical profiles and outcome. We will review sources of 114"
heterogeneity that contribute to challenges in TBI research, highlight methodological advances in 115"
statistical analysis and in other fields who also serve populations with high degrees of 116"
heterogeneity (e.g. psychiatry) that may be fruitfully applied to decomposing heterogeneity in 117"
TBI, and offer an example from our research group incorporating this approach. We believe that 118"
by taking a new stance on heterogeneity in TBI, and employing new approaches designed with 119"
heterogeneity in mind, we can make significant breakthroughs in advancing rehabilitation 120"

following TBI in ways that address the distinctive profiles and needs of each unique patient with 121"
TBI. 122"
The “Problem” of Heterogeneity 123"
Heterogeneity is one of the most commonly cited barriers to translational research in TBI. 124"
In a systematic review of three decades of RCTs in TBI, Lu and colleagues write, “the 125"
heterogeneity of this population remains one of the most important factors affecting outcomes of 126"
intervention following TBI” (Lu et al., 2012). While all brain pathology involves some degree of 127"
heterogeneity in presentation, TBI is among the most heterogeneous neurological conditions, 128"
resulting in vastly different constellations of neuroanatomical damage, physical disability, and 129"
cognitive deficits across patients. There is a wide range of inter-individual variability in TBI, 130"
which spans multiple levels of analysis: variability in premorbid patient characteristics, injury 131"
characteristics, pathoanatomic characteristics, and in physical, cognitive, and psychosocial 132"
profiles following injury. Heterogeneity at each of these levels of analysis likely contributes to 133"
the highly variable long-term outcomes observed following TBI (Dahdah et al., 2016; Hart et al., 134"
2014). In the following section, we highlight the myriad sources of heterogeneity across patients, 135"
to draw attention to the factors that researchers should be considering prior to study design, and 136"
will then go on to underscore the importance of new approaches that allow for characterization 137"
of this multidimensional variability. 138"
Natural Sources of Heterogeneity 139"
Heterogeneity in Pre-Injury Characteristics 140"
Inter-individual differences in demographic, psychosocial, and cognitive factors prior to 141"
injury impact how a TBI will affect a particular person, even when injury characteristics and 142"
severity are held constant across individuals. There is evidence for a non-linear effect of age, 143"

such that outcomes are poorer for survivors of pediatric TBI (Lah, Epps, Levick, & Parry, 2011; 144"
Verger et al., 2000) as well as older adults (de la Plata et al., 2008; Senathi-Raja, Ponsford, & 145"
Schönberger, 2010), compared to younger adults. Sex differences in response to TBI have been 146"
documented in social cognition (Turkstra et al., 2020), postconcussive symptoms (Bazarian, 147"
Blyth, Mookerjee, He, & McDermott, 2010), and functional outcomes (Mollayeva, Mollayeva, & 148"
Colantonio, 2018; Saban, Smith, Collins, & Pape, 2011). TBI also differentially impacts 149"
individuals with differing levels of educational attainment or “cognitive reserve” (Schneider et 150"
al., 2014). 151"
Basic science and intervention work differs in the degree to which these premorbid 152"
factors are considered or even reported. Older adults are under-represented in treatment studies, 153"
despite the fact that older adults are at particular risk for TBI compared to the general population 154"
(Corrigan et al., 2012; Gaastra et al., 2016; Gardner, Dams-O’Connor, Morrissey, & Manley, 155"
2018). Sex differences in response to intervention are understudied (Mollayeva et al., 2018) and 156"
female survivors of TBI are under-represented in research, particularly in military settings 157"
(Amoroso & Iverson, 2017). Despite high co-occurrence of psychiatric disease alongside 158"
neurological conditions like TBI, a majority of studies exclude patients with psychiatric 159"
conditions, limiting generalizability of findings (Trivedi & Humphreys, 2015). 160"
Heterogeneity in Injury Characteristics 161"
The cause of initial injury in TBI varies widely across patients, but this heterogeneity in 162"
injury characteristics is usually collapsed into a single diagnostic group in basic science and 163"
treatment research. Pathoanatomically, there are a wide range of possible injuries (e.g. skull 164"
fractures, hemorrhage (epidural, subdural, subarachnoid, intraparenchymal, intraventricular), 165"
parenchymal contusion/laceration). In addition, TBI patients differ significantly in the number, 166"

severity, and distribution of such lesions, and many incur both focal and diffuse lesions 167"
(Hawryluk & Manley, 2015; Saatman et al., 2008). Even in mild TBI, there is significant 168"
heterogeneity in injury characteristics. Rosenbaum and Lipton write, ““each patient is likely to 169"
experience a truly unique mechanism of brain injury at the tissue level” (Rosenbaum & Lipton, 170"
2012). 171"
The most widely used method for parsing some of the variability in injury characteristics 172"
is categorization of injuries by their “severity”. Some severity classification schemes utilize the 173"
Glasgow Coma Scale, (GCS), with cutoff scores defining “mild”, “moderate”, and “severe” TBI 174"
(Sherer, Struchen, Yablon, Wang, & Nick, 2008; Teasdale & Jennett, 1974). While the use of the 175"
GCS as a classification system has proven useful for group-level prognostication in large sample 176"
studies, individual patients with equivalent GCS factors can vary widely in important ways, with 177"
differing pathoanatomic characteristics and cognitive sequelae. For example, patients classified 178"
as having sustained a “severe” TBI based on GCS scores demonstrate brain damage that varies 179"
widely in terms of type of neuropathology and lesion location. Based upon this observation, 180"
according to Saatman and colleagues, “it is difficult to see how a therapy targeted simply for 181"
severe TBI [as determined by GCS score] could effectively treat all of these different types of 182"
injury” (Saatman et al., 2008). Injury severity classification is one of the most frequently utilized 183"
systems for subgrouping TBI patients in clinical trials, including for studies of cognitive 184"
rehabilitation. However, in the subacute and chronic phases of TBI management, significant 185"
heterogeneity in constellations of pathoanatomic and neuropsychological impairments make 186"
severity ratings based on GCS insensitive to important clinical factors for allied health 187"
rehabilitation. Finally, the GCS has been shown to be a poor discriminator for less severe TBI, 188"

which account for a majority of cases and may represent the portion of the population most likely 189"
to benefit from behaviorally-mediated rehabilitation. 190"
Heterogeneity in Symptom Presentation 191"
Common impairments following TBI include deficits in processing speed, attention, 192"
memory, social cognition and communication, and executive function (Finnanger et al., 2013; 193"
Frencham, Fox, & Maybery, 2005; McDonald, 2013; Rabinowitz & Levin, 2014; Vakil, 2005). 194"
However, it is clear that not every patient with TBI (even following severe TBI) demonstrates 195"
impairments in each of these domains. Rather, each individual patient has a unique profile, 196"
characterized by a particular constellation of impaired and spared ability across domains. Even in 197"
mild TBI, there is significant heterogeneity in whether or not patients report changes in cognition 198"
and communication and in how long these symptoms persist (Bigler et al., 2013; Pertab, James, 199"
& Bigler, 2009; Rosenbaum & Lipton, 2012). 200"
Researcher-Imposed Heterogeneity 201"
Shifting Definitions Across Time and Across Labs 202"
The inherent heterogeneity that is hallmark in TBI is further complicated by researcher-203"
driven sources of heterogeneity. In particular, definitional heterogeneity, across labs and across 204"
time, impedes efforts to combine datasets and meta-analyze findings over time. For example, 205"
Rosenbaum and Lipton highlight significant differences in operational definitions of “mild” TBI 206"
across various working groups and consortiums (e.g. CDC, WHO, ACRM) (Rosenbaum & 207"
Lipton, 2012). Groups differ in their diagnostic requirements for “mild” TBI (e.g. presence and 208"
length of loss of consciousness, GCS scores) and whether they further distinguish between 209"
subgroups within mild TBI (e.g. “uncomplicated” vs. “complicated”). In moderate-severe TBI, 210"

some groups rely on a single indicator to determine injury severity (e.g. GCS, Teasdale & 211"
Jennett, 1974) while others assess severity based on multiple indicators (Malec et al., 2007). 212"
“Acquired Brain Injury” Samples 213"
In both basic science and treatment literature, patients with TBI have frequently been 214"
studied alongside patients with stroke, lumped together as individuals with “acquired brain 215"
injury”. However, in systematic reviews of cognitive rehabilitation for acquired brain injury 216"
(including both TBI and stroke etiologies), effect sizes are modest (Cicerone et al., 2000, 2005, 217"
2011). Importantly, the majority of studies demonstrating significant treatment effects in these 218"
reviews are in stroke, not in TBI (Elliott & Parente, 2014; Rohling et al., 2009). In our view, 219"
basic science and intervention work should avoiding collapsing across these differing etiologies. 220"
Doing so introduces researcher-imposed heterogeneity that impedes meta-analysis and muddies 221"
evidence-based clinical decision-making. 222"
The “Problem” of Heterogeneity: Interactions across Multiple Levels of Analysis 223"
In sum, heterogeneity in TBI arises at multiple levels of analysis (including premorbid, 224"
injury, and symptom profiles). Importantly, inter-individual similarity at one level of analysis 225"
does not mean that individuals will be similar at another level of analysis. In addition, inter-226"
individual differences across levels of analysis likely interact, resulting in outcomes that are 227"
multiply-determined. Each source of heterogeneity is potentially important to treatment decision-228"
making and long-term outcomes. Unfortunately, the most commonly employed research designs 229"
in the field do not adequately capture this variation, but instead collapse and average across it, 230"
resulting in effects that represent the ability or treatment response of a fictional “average” 231"
patient. The next sections describe how our most commonly employed research designs fail to 232"
account for meaningful heterogeneity and may be impeding our ability to maximize treatment 233"

gains for individual patients, and point to advances in other fields who contend with similar 234"
heterogeneity in ways that may be fruitfully applied to rehabilitation science. 235"
Heterogeneity is a Limitation for the Most Commonly Employed Designs in TBI Research 236"
Much of the basic science and treatment literature in TBI research employs research 237"
designs that hinge on group-level analyses1. In case-control designs, participants with TBI are 238"
compared as a group to a group of “controls” (usually demographically-matched healthy adults). 239"
This design reduces a heterogeneous group of individuals to a single group-level point estimate 240"
(e.g. average performance). Any meaningful inter-individual differences are averaged out in the 241"
results. This results in what is known as the ecological fallacy: while results of group treatment 242"
studies can establish the better treatment on average, and results of case-control studies can 243"
establish differences between two groups on average, these results can not be translated directly 244"
to any particular individual. 245"
""""""""""""""""""""""""""""""""""""""""""""""""""""""""
1"Single"case"research"designs"are"not"discussed"in"the"main"text,"as"they"are"not"within"the"
purview"of"current"manuscript."Importantly,"this"should'not"be"taken"as"an"argument"
against"such"research"designs."SingleGcase"designs"contribute"critically"to"Phase"I"clinical"
research"(Robey"&"Schultz,"1998)."Within"this"framework,"singleGcase"research"is"essential"
for"the"development"of"new"treatment"approaches,"for"investigating"how"changes"in"
treatment"dose,"content,"and"other"factors"that"may"impact"response"to"treatment,"and"for"
providing"initial"evidence"for"the"characteristics"that"determine"which"patients"may"be"
good"candidates"for"a"particular"intervention."In"fact,"it"is"the"opinion"of"the"authors"that,"in"
cases"in"which"(for"reasons"of"time,"resources,"or"other"pragmatic"considerations)"a"
research"team"is"considering"a"smallGn"group"design,"the"field"might"be"better"served"by"a"
series"of"singleGcase"designs"that"meet"standards"for"rigorous"design"(e.g."see"Tate,"
Perdices,"McDonald,"Togher,"&"Rosenkoetter,"2014)"and"reporting"(see"Tate"et"al.,"2016),"
rather"than"performing"groupGlevel"statistical"tests"on"a"sample"that"is"underpowered"(for"
reasons"described"in"the"main"text)."Critically,"just"as"we"argue"in"this"Viewpoint'for"
improved"reporting"of"patient"characteristics,"open"sharing"of"data,"and"examining"
outcomes"across"multiple"levels"of"analysis,"similar"standards"should"be"applied"to"single"
case"designs."Transparent"reporting"of"patient"characteristics,"detailed"descriptions"of"
treatment"methods,"and"open"materials"and"data"will"improve"secondary"analyses"of"data"
and"metaGanalysis"across"studies.""
"

Beyond the problems in translating group-level results to individuals raised by the 246"
ecological fallacy, the issues that arise from the group-level analyses are compounded by the 247"
small sample sizes that characterize most studies in TBI rehabilitation research. For example, 248"
metanalysis of Cicerone and colleague’s seminal reviews of cognitive rehabilitation treatment 249"
research suggest an average treatment group size of 16.9 (Cicerone et al., 2000, 2005; Rohling et 250"
al., 2009). Unfortunately, the combination of case-control design and small sample size results in 251"
significant problems in the context of a heterogeneous population, including 1) high likelihood of 252"
biased sampling and 2) unstable effect size estimates (see Lombardo, Lai, & Baron-Cohen, 2019 253"
for detailed discussion). Together, these issues decrease the replicability and generalizability of 254"
our findings, lead to conflicting findings across research groups, and impede meta-analysis 255"
across studies. In the next section, we hope to convince the reader that we as a field are currently 256"
ill-served by the current research model, coming to conclusions that are suspect and do not 257"
reflect the characteristic heterogeneity of patients with TBI. 258"
High Likelihood of Biased Sampling across Research Groups 259"
Simulations by Lombardo and colleagues demonstrate widespread variability in the 260"
representativeness of small samples in populations characterized by heterogeneity. For example, 261"
a subgroup which comprises 20% of the true population can make up anywhere between 0% and 262"
60% of the sample for study with a sample size of 20 (Lombardo et al., 2019). This range 263"
significantly decreases as sample size increases; but, with an average sample size of ~17 in 264"
cognitive rehabilitation research, there is a high likelihood that differing results across similar 265"
studies may be driven by underlying imbalances in sampling across meaningful subgroups. 266"
Suppose that heterogeneity within the larger population of individuals with TBI can be 267"
subdivided into meaningful subgroups, based on inter-individual differences. In the context of 268"

heterogeneity and small sample sizes, it is possible (probable!) that two studies examining the 269"
same phenomenon (e.g. procedural memory) in the “same” population will come to opposite 270"
conclusions if they draw samples that do not reflect the population-level distribution across these 271"
underlying subgroups. Consider the following imagined example (see Lombardo et al., 2019 for 272"
a similar example in autism): Say that for a majority of the true population of individuals with 273"
TBI, procedural memory is spared (80%, PM_spared); but a subgroup within the larger 274"
population has significant procedural memory deficits (20%, PM_deficit). Ideally, all group 275"
studies examining procedural memory abilities in TBI would recruit a sample that reflects this 276"
population-distribution (80% PM_spared and 20% PM_deficit). In this ideal world, a case-277"
control study with a sample size of 20 would recruit 16 patients from the underlying PM_spared 278"
subgroup and 4 patients from the PM_deficit subgroup, and come to the “correct” group-level 279"
conclusion that there is no difference between patients with TBI and controls; therefore, 280"
procedural memory is spared in patients with TBI. However, a given study may (unknowingly) 281"
sample too heavily from the PM_deficit group, recruiting 10 patients with PM_spared and 10 282"
patients from PM_deficit. In this case, the group-level analysis will likely demonstrate a 283"
significant difference in procedural memory ability between the patient and control groups, 284"
resulting in a finding that conflicts with other findings in the literature. Importantly, it will be 285"
impossible to adjudicate between these conflicting results without a large-scale replication 286"
attempt. However, even in the “ideal” scenario, where the sample is unbiased across subgroups, 287"
we will miss the point: there are meaningful individual differences within the population at large 288"
(i.e. there are some individuals with significant deficits in a given ability that are concealed by 289"
the spared ability in the group analysis). These differences likely matter in assessment and for 290"
treatment decision-making. For example, when rehabilitative intervention in TBI is built around 291"

the premise that the whole population has a spared ability that can be leveraged for improving 292"
outcomes (e.g. errorless-learning interventions dependent on intact procedural learning), that 293"
treatment strategy will not be effective for all individuals, even if it is shown to be efficacious at 294"
the group level. 295"
Unstable Effect Sizes in Small Samples 296"
The typical sample size in TBI research in the field means that most studies have 297"
extremely low statistical power (Rohling et al., 2009). This low statistical power has two 298"
seemingly contradictory consequences: 1) there is a reduced chance of detecting true effects and 299"
2) an increased likelihood that statistically significant results actually reflect (at worst) false 300"
positives or (at best) inflated true effects (Button et al., 2013). The first of these two 301"
consequences is a defining characteristic of low statistical power and is discussed in most 302"
introductory statistics textbooks. The second of these consequences arises as a result of low 303"
statistical power in the context of publication biases (i.e. statistically significant findings are 304"
more likely to published than null findings). In a small-sample study (which will have low 305"
statistical power by definition), the only observed effects that will reach statistical significance 306"
will be those that are either 1) particularly large effects or 2) effects inflated by random noise in 307"
the data (which has a larger influence on effect sizes in small samples). These potential false 308"
positives are more likely to be published by virtue of biases in the incentive structure in scientific 309"
publishing (Button et al., 2013; Ioannidis, Munafo, Fusar-Poli, Nosek, & David, 2014). As a 310"
result, attempts to replicate findings demonstrated in small sample sizes in larger samples are 311"
likely to either 1) demonstrate a failure to replicate or 2) indicate that the true effect is actually 312"
much smaller than suggested by the original small-sample study. 313"

The combination of small sample size and publication biases is most likely to result in 314"
inflated or spurious significant effects. However, in a heterogeneous population with underlying 315"
meaningful subgroups, there is also the potential to conceal important effects that apply only to a 316"
subgroup of the larger population. In treatment research, Kent and colleagues write, “a 317"
treatment’s effect may be null overall, even though it provides substantial benefit in a patient 318"
subgroup” (Kent, Rothwell, Ioannidis, Altman, & Hayward, 2010). In sum, small sample sizes 319"
always limit the replicability and generalizability of our research findings, but this potential is 320"
augmented even further in heterogeneous populations like TBI. The most common historical 321"
approaches to studying TBI have significant flaws that limit our ability to translate research 322"
efforts into meaningful clinical change. 323"
A Shift in Strategy 324"
Thus far, we have discussed the inherent heterogeneity of individuals with TBI: 325"
premorbid patient characteristics, the unique quality of each injury, and resulting variability in 326"
clinical profiles are all likely sources of meaningful inter-individual heterogeneity that may 327"
influence treatment efficacy for an individual patient. Historically, a majority of the field’s basic 328"
science and treatment research has depended on group-level designs and analyses that have clear 329"
disadvantages in the context of this significant heterogeneity. These problems are compounded 330"
by small sample sizes, which results in a high likelihood of biased sampling across meaningful 331"
subgroups as well as imprecise effect size estimates. Taken together, these drawbacks of our 332"
most widely used research designs suggest a new approach is required if we hope to advance 333"
rehabilitation research and better tailor our interventions to individual patients. Next, we 334"
highlight several examples from the field of psychiatry, which serves a number of populations 335"
characterized by high degrees of heterogeneity. We highlight research methods in both basic 336"

science and intervention research that might be fruitfully employed in TBI research to 337"
decompose heterogeneity, predict response to intervention, and measure change in individual 338"
patients. 339"
New Approaches to Understand Heterogeneity: Insights from Psychiatry 340"
Patients with TBI are not the only population of clinical interest that is characterized by 341"
significant heterogeneity. In the field of psychiatry, new methodologies and perspectives are 342"
being applied to understand heterogeneity in autism (Lombardo et al., 2016, 2019), bipolar 343"
disorder (Rheenen et al., 2017), and schizophrenia (Ahmed, Strauss, Buchanan, Kirkpatrick, & 344"
Carpenter, 2018), among many others. Research in psychiatric populations has contended with 345"
similar problems in study design and small samples, leading to calls for approaches that take a 346"
“precision medicine” or “stratified psychiatry” approach (Insel, 2014; Kapur, Phillips, & Insel, 347"
2012); that is, endeavoring to parse heterogeneity within populations in ways that meaningfully 348"
influence treatment decision-making for individual patients. In this section, we describe 349"
approaches to heterogeneity that move away from case-control designs towards methods that 350"
model the inter-individual variability that is a hallmark of these conditions, with two examples of 351"
approaches for both basic science and intervention research. 352"
Modeling Heterogeneity 353"
There are many potential methods for modeling heterogeneity in a given population: the 354"
utility of a given method will depend on the aims of a particular research project. Our goal in this 355"
section is not to provide a comprehensive review of methods that take heterogeneity into 356"
account, nor to suggest that one statistical approach is preferable to another, but rather to provide 357"
a few illustrative examples from both basic science and intervention research to spark further 358"
discussion about the future directions of rehabilitation research. 359"

Basic Science Research 360"
Clustering Approaches. 361"
Unsupervised clustering techniques take multidimensional data and cluster individuals 362"
based on their similarity to one another. Lombardo and colleagues demonstrate the power of 363"
clustering approaches for parsing heterogeneity in an analysis of the “Reading the Mind in the 364"
Eyes” Test (RMET) in adults with autism (Lombardo et al., 2016). In a prior large-sample case-365"
control study, adults with autism performed more poorly on average compared to controls 366"
(Baron-Cohen et al., 2015). Reporting only group-level analyses, however, obscures important 367"
inter-individual differences. In a second study, after applying cluster analyses, the authors 368"
demonstrate reliable subgroups: most individuals with autism are completely unimpaired on the 369"
RMET while only a minority are significantly impaired. Thus, the group level difference in 370"
RMET scores is primarily driven by the significantly impaired ability of a subset of individuals. 371"
Importantly, the reliability of these subgroupings was tested and confirmed in an independent 372"
dataset. The discovery of underlying subgroups allows for the possibility of tailored intervention 373"
approaches for differing response patterns to the RMET. The authors write that, in the context of 374"
mixed intervention results in autism treatment studies, “it may become clearer after subgrouping 375"
that some treatments do systematically work for particular subgroups but not others” (Lombardo 376"
et al., 2016). For a review of the benefits and drawbacks of clustering methods for stratifying 377"
heterogeneous populations, see (Marquand, Wolfers, Mennes, Buitelaar, & Beckmann, 2016). 378"
Normative Modeling. 379"
Another approach to modeling inter-individual heterogeneity is normative modeling. 380"
Normative modeling has been recently utilized to model inter-individual differences in 381"
schizophrenia and bipolar disorder (Wolfers et al., 2018). In this approach, inter-individual 382"

variability in a large healthy sample is modeled, and symptoms in individual patients are 383"
conceptualized as extreme values within this distribution (Marquand et al., 2019; Marquand, 384"
Rezek, Buitelaar, & Beckmann, 2016; Marquand, Wolfers, et al., 2016). Using this approach, 385"
Marquand and colleagues modeled delayed-discounting behavior in a large healthy sample and 386"
identified participants who deviated significantly from this normative model. They found that the 387"
degree of deviance for a particular participant was related to ADHD symptoms (i.e. elevated 388"
hyperactivity) and that participants who deviated significantly from the normative model 389"
demonstrated abnormal brain responses to reward. Normative modeling has a number of 390"
advantages: it allows for statistical inference at the level of the individual, it models healthy 391"
variation in the typical population, and it models deviation away from this normal functioning 392"
without artificially subgrouping participants. 393"
Intervention Research 394"
Moderation Analyses and Modeling Heterogeneous Treatment Effects. 395"
In intervention studies, heterogeneity in response to treatment may be masked by group-396"
level analyses. If there are clinically-meaningful subgroups embedded in the treatment sample, 397"
the “average” treatment effect as determined by group-level analysis may not apply to a 398"
significant proportion of the patients enrolled in treatment. In these cases, moderation analyses 399"
allow for exploration of whether particular patient characteristics influence the magnitude or 400"
direction of the group-level treatment effect (Kraemer, Wilson, Fairburn, & Agras, 2002). These 401"
analyses are initially exploratory: following the completion of an RCT and analysis for the main 402"
effect of treatment, researchers can test whether individual differences in baseline characteristics 403"
(e.g. age, education, injury severity) influence who benefits from intervention by testing for 404"
Treatment*Moderator interactions. Where this interaction is significant, the effect of treatment 405"

for individual participants depends on their level of the moderator, and indicates for whom the 406"
treatment is most effective. This approach differs from others (e.g. ANCOVA), in which we 407"
include covariates to “control” for them (i.e. what would be the effect of treatment if all 408"
participants had the same scores on this covariate?). Instead, in moderation analyses we are 409"
interested in the influence of the covariate on the effect of treatment (i.e. what effect does 410"
treatment have on participants with different scores on this covariate?) (Brown, 2014). 411"
Importantly, moderators identified in these post-hoc analyses need to be validated in a new 412"
sample, with a new RCT powered for the interaction between treatment and the identified 413"
baseline characteristic. 414"
Moderation analyses that have not been validated in an independent sample run into 415"
similar statistical issues as small-sample group designs: because these analyses are often 416"
underpowered, they are prone to false positives (Kent, Steyerberg, & Van Klaveren, 2018; 417"
Naggara, Raymond, Guilbert, & Altman, 2011). One alternative approach to testing the potential 418"
moderating effects of these characteristics “one-at-a-time” is pre-specified subgroup analyses 419"
based upon multidimensional (rather than single variable) subgroups (Kent et al., 2010). For 420"
example, heterogeneity in behavioral treatment response was demonstrated based on subgroup 421"
membership in a treatment study of individuals with binge-eating disorder (Sysko, Hildebrandt, 422"
Wilson, Willfely, & Agras, 2010). A latent class analysis, completed prior to treatment, clustered 423"
patients into four underlying subgroups. Following randomized assignment to one of three 424"
different treatment strategies, response to intervention across subgroups was examined. Two of 425"
these subgroups demonstrated improvements regardless of which treatment was administered. 426"
Each of the remaining two subgroups demonstrated improvement with only one (but opposing) 427"
treatment methods. This demonstrates the potential of models that take heterogeneity into 428"

account for appropriate matching of individual patients to particular treatments, maximizing 429"
treatment gains based on patient profiles. 430"
Measures of Individual Change. 431"
Rehabilitation following TBI is characterized by assessment, treatment based on results 432"
of this assessment, and periodic re-assessment to evaluate whether treatment is inducing change. 433"
New approaches for assessing the significance of treatment-induced change may be useful for 434"
evaluating individual-level response to intervention in the context of heterogeneity. Weiss, Wang 435"
and colleagues have developed Item Response Theory (IRT)-based methods for assessing 436"
significant longitudinal change (Kim-Kang & Weiss, 2008; Wang, Weiss, & Suen, 2020; Weiss 437"
& Von Minden, 2011). These methods allow for the assessment of significant change over time 438"
at the individual patient level. Wang and colleagues tested the utility of these new methods in an 439"
acute rehabilitation setting. Patients in in-patient rehab completed a computer-adaptive patient-440"
reported outcome measure on multiple occasions during their hospital stay. Results demonstrated 441"
that, at the individual patient level, some patients showed significant improvement over time, 442"
others did not demonstrate change, and still others significantly declined over time. Use of these 443"
methods may allow for parsing heterogeneity in treatment response by identifying which patients 444"
improve with treatment at the individual level, compared to those who do not. 445"
Requirements of Heterogeneity-Focused Research Designs 446"
Larger sample sizes and multidimensional data 447"
Increasing sample sizes is a perennial call in TBI research, even for traditional research 448"
designs. However, for designs that characterize and decompose heterogeneity, large sample sizes 449"
are imperative. It is critical that we recruit samples across the full range of heterogeneous 450"
characteristics within the larger population. (Notably, each of the highlighted studies described in 451"

the previous section reported sample sizes of greater than 200 participants). In addition, data 452"
should be multidimensional, with multiple measures within each individual participant that span 453"
multiple levels of analysis. Collecting “broad” (across many people) and “deep” (multiple 454"
measures across levels of analysis) datasets allow us to leverage heterogeneity at one level of 455"
analysis to predict outcomes at another level of analysis (Lombardo et al., 2019; Zhao & 456"
Castellanos, 2016). 457"
Leveraging Heterogeneity in Learning and Memory Profiles 458"
As described in the previous section, there are many combinations of variables, spanning 459"
multiple levels of analysis, by which we might decompose heterogeneity in TBI. This is a 460"
significant barrier to overcome: it will always be possible to cluster patients based on some set of 461"
characteristics, but whether or not these subgroupings are useful depends on whether revealed 462"
subgroups could have an impact on clinical decision-making or improve prediction of outcomes 463"
at a different level analysis than the groupings themselves. 464"
It is our perspective that decomposing heterogeneity in cognitive sequelae will be the 465"
most useful in advancing behavioral rehabilitation, as this is the level at which clinicians in our 466"
field are seeking to modify patient outcomes. Therefore, we propose that a helpful framework for 467"
accounting for inter-individual differences in TBI is one that parses patterns of impaired and 468"
intact learning and memory across multiple memory systems. Memory deficits are common 469"
following TBI, but there is significant inter-individual heterogeneity in the degree and quality of 470"
these memory impairments. In addition, characterizing heterogeneity in memory ability may 471"
have direct impacts on clinical decision-making: any memory dysfunction likely impacts all 472"
behavioral treatment efforts. That is, the success of any behavioral intervention that requires 473"
(re)learning new information, a new strategy, or new skill may be impacted by the presence of 474"

memory impairment. A characterization of a patient’s memory and learning strengths and 475"
weakness bears directly on all aspects of their post-acute rehabilitation; most of which involves 476"
engaging in behavioral therapies. These behavioral therapies each require that patients learn and 477"
retain new or previously-learned information. Thus, characterization of heterogeneity in memory 478"
profiles could provide wide-reaching benefits to patients and to clinicians in choosing 479"
appropriate interventions. 480"
To determine whether patients with TBI can be clustered based on meaningful individual 481"
differences in memory ability, we are collecting data that is both “broad” and “deep”. We are 482"
recruiting a sample of 100 individuals with moderate-severe TBI, and are collecting data on 483"
memory ability across multiple memory systems (i.e. declarative, procedural, working memory; 484"
with multiple measures for each system). Patients will be clustered based on these memory 485"
measures using an unsupervised agglomerative hierarchical clustering algorithm. We will test 486"
whether patient subgroups derived from this clustering analysis are predictive of outcomes at 487"
other levels of analysis, including psychosocial re-integration and neuroimaging findings. This 488"
initial work is inherently exploratory – if meaningful subgroups emerge, we will need to validate 489"
these memory and learning profiles in a new, independent sample to ensure they are replicable. If 490"
valid and robust memory and learning profiles do emerge, we can test the effectiveness of 491"
different interventions by subgroup (as in Sysko et al., 2010; described above). 492"
Case-control studies in TBI have, in most cases, suggested impairments in declarative 493"
memory and preserved procedural memory in patients with moderate-severe TBI (Vakil, 2005). 494"
The assumption that procedural memory is spared in TBI and can be leveraged in rehabilitation 495"
serves as the foundation of a number of therapy approaches (e.g. errorless learning; Glisky, 496"
1993). However, given the inherent weaknesses of case-control studies in heterogeneous 497"

populations, described previously, it may be possible that group-level analyses are obscuring 498"
subgroups of patients with impaired procedural memory. In addition, there are conflicting 499"
findings suggesting impairments in procedural memory in some samples (e.g. Kraus, Little, 500"
Wojtowicz, & Sweeney, 2010; Vakil, Kraus, Bor, & Groswasser, 2002). Therefore, the assertion 501"
that declarative memory is impaired in TBI while procedural memory is spared may not hold 502"
true for the population as a whole: meaningful inter-individual differences in memory ability 503"
across memory systems may impact an individual patient’s ability to benefit from treatments that 504"
rely heavily on a particular memory system. 505"
Future Directions: A Call for Rich Data in TBI Rehabilitation Research 506"
Following TBI, cognitive-communication impairments are common and often chronic, 507"
yet there have been no significant advances in reducing the rate of brain injury-related disability 508"
over the past several decades (Roozenbeek, Maas, & Menon, 2013). There are many sources of 509"
heterogeneity in TBI that make treatment decision-making challenging. We propose that these 510"
factors ought not to be “controlled” but rather explored. To facilitate this shift in perspective, 511"
we’ve surveyed a set of methodologies, pointing to work in psychiatry that addresses similar 512"
issues of heterogeneity in patient profiles and response to intervention. The methods laid out in 513"
this paper are not meant to be exhaustive, nor prescriptive, and we recognize that there are 514"
challenges and limitations that need to be addressed in the nearer term. We hope to further the 515"
discussions initiated at ICCDC and make initial steps towards a framework that considers 516"
heterogeneity from the outset. What follows are a set of potential next steps. 517"
Larger Samples 518"
A major hurdle impeding the approaches laid out in this article is the lack of “big data” in 519"
our subfield. For any approach that seeks to decompose heterogeneity, larger sample sizes than 520"

are historically typical in our field are required. There may, however, be an upside to changing 521"
perspective from heterogeneity-as-limitation to heterogeneity-as-opportunity: by seeking to 522"
sample across heterogeneity, researchers are freed from applying stringent exclusionary criteria 523"
in hopes of obtaining a “clean” sample. Instead, we can exclude fewer patients, enroll more 524"
participants, and leverage methods for interrogating heterogeneous patient characteristics to 525"
better align our treatment strategies to individual patients in the clinic. 526"
Nevertheless, even with this “loosening” of exclusionary criteria, reaching sample sizes 527"
large enough to achieve adequate power will likely require collaborative, multi-site projects. 528"
There are several large-scale data sets in TBI research (e.g. TBI Model Systems). However, for 529"
the purposes of our field these data sets lack richness: while the datasets are “broad” (large 530"
sample sizes), the data collected that is of interest to rehabilitation scientists is thin, rather than 531"
“deep”. For example, large-scale outcomes research in TBI may only collect a single self-report 532"
measure relevant to cognition or communication. The acquisition of a large data set that is both 533"
broad and deep, and of relevance to cognitive-communication rehabilitation research will be a 534"
challenging undertaking: collecting this kind of data requires significant time with each 535"
participant and multiple return visits. To meet this challenge, as a field we should move towards 536"
deep data sets that can be combined, to increase their breadth and improve our samples so that 537"
they are representative of the range of heterogeneity inherent in this population. 538"
We note that increased sample size will not only afford the opportunity to deploy new 539"
statistical methods for decomposing heterogeneity, but will also vastly improve the quality of 540"
more traditional research designs. Increased sample sizes in case-control designs increases the 541"
likelihood that our samples are reflective of the broader population, improves the likelihood that 542"
results will replicate, and improves the precision of effect size estimates. An approach that has 543"

been fruitful in our own lab for increasing sample sizes has been the establishment of a patient 544"
registry, to facilitate longitudinal data collection and data sets that are both deep and broad (also 545"
see Fellows, Stark, Berg, & Chatterjee, 2008). 546"
Improved Reporting and Measurement 547"
Systematic reviews and meta-analyses suggest scattershot reporting across studies, with 548"
insufficient descriptions of treatment and patient characteristics. One particular challenge arises 549"
in characterizing injury characteristics from incomplete medical records – many of the metrics 550"
by which we judge injury severity are not recoverable if they have not been noted in a patient’s 551"
medical record (e.g., GCS or loss of consciousness). In these cases, we should collect and report 552"
as much injury characterization as possible, but this common problem suggests the use of 553"
categorization schemes that work across missing clinical data (e.g. the Mayo Classification Scale 554"
for TBI severity; Malec et al., 2007). Where possible, we should report and include TBI 555"
Common Data Elements, to improve data aggregation and comparisons across studies (Meeuws 556"
et al., 2020). Detailed reporting of treatment characteristics is also critical; many intervention 557"
studies provide inadequate detail to allow for replication or comparison across trials (Whyte & 558"
Hart, 2003). With journals no longer constrained by physical, print format, this lack of detail is 559"
no longer excusable (see Open Science Practices, below). 560"
Collecting data that is both “broad” and “deep” will only be effective insofar as the 561"
measures that we use to quantify patient characteristics are psychometrically sound. Nothing will 562"
be gained by new approaches if we simply increase the number of poor quality measurements. In 563"
an ongoing systematic review of memory interventions following TBI, we have found a striking 564"
lack of consensus in outcome measures, with many investigators utilizing ad hoc measures 565"
without attempts to ascertain their reliability or validity (Covington, de Riesthal, & Duff, 2018). 566"

Open Science Practices 567"
There are many resources available for improving the rigor, reproducibility, and 568"
replicability of our science (Munafò et al., 2017). Among these are pre-registration of study 569"
design and analysis plans, open data sharing, and an increased focus on replication of research 570"
findings. These practices are particularly critical for ensuring the utility of data collected and 571"
analyzed with heterogeneity in mind: there are many reasonable ways to analyze 572"
multidimensional data. Pre-registration of study design and analysis plans mitigates against p-573"
hacking or “fishing expeditions” and could reduce bias and rates of false positives in the 574"
literature. Options for pre-registration include the Open Science Framework (osf.io) and 575"
AsPredicted (aspredicted.org) for basic and treatment research and PROSPERO 576"
(crd.york.ac.unk/prospero) for systematic reviews and meta-analyses. 577"
Upon study completion, data, experimental measures, and treatment protocols can be 578"
openly shared on open data repositories (e.g. Open Science Framework; osf.io). These 579"
repositories allow for centralized sharing of study stimuli, presentation and analysis scripts, and 580"
protocols at a level of detail that is not feasible in journal articles. Sharing of these materials 581"
allows for direct replication of basic science and treatment studies, which are critical for ensuring 582"
we can trust published research findings. 583"
Conclusions 584"
Heterogeneity in patient characteristics, injuries, and outcomes following TBI are not 585"
going away: we would do well to accept this as a key feature in TBI research. Heterogeneity 586"
across these multiple levels of analysis is indeed a significant limitation, if we continue to draw 587"
conclusions based upon under-powered case-control designs. Fortunately, we have at our 588"
disposal a number of new approaches that will allow us to leverage the heterogeneity that is a 589"

hallmark of TBI and build models that may help to predict outcomes across levels of analysis 590"
and better assign particular patients to effective treatments. To do this well, we will need to 591"
commit to large sample sizes and rich data collection. This is not an easy endeavor and will 592"
require collaboration across labs. But, we cannot no longer delude ourselves into thinking that 593"
small sample case-control studies alone are serving us well – decades without appreciable 594"
changes in TBI-related disability suggest that a new approach is long overdue. 595"
596"
Acknowledgements: The authors thank the attendees of the International Cognitive-597"
Communication Disorders Conference, for discussions that prompted this manuscript. 598"
599"
600"
References 601"
Ahmed, A. O., Strauss, G. P., Buchanan, R. W., Kirkpatrick, B., & Carpenter, W. T. (2018). 602"
Schizophrenia heterogeneity revisited: Clinical, cognitive, and psychosocial correlates of 603"
statistically-derived negative symptoms subgroups. Journal of Psychiatric Research, 97, 8–604"
15. 605"
Amoroso, T., & Iverson, K. M. (2017). Acknowledging the risk for traumatic brain injury in 606"
women veterans. Journal of Nervous and Mental Disease, 205(4), 318–323. 607"
Andelic, N., Ye, J., Tornas, S., Roe, C., Lu, J., Bautz-Holter, E., … Aas, E. (2014). Cost-608"
Effectiveness Analysis of an Early-Initiated, Continuous Chain of Rehabilitation after 609"
Severe Traumatic Brain Injury. Journal of Neurotrauma, 31(14), 1313–1320. 610"
Baron-Cohen, S., Bowen, D. C., Holt, R. J., Allison, C., Auyeung, B., Lombardo, M. V., … Lai, 611"
M. C. (2015). The “reading the mind in the eyes” test: Complete absence of typical sex 612"

difference in 400 men and women with autism. PLoS ONE, 10(8), 1–17. 613"
Bazarian, J. J., Blyth, B., Mookerjee, S., He, H., & McDermott, M. P. (2010). Sex Differences in 614"
Outcome after Mild Traumatic Brain Injury. Journal Of Neurotrauma, 27(3), 527–539. 615"
Bigler, E. D., Farrer, T. J., Pertab, J. L., James, K., Petrie, J. A., & Hedges, D. W. (2013). 616"
Reaffirmed limitations of meta-analytic methods in the study of mild traumatic brain injury: 617"
A response to Rohling et al. Clinical Neuropsychologist, 27(2), 176–214. 618"
Brown, J. D. (2014). Moderation. In Linear Models in Matrix Form. Springer, Cham. 619"
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & 620"
Munafò, M. R. (2013). Power failure: Why small sample size undermines the reliability of 621"
neuroscience. Nature Reviews Neuroscience, 14(5), 365–376. 622"
Cicerone, K. D., Dahlberg, C., Kalmar, K., Langenbahn, D. M., Malec, J. F., Bergquist, T. F., … 623"
Morse, P. A. (2000). Evidence-based cognitive rehabilitation: Recommendations for clinical 624"
practice. Archives of Physical Medicine and Rehabilitation, 81(12), 1596–1615. 625"
Cicerone, K. D., Dahlberg, C., Malec, J. F., Langenbahn, D. M., Felicetti, T., Kneipp, S., … 626"
Catanese, J. (2005). Evidence-based cognitive rehabilitation: Updated review of the 627"
literature from 1998 through 2002. Archives of Physical Medicine and Rehabilitation, 86(8), 628"
1681–1692. 629"
Cicerone, K. D., Langenbahn, D. M., Braden, C., Malec, J. F., Kalmar, K., Fraas, M., … 630"
Ashman, T. (2011). Evidence-Based Cognitive Rehabilitation: Updated Review of the 631"
Literature From 2003 Through 2008. ARCHIVES OF PHYSICAL MEDICINE AND 632"
REHABILITATION, 92(4), 519–530. 633"
Cicerone, K. D., Mott, T., Azulay, J., Sharlow-Galella, M. A., Ellmo, W. J., Paradise, S., & Friel, 634"
J. C. (2008). A Randomized Controlled Trial of Holistic Neuropsychologic Rehabilitation 635"

After Traumatic Brain Injury. Archives of Physical Medicine and Rehabilitation, 89(12), 636"
2239–2249. 637"
Cooney, M. T., & Carroll, A. (2016). Cost effectiveness of inpatient rehabilitation in patients 638"
with brain injury. Clinical Medicine, Journal of the Royal College of Physicians of London, 639"
16(2), 109–113. 640"
Coronado, V. G., Xu, L., Basavaraju, S. V, McGuire, L. C., Wald, M. M., Faul, M. D., … 641"
Centers for Disease Control and Prevention (CDC). (2011). Surveillance for traumatic brain 642"
injury-related deaths--United States, 1997-2007. Morbidity and Mortality Weekly Report. 643"
Surveillance Summaries (Washington, D.C.!: 2002), 60(5), 1–32. 644"
Corrigan, J. D., Cuthbert, J. P., Whiteneck, G. G., Dijkers, M. P., Coronado, V., Heinemann, A. 645"
W., … Graham, J. E. (2012). Representativeness of the Traumatic Brain Injury Model 646"
Systems National Database. JOURNAL OF HEAD TRAUMA REHABILITATION, 27(6), 647"
391–403. 648"
Covington, N. V, de Riesthal, M., & Duff, M. C. (2018). A systematic review of memory 649"
interventions in traumatic brain injury. PROSPERO CRD42018089361. 650"
Dahdah, M. N., Barnes, S., Buros, A., Dubiel, R., Dunklin, C., Callender, L., … Shafi, S. (2016). 651"
Variations in Inpatient Rehabilitation Functional Outcomes Across Centers in the Traumatic 652"
Brain Injury Model Systems Study and the Influence of Demographics and Injury Severity 653"
on Patient Outcomes. Archives of Physical Medicine & Rehabilitation, 97(11), 1821–1831. 654"
de la Plata, C. A. M., Hart, T., Hammond, F. H., Frol, A. B., Hudak, A., Harper, C. R., … Diaz-655"
Arrastia, R. (2008). Impact of age on long-term recovery from traumatic brain injury. 656"
ARCHIVES OF PHYSICAL MEDICINE AND REHABILITATION, 89(5), 896–903. 657"
Elliott, M., & Parente, F. (2014). Efficacy of memory rehabilitation therapy: a meta-analysis of 658"

TBI and stroke cognitive rehabilitation literature. Brain Injury, 28(12), 1610–1616. 659"
Fellows, L. K., Stark, M., Berg, A., & Chatterjee, A. (2008). Patient registries in cognitive 660"
neuroscience research: Advantages, challenges, and practical advice. Journal of Cognitive 661"
Neuroscience, 20(6), 1107–1113. 662"
Finnanger, T. G., Skandsen, T., Andersson, S., Lydersen, S., Vik, A., & Indredavik, M. (2013). 663"
Differentiated patterns of cognitive impairment 12 months after severe and moderate 664"
traumatic brain injury. Brain Injury, 27(13–14), 1606–1616. 665"
Frencham, K. A. R., Fox, A. M., & Maybery, M. T. (2005). Neuropsychological studies of mild 666"
traumatic brain injury: A meta-analytic review of research since 1995. Journal of Clinical 667"
and Experimental Neuropsychology, 27(3), 334–351. 668"
Gaastra, B., Longworth, A., Matta, B., Snelson, C., Whitehouse, T., Murphy, N., & Veenith, T. 669"
(2016). The ageing population is neglected in research studies of traumatic brain injury. 670"
British Journal of Neurosurgery, 30(2), 221–226. 671"
Gardner, R. C., Dams-O’Connor, K., Morrissey, M. R., & Manley, G. T. (2018). Geriatric 672"
Traumatic Brain Injury: Epidemiology, Outcomes, Knowledge Gaps, and Future Directions. 673"
Journal of Neurotrauma, 35(7), 889–906. 674"
Geurtsen, G. J., van Heugten, C. M., Martina, J. D., & Geurts, A. C. (2010). Comprehensive 675"
rehabilitation programmes in the chronic phase after severe brain injury: a systematic 676"
review. Journal of Rehabilitation Medicine (Stiftelsen Rehabiliteringsinformation), 42(2), 677"
97–110. 678"
Glisky, E. (1993). Training Persons with Traumatic Brain Injury for Complex Computer Jobs: 679"
The Domain-Specific Approach. In D. Thomas, F. Menz, & D. McAlees (Eds.), 680"
Community-Based Employment Following Traumatic Brain Injury (pp. 3–29). 681"

Goranson, T. E., Graves, R. E., Allison, D., & La Freniere, R. (2003). Community integration 682"
following multidisciplinary rehabilitation for traumatic brain injury. Brain Injury, 17(9), 683"
759–774. 684"
Griesbach, G. S., Kreber, L. A., Harrington, D., & Ashley, M. J. (2015). Post-acute traumatic 685"
brain injury rehabilitation: Effects on outcome measures and life care costs. Journal of 686"
Neurotrauma, 32(10), 704–711. 687"
Hart, T., Kozlowski, A. J., Whyte, J., Poulsen, I., Kristensen, K., Nordenbo, A., & Heinemann, 688"
A. W. (2014). Functional Recovery After Severe Traumatic Brain Injury: An Individual 689"
Growth Curve Approach. Archives of Physical Medicine and Rehabilitation, 95(11), 2103–690"
2110. 691"
Hawryluk, G. W. J., & Manley, G. T. (2015). Classification of traumatic brain injury: past, 692"
present, and future. In J. Grafman & A. Salazar (Eds.), Handbook of Clinical Neurology 693"
(3rd ed., Vol. 127, pp. 15–21). Elsevier. 694"
Insel, T. R. (2014). The NIMH Research Domain Criteria (RDoC) Project: Precision Medicine 695"
for Psychiatry. American Journal of Psychiatry, 171(4), 395–397. 696"
Ioannidis, J. P. A., Munafo, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014). 697"
Publication and other reporting biases in cognitive sciences: detection, prevalence and 698"
prevention. Trends in Cognitive Sciences, 18(5), 235–241. 699"
Kapur, S., Phillips, A. G., & Insel, T. R. (2012). Why has it taken so long for biological 700"
psychiatry to develop clinical tests and what to do about it. Molecular Psychiatry, 17(12), 701"
1174–1179. 702"
Kennedy, M. R. T., Coelho, C., Turkstra, L., Ylvisaker, M., Moore Sohlberg, M., Yorkston, K., 703"
… Kan, P. F. (2008). Intervention for executive functions after traumatic brain injury: A 704"

systematic review, meta-analysis and clinical recommendations. Neuropsychological 705"
Rehabilitation, 18(3), 257–299. 706"
Kent, D. M., Rothwell, P. M., Ioannidis, J. P. A., Altman, D. G., & Hayward, R. A. (2010). 707"
Assessing and reporting heterogeneity in treatment effects in clinical trials: A proposal. 708"
Trials, 11, 1–11. 709"
Kent, D. M., Steyerberg, E., & Van Klaveren, D. (2018). Personalized evidence based medicine: 710"
Predictive approaches to heterogeneous treatment effects. BMJ (Online), 363. 711"
Kim-Kang, G., & Weiss, D. J. (2008). Adaptive measurement of individual change. Journal of 712"
Psychology, 216(1), 49–58. 713"
Konigs, M., Beurskens, E. A., Snoep, L., Scherder, E. J., & Oosterlaan, J. (2018). Effects of 714"
Timing and Intensity of Neurorehabilitation on Functional Outcome After Traumatic Brain 715"
Injury: A Systematic Review and Meta-Analysis. Archives of Physical Medicine and 716"
Rehabilitation. 717"
Kraemer, H. C., Wilson, G. T., Fairburn, C. G., & Agras, W. S. (2002). Mediators and 718"
moderators of treatment effects in randomized clinical trials. Archives of General 719"
Psychiatry, 59(10), 877–883. 720"
Kraus, M., Little, D., Wojtowicz, S., & Sweeney, J. (2010). Procedural learning impairments 721"
identified via predictive saccades in chronic traumatic brain injury. Cognitive and 722"
Behavioral Neurology, 23(4), 210–217. 723"
Lah, S., Epps, A., Levick, W., & Parry, L. (2011). Implicit and explicit memory outcome in 724"
children who have sustained severe traumatic brain injury: impact of age at injury 725"
(preliminary findings). Brain Injury, 25(1), 44–52. 726"
Lombardo, M. V., Lai, M. C., Auyeung, B., Holt, R. J., Allison, C., Smith, P., … Baron-Cohen, 727"

S. (2016). Unsupervised data-driven stratification of mentalizing heterogeneity in autism. 728"
Scientific Reports, 6(April), 1–15. 729"
Lombardo, M. V., Lai, M. C., & Baron-Cohen, S. (2019). Big data approaches to decomposing 730"
heterogeneity across the autism spectrum. Molecular Psychiatry, 24(10), 1435–1450. 731"
Lu, J., Gary, K. W., Neimeier, J. P., Ward, J., & Lapane, K. L. (2012). Randomized controlled 732"
trials in adult traumatic brain injury. Brain Injury, 26(December), 1523–1548. 733"
Maas, A. I. R., Menon, D. K., Lingsma, H. F., Pineda, J. A., Sandel, M. E., & Manley, G. T. 734"
(2012). Re-orientation of clinical research in traumatic brain injury: Report of an 735"
international workshop on comparative effectiveness research. Journal of Neurotrauma, 736"
29(1), 32–46. 737"
Malec, J. F., Brown, A. W., Leibson, C. L., Flaada, J. T., Mandrekar, J. N., Diehl, N. N., & 738"
Perkins, P. K. (2007). The mayo classification system for traumatic brain injury severity. 739"
Journal of Neurotrauma, 24(9), 1417–1424. 740"
Marquand, A. F., Kia, S. M., Zabihi, M., Wolfers, T., Buitelaar, J. K., & Beckmann, C. F. 741"
(2019). Conceptualizing mental disorders as deviations from normative functioning. 742"
Molecular Psychiatry, 24(10), 1415–1424. 743"
Marquand, A. F., Rezek, I., Buitelaar, J., & Beckmann, C. F. (2016). Understanding 744"
Heterogeneity in Clinical Cohorts Using Normative Models: Beyond Case-Control Studies. 745"
Biological Psychiatry, 80(7), 552–561. 746"
Marquand, A. F., Wolfers, T., Mennes, M., Buitelaar, J., & Beckmann, C. F. (2016). Beyond 747"
Lumping and Splitting: A Review of Computational Approaches for Stratifying Psychiatric 748"
Disorders. Biological Psychiatry: Cognitive Neuroscience and Neuroimaging, 1(5), 433–749"
447. 750"

McDonald, S. (2013). Impairments in social cognition following severe traumatic brain injury. 751"
Journal of the International Neuropsychological Society, 19(3), 231–246. 752"
Meeuws, S., Yue, J. K., Huijben, J. A., Nair, N., Lingsma, H. F., Bell, M. J., … Maas, A. (2020). 753"
Common Data Elements – A critical assessment of harmonization between current 754"
multicenter traumatic brain injury studies. Journal of Neurotrauma, 8, 1–8. 755"
Mollayeva, T., Mollayeva, S., & Colantonio, A. (2018). Traumatic brain injury: sex, gender and 756"
intersecting vulnerabilities. Nature Reviews Neurology, 14(12), 711–722. 757"
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie Du Sert, 758"
N., … Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human 759"
Behaviour, 1(1), 1–9. 760"
Naggara, O., Raymond, J., Guilbert, F., & Altman, D. G. (2011). The problem of subgroup 761"
analyses: An example from a trial on ruptured intracranial aneurysms. American Journal of 762"
Neuroradiology, 32(4), 633–636. 763"
O’Neil-Pirozzi, T. M., Kennedy, M. R. T., & Sohlberg, M. M. (2016). Evidence-Based Practice 764"
for the Use of Internal Strategies as a Memory Compensation Technique After Brain Injury: 765"
A Systematic Review. The Journal of Head Trauma Rehabilitation, 31(4), E1–E11. 766"
Pagulayan, K. F., Temkin, N. R., Machamer, J., & Dikmen, S. S. (2006). A Longitudinal Study 767"
of Health-Related Quality of Life After Traumatic Brain Injury. Archives of Physical 768"
Medicine and Rehabilitation, 87(5), 611–618. https://doi.org/10.1016/j.apmr.2006.01.018 769"
Pertab, J. L., James, K. M., & Bigler, E. D. (2009). Limitations of mild traumatic brain injury 770"
meta-analyses. Brain Injury, 23(6), 498–508. 771"
Ponsford, J. L., Downing, M. G., Olver, J., Ponsford, M., Acher, R., Carty, M., & Spitz, G. 772"
(2014). Longitudinal Follow-Up of Patients with Traumatic Brain Injury: Outcome at Two, 773"

Five, and Ten Years Post-Injury. Journal of Neurotrauma, 31(1), 64–77. 774"
Rabinowitz, A. R., & Levin, H. S. (2014). Cognitive Sequelae of Traumatic Brain Injury. 775"
Psychiatric Clinics of North America, 37(1), 1–11. 776"
Rheenen, T. E. Van, Lewandowski, K., Tan, E. J., Ospina, L. H., Ongur, O., Neill, E., … 777"
Burdick, K. E. (2017). Characterizing cognitive heterogeneity on the schizophrenia – 778"
bipolar disorder spectrum. Psychological Medicine, 47(10), 1848-1864. 779"
Robey, R., & Schultz, M. (1998). A model for conducting clinical-outcome research: An 780"
adaptation of the standard protocol for use in aphasiology. Aphasiology, 12(9), 787–810. 781"
Rohling, M. L., Faust, M. E., Beverly, B., & Demakis, G. (2009). Effectiveness of cognitive 782"
rehabilitation following acquired brain injury: A meta-analytic re-examination of Cicerone 783"
et al.’s (2000, 2005) systematic reviews. Neuropsychology, 23(1), 20–39. 784"
Roozenbeek, B., Maas, A. I. R., & Menon, D. K. (2013). Changing patterns in the epidemiology 785"
of traumatic brain injury. Nature Reviews Neurology, 9(4), 231–236. 786"
Rosenbaum, S. B., & Lipton, M. L. (2012). Embracing chaos: The scope and importance of 787"
clinical and pathological heterogeneity in mTBI. Brain Imaging and Behavior, 6(2), 255–788"
282. 789"
Saatman, K. E., Duhaime, A. C., Bullock, R., Maas, A. I., Valadka, A., & Manley, G. T. (2008). 790"
Classification of traumatic brain injury for targeted therapies. J.Neurotrauma., 25(7), 719–791"
738. 792"
Saban, K. L., Smith, B. M., Collins, E. G., & Pape, T. L.-B. (2011). Sex differences in perceived 793"
life satisfaction and functional status one year after severe traumatic brain injury. Journal of 794"
Women’s Health (15409996), 20(2), 179–186. 795"
Schneider, E. B., Sur, S., Raymont, V., Duckworth, J., Kowalski, R. G., Efron, D. T., … Stevens, 796"

R. D. (2014). Functional recovery after moderate/severe traumatic brain injury A role for 797"
cognitive reserve? NEUROLOGY, 82(18), 1636–1642. 798"
Senathi-Raja, D., Ponsford, J., & Schönberger, M. (2010). Impact of age on long-term cognitive 799"
function after traumatic brain injury. Neuropsychology, 24(3), 336–344. 800"
Sherer, M., Struchen, M. A., Yablon, S. A., Wang, Y., & Nick, T. G. (2008). Comparison of 801"
indices of traumatic brain injury severity: Glasgow Coma Scale, length of coma and post-802"
traumatic amnesia. Journal of Neurology, Neurosurgery & Psychiatry, 79(6), 678–685. 803"
Stolwyk, R. J., Gooden, J. R., Kim, J., & Cadilhac, D. A. (2019). What is known about the cost-804"
effectiveness of neuropsychological interventions for individuals with acquired brain 805"
injury? A scoping review. Neuropsychological Rehabilitation, 0(0), 1–29. 806"
Sysko, R., Hildebrandt, T., Wilson, G. T., Willfely, D., & Agras, W. S. (2010). Heterogeneity 807"
Moderates Treatment Response among Patients with Binge Eating Disorder. Journal of 808"
Consulting and Clinical Psychology, 78(5), 681–690. 809"
Tate, R. L., Perdices, M., McDonald, S., Togher, L., & Rosenkoetter, U. (2014). The design, 810"
conduct and report of single-case research: Resources to improve the quality of the 811"
neurorehabilitation literature. Neuropsychological Rehabilitation, 24(3–4), 315–331. 812"
Tate, R. L., Perdices, M., Rosenkoetter, U., McDonald, S., Togher, L., Shadish, W., … Vohra, S. 813"
(2016). The Single-Case Reporting Guideline In BEhavioural Interventions (SCRIBE) 814"
2016: Explanation and elaboration. Archives of Scientific Psychology, 4(1), 10–31. 815"
Teasdale, G., & Jennett, B. (1974). Assessment of Coma and Impaired Consciousness. The 816"
Lancet, 2(7872), 81–84. 817"
Trivedi, R. B., & Humphreys, K. (2015). Participant exclusion criteria in treatment research on 818"
neurological disorders: Are unrepresentative study samples problematic? Contemporary 819"

Clinical Trials, 44, 20–25. 820"
Turkstra, L. S., Mutlu, B., Ryan, C. W., Despins Stafslien, E. H., Richmond, E. K., Hosokawa, 821"
E., & Duff, M. C. (2020). Sex and Gender Differences in Emotion Recognition and Theory 822"
of Mind After TBI: A Narrative Review and Directions for Future Research. Frontiers in 823"
Neurology, 11(February), 1–9. 824"
Vakil, E. (2005). The effect of moderate to severe traumatic brain injury (TBI) on different 825"
aspects of memory: a selective review. Journal of clinical and experimental 826"
neuropsychology (Vol. 27). 827"
Vakil, E., Kraus, A., Bor, B., & Groswasser, Z. (2002). Impaired skill learning in patients with 828"
severe closed-head injury as demonstrated by the serial reaction time (SRT) task. Brain and 829"
Cognition, 50(2), 304–315. 830"
Verger, K., Junqué, C., Jurado, M. A., Tresserras, P., Bartumeus, F., Nogués, P., & Poch, J. M. 831"
(2000). Age effects on long-term neuropsychological outcome in paediatric traumatic brain 832"
injury. Brain Injury, 14(6), 495–503. 833"
Wang, C., Weiss, D. J., & Suen, K. Y. (2020). Hypothesis Testing Methods for Multivariate 834"
Multi-Occasion Intra-Individual Change. Multivariate Behavioral Research, 0(0), 1–17. 835"
Weiss, D. J., & Von Minden, S. (2011). Measuring Individual Growth With Conventional and 836"
Adaptive Tests. Journal of Methods and Measurement in the Social Sciences, 2(2), 80. 837"
Whyte, J., & Hart, T. (2003). It’s More Than a Black Box; It’s a Russian Doll. American Journal 838"
of Physical Medicine & Rehabilitation, 82(8), 639–652. 839"
Wolfers, T., Doan, N. T., Kaufmann, T., Alnæs, D., Moberget, T., Agartz, I., … Marquand, A. F. 840"
(2018). Mapping the Heterogeneous Phenotype of Schizophrenia and Bipolar Disorder 841"
Using Normative Models. JAMA Psychiatry, 75(11), 1146–1155. 842"

Zaloshnja, E., Miller, T., Langlois, J. A., & Selassie, A. W. (2008). Prevalence of Long-Term 843"
Disability From Traumatic Brain Injury in the Civilian Population of the United States, 844"
2005. Journal of Head Trauma Rehabilitation, 23(6), 394–400. 845"
Zhao, Y., & Castellanos, F. X. (2016). Annual Research Review: Discovery science strategies in 846"
studies of the pathophysiology of child and adolescent psychiatric disorders - Promises and 847"
limitations. Journal of Child Psychology and Psychiatry and Allied Disciplines, 57(3), 421–848"
439. 849"
850"
851"