Available via license: CC BY

Content may be subject to copyright.

RESEARCH ARTICLE

Asymmetrical reliability of the Alda score

favours a dichotomous representation of

lithium responsiveness

Abraham NunesID

1,2

, Thomas TrappenbergID

2

, Martin Alda

1¤

*, The international

Consortium on Lithium Genetics (ConLiGen)

¶

1Department of Psychiatry, Dalhousie University, Halifax, Nova Scotia, Canada, 2Faculty of Computer

Science, Dalhousie University, Halifax, Nova Scotia, Canada

¤Current address: QEII Health Sciences Centre, Halifax, Nova Scotia, Canada

¶ Membership list can be found in the Acknowledgments section.

*malda@dal.ca

Abstract

The Alda score is commonly used to quantify lithium responsiveness in bipolar disorder.

Most often, this score is dichotomized into “responder” and “non-responder” categories,

respectively. This practice is often criticized as inappropriate, since continuous variables

are thought to invariably be “more informative” than their dichotomizations. We therefore

investigated the degree of informativeness across raw and dichotomized versions of the

Alda score, using data from a published study of the scale’s inter-rater reliability (n = 59 rat-

ers of 12 standardized vignettes each). After learning a generative model for the relationship

between observed and ground truth scores (the latter defined by a consensus rating of the

12 vignettes), we show that the dichotomized scale is more robust to inter-rater disagree-

ment than the raw 0-10 scale. Further theoretical analysis shows that when a measure’s reli-

ability is stronger at one extreme of the continuum—a scenario which has received little-to-

no statistical attention, but which likely occurs for the Alda score 7—dichotomization of a

continuous variable may be more informative concerning its ground truth value, particularly

in the presence of noise. Our study suggests that research employing the Alda score of lith-

ium responsiveness should continue using the dichotomous definition, particularly when

data are sampled across multiple raters.

Introduction

The Alda score is a validated index of lithium responsiveness commonly used in bipolar disor-

der (BD) research [1]. This scale has two components. The first is the “A” subscale that pro-

vides an ordinal score (from 0 to 10, inclusive) of the overall “response” in a therapeutic trial

of lithium. The second component is the “B” subscale that attempts to qualify the degree to

which any improvement was causally related to lithium. The total Alda score is computed

based on these two subscale scores, and takes integer values between 0 and 10. Many studies

that employ the Alda score as a target variable dichotomize it, such that individuals with scores

7 are classified as “responders,” and those with scores <7 are “non-responders.”

PLOS ONE | https://doi.org/10.1371/journal.pone.0225353 January 27, 2020 1 / 15

a1111111111

a1111111111

a1111111111

a1111111111

a1111111111

OPEN ACCESS

Citation: Nunes A, Trappenberg T, Alda M, The

international Consortium on Lithium Genetics

(ConLiGen) (2020) Asymmetrical reliability of the

Alda score favours a dichotomous representation

of lithium responsiveness. PLoS ONE 15(1):

e0225353. https://doi.org/10.1371/journal.

pone.0225353

Editor: Vincenzo De Luca, University of Toronto,

CANADA

Received: October 31, 2019

Accepted: December 27, 2019

Published: January 27, 2020

Peer Review History: PLOS recognizes the

benefits of transparency in the peer review

process; therefore, we enable the publication of

all of the content of peer review and author

responses alongside final, published articles. The

editorial history of this article is available here:

https://doi.org/10.1371/journal.pone.0225353

Copyright: ©2020 Nunes et al. This is an open

access article distributed under the terms of the

Creative Commons Attribution License, which

permits unrestricted use, distribution, and

reproduction in any medium, provided the original

author and source are credited.

Data Availability Statement: All relevant data are

within the manuscript and its Supporting

Information files.

A common criticism that arises from this practice is that continuous variables should not

be discretized by virtue of “information loss.” Indeed, discretizing continuous variables is

widely viewed as an inappropriate practice [2–12]. However, the practice remains common

across many areas of research, including our group’s work on lithium responsiveness in BD

[13]. The primary justification for using the dichotomized Alda score as the lithium respon-

siveness definition has been based on the inter-rater reliability study by Manchia et al. [1], who

showed that a cut-off of 7 had strong inter-rater agreement (weighted kappa 0.66). Further-

more, using mixture modeling, they also found that the empirical distribution of Alda scores

supports the discretized definition. Therefore, there exist competing arguments regarding

the appropriateness of dichotomizing lithium response. Resolving this dispute is critical, since

the operational definition of lithium responsiveness is a concept upon which a large body of

research will depend.

Although the Manchia et al. [1] analysis provides some justification for using a dichoto-

mous lithium response definition, it does not dispel the argument of discretization-induced

information loss entirely. However, there is some intuitive reason to believe that discretization

is, at least pragmatically, the best approach to defining lithium response using the Alda score.

First, the Alda score remains inherently subjective to some degree and is not based on precise

biological measurements; an individual whose “true” Alda score is 6, for example, could have

observed scores that vary widely across raters. Second, it is possible that responders may be

more reliably identified than non-responders. For example, unambiguously “excellent” lithium

response is a phenomenon that undoubtedly exists in naturalistic settings [14,15], and for

which the space of possible Alda scores is substantially smaller than for non-responders; that

is, an Alda score of 8 can be obtained in far fewer ways than an Alda score of 5. As such, we

hypothesize that agreement on the Alda score is higher at the upper end of the score range,

and that this asymmetric agreement is a scenario in which dichotomization of the score is

more informative than the raw measure. To evaluate this, we present both empirical re-analy-

sis of the ConLiGen study by Manchia et al. [1] and analyses of simulated data with varying

levels of asymmetrical inter-rater reliability.

Materials and methods

Data

Detailed description of data and collection procedures is found in Manchia et al. [1]. Samples

included in our analysis are detailed in Table 1, including the number of raters included across

sites, and the average ratings obtained at each of those sites across the 12 assessment vignettes.

As a gold standard, we used ratings that were assigned to each case vignette using a consensus

process at the Halifax site (scores are noted in the first row of Table 1). The lithium responsive-

ness inter-rater reliability data are available in S1 File (total Alda score), and S2 File (Alda A-

score).

Empirical analysis of the Alda score

In this analysis, we seek to evaluate whether discretization of the Alda score under the existing

inter-rater reliability values preserves more mutual information (MI) between the observed

and ground truth labels than does the raw scale representation. To accomplish this, we first

develop a probabilistic formulation of raters’ score assignments based on a multinomial-

Dirichlet model, which we describe below. Since the Dirichlet distribution is the conjugate

prior for the multinomial distribution, the posterior distribution over ratings (and ultimately

the MI with respect to the “ground truth” Alda score) can be expressed as a closed-form func-

tion of the prior uncertainty, which increases the precision and efficiency of our experiments.

Asymmetrical reliability of the Alda score favours a dichotomous representation of lithium responsiveness

PLOS ONE | https://doi.org/10.1371/journal.pone.0225353 January 27, 2020 2 / 15

Funding: Genome Canada (MA, AN; https://www.

genomecanada.ca), Dalhousie Department of

Psychiatry Research Fund (MA, AN; https://

medicine.dal.ca/departments/department-sites/

psychiatry.html), Canadian Institutes of Health

Research #64410 (MA; http://www.cihr-irsc.gc.ca),

Natural Science and Engineering Research Council

of Canada (TT; https://www.nserc-crsng.gc.ca),

Nova Scotia Health Research Foundation Scotia

Scholars Graduate Scholarship (AN; https://nshrf.

ca), Killam Postgraduate Scholarship (AN; http://

www.killamlaureates.ca) and Dalhousie Medical

Research Foundation and the Lindsay family (AN

and MA). The funders had no role in study design,

data collection and analysis, decision to publish, or

preparation of the manuscript.

Competing interests: The authors have declared

that no competing interests exist.

Let nðkÞ

i2Nþdenote the number of raters who assigned an Alda score i2A, with

A¼ f0;1; :::; 10gto an individual whose gold standard Alda score is k2A. The vector of

rating counts for the gold standard score kis is nðkÞ¼ ðnðkÞ

iÞi2A. The probability of n

(k)

is multino-

mial with parameter vector θðkÞ¼ ðyðkÞ

iÞi2A, which is itself Dirichlet distributed θ

(k)

Dir(θ|α),

where αis a pseudocount denoting the prior expectation of the number of ratings received for

each score i2A. In the present analysis, we assume that αis equal across all scores in A, and

thus we denote it simply as a scalar α=α; this has the effect of increasing the uncertainty of θ

(k)

(i.e. the ratings become more “noisy”).

The posterior of θ

(k)

given n

(k)

and αis Dirichlet with parameters α0¼naþnðkÞ

i1o10

i¼0,

and its maximum a posteriori (MAP) estimate is

^

θanðkÞ

ð Þ ¼ aþnðkÞ

i1

P10

j¼0aþnðkÞ

j1

( )10

i¼0

;ð1Þ

which can be viewed as the conditional distribution over scores Afor any given rater when the

gold standard is k. In cases where no assessment vignette had a gold standard rating of k, we

assumed that

nðkÞ¼

1

2nðk1Þþnðkþ1Þ

ð Þ 0<k<10

nðkþ1Þk¼0

nðk1Þk¼10

8

>

>

>

<

>

>

>

:ð2Þ

Table 1. Number of raters and mean scores across sites. The total number of raters (n

r

) was 59.

Case Vignette

Site n

r

1 2 3 4 5 6 7 8 9 10 11 12

Gold standard 8 9 6 7 9 3 5 9 3 9 5 1

Halifax 9 8.4 8.6 6.6 6.9 9.2 3 3.9 8.8 3.1 9.1 4.7 1.2

NIMH 4 7.8 8.2 6.2 7 8.8 3.2 4 8.5 2.2 8.5 3.2 1.8

Poznan 2 9 8.5 6.5 5.5 9 4 7.5 9 5 8 4.5 4.5

Dresden 2 8.5 7.5 6 5 8.5 1.5 6 9 3.5 8.5 4 1.5

Japan 4 8 8.2 4.8 6.5 8.5 2 3 8.5 1 8.2 4.5 1.5

Wuerzburg 2 7.5 7.5 4 6.5 8 1.5 3 9 0 7 3 0.5

Cagliari 3 7.7 9 4.3 7 5.7 4 1.3 9 0.7 7.3 4 2

San Diego 2 7.5 8.5 7.5 7 9 5 7.5 8.5 3.5 8.5 6 3.5

Boston 2 8.5 8.5 6 7 9 3 3.5 8.5 1.5 9 4 1

Gottingen 2 9.5 9 4 6 9 1 1 9 1.5 9 4 3

Berlin 1 7 9 4 6 9 2 3 8 0 7 0 2

Taipeh 1 8 8 5 8 9 5 6 9 4 9 8 1

Prague 1 7 9 4 8 9 3 6 9 3 9 6 1

Johns Hopkins 7 8 8.7 5.3 5.9 8.3 2.7 2.4 9.1 2 8.3 4.4 1.1

Mayo 6 8 8.2 6 8 9 4.2 3 9 4.2 8.8 3.7 0.3

Brasil 3 8 8.3 5.3 6.3 8.7 2 4 9 4.3 8 4.7 0.7

Medellin 4 7.5 9 5.5 6.5 5 2.5 4 7.2 4.8 8.8 1.2 2

Geneve 3 7.7 8.7 6.7 5.3 9.7 5 6 8.7 1.3 9 3.7 0.3

https://doi.org/10.1371/journal.pone.0225353.t001

Asymmetrical reliability of the Alda score favours a dichotomous representation of lithium responsiveness

PLOS ONE | https://doi.org/10.1371/journal.pone.0225353 January 27, 2020 3 / 15

The dichotomized Alda scores are defined as T¼ fd½it:8i2Ag, where τis the

dichotomization threshold (set at τ= 7 for the Alda score), and where δ[] is an indicator func-

tion that evaluates to 1 if the argument is true, and 0 otherwise. Given threshold τ(Responders

τand Non-responders <τ), the dichotomous counts are represented as follows

cð0Þ

0¼Pt1

k¼0Pt1

i¼0nðkÞ

iObserved <t;Gold Standard <t

cð1Þ

0¼P10

k¼tPt1

i¼0nðkÞ

iObserved <t;Gold Standard t

cð0Þ

1¼Pt1

k¼0P10

i¼tnðkÞ

iObserved t;Gold Standard <t

cð1Þ

1¼P10

k¼tP10

i¼tnðkÞ

iObserved t;Gold Standard t

ð3Þ

with c

(k)

Multinomial(ϕ

k

), and ϕ

k

Dir(ϕ|ξ), where ξis a pseudocount for the number of

dichotomized ratings assigned to each of non-responders and responders. We can thus esti-

mate the conditional distribution over observed dichotomized response ratings as

^

�xcðkÞ

ð Þ ¼ (xþcðkÞ

01

2x2þcðkÞ

0þcðkÞ

1

;xþcðkÞ

11

2x2þcðkÞ

0þcðkÞ

1g ð4Þ

Mutual information of raw and dichotomized Alda score representations. Mutual infor-

mation is a general measure of dependence that expresses the degree to which uncertainty

about one variable is reduced by observation of another. Whereas the correlation coefficient

depends on the existence of a linear association, MI can detect nonlinear relationships

between variables by comparing their joint probability against the product of their marginal

distributions.

Let

xopðxojxÞ ¼ Categorical^

θaðnðxÞÞð5Þ

denote a given observed raw Alda score assigned to a case with ground truth score of x2A.

Given uniform priors on the true classes, the joint distribution is

pðxo;xÞ ¼ pðxojxÞpðxÞ ¼ (1

11 ^

θanðx¼kÞ

)k¼0;1;:::;10

:ð6Þ

For the binarized classes, we have a prior of pðy¼1Þ ¼ 4

11, and the joint distribution is

thus

pðyo;yÞ ¼ pðyojyÞpðyÞ ¼ npðy¼kÞ^

�xðcðy¼kÞÞok2 f0;1g:ð7Þ

The MI for these distributions can be computed as functions of the prior pseudocounts α

and ξ:

Ia½xojjx ¼ X

xoX

x

pðxo;xÞlog pðxo;xÞ

pðxoÞpðxÞð8Þ

Ix½yojjy ¼ X

yoX

y

pðyo;yÞlog pðyo;yÞ

pðyoÞpðyÞð9Þ

Asymmetrical reliability of the Alda score favours a dichotomous representation of lithium responsiveness

PLOS ONE | https://doi.org/10.1371/journal.pone.0225353 January 27, 2020 4 / 15

for the raw and dichotomized Alda scores, respectively. We can express the MI of the raw and

dichotomized Alda score distributions both in terms of α, such that both distributions have

an equivalent total “concentration:” ξ=αwhen ξ= 11α/2. This is equivalent to saying that our

prior assumption about the uncertainty of the raw and dichotomized distributions assumes

the same number of a priori ratings.

Our primary hypothesis—that the dichotomized Alda score is more informative with greater

observation uncertainty—is evaluated by determining whether I

ξ

[y

o

||y

] exceeds I

α

[x

o

||x

] as we

increase the a priori observation noise (αand ξ).

Theoretical modeling of dichotomization under asymmetrical reliability

The previous experiment regarding dichotomization of the raw Alda score did not fully cap-

ture the effect of dichotomization of a continuous variable, since the raw Alda score is still

discrete (albeit with a larger domain of support). Thus, we sought to investigate whether

dichotomization of a truly continuous, though asymmetrically reliable, variable would show a

similar pattern of preserving MI and statistical power under higher levels of observation noise

and agreement asymmetry.

Synthetic datasets. The simplest synthetic dataset generated was merely a sample of regu-

larly spaced points across the [0, 10] interval in both the x and y directions. This dataset was

merely used to conduct a “sanity check” that our methods for computing MI correctly identi-

fied a value of 0. This was necessary since data with uniform random noise over the same inter-

val will only yield MI of 0 in the limit of large sample sizes.

The main synthetic dataset accepted “ground truth” values x2[0, 10] and yielded “observed”

values y2[0, 10] based on the following formula for the i

th

sample:

yi¼ofðxiÞ þ ð1oÞUniformð0;10Þ;ð10Þ

where 0 ω1 is a parameter governing the degree to which observed values are coupled

to the ground truth based on f(x

i

) (data are entirely uniform random noise when ω= 0, and

come entirely from f(x

i

) when ω= 1). The function f(x

i

) governing the agreement between

ground truth and observed is essentially a 1:1 correspondence between xand yto which we add

noise along the diagonal based on a uniform random variate ~

Uð s;sÞwith width σ.

We simulated two forms of diagonal spread. The first is constant across all values x2[0,

10], which we call the symmetrical case, and which is represented by a parameter β= 1. The

other is an asymmetrical case (represented as β= 0), in which the agreement between xand y

is not constant across the [0, 10] range. Overall, the function f(x

i

) is defined as

fðxiÞ ¼ bRð0;10Þxiþ~

Uð s;sÞ

1þe0:75 xiþ5

þ ð1bÞRð0;10Þxiþ~

Uð s;sÞ

;ð11Þ

where R

(l,u)

() is a function to ensure that all points remain within the [l,u] interval in both

axes. In the asymmetrical case, R

(l,u)

() reflects points at the [0, 10] bounds. In the symmetrical

case, the data are all simply rescaled to lie in the [0, 10] interval.

Demonstration of the simulated synthetic data are shown in Fig 1. Every synthetically gen-

erated dataset included 750 samples, and for notational simplicity, we denote the k

th

synthetic

dataset (given parameters β,ω,σ) as DðkÞ

b;o;s¼xðkÞ

j;yðkÞ

jj¼1;2;:::;750.

Computation of mutual information for continuous and discrete distributions.

Mutual information was computed for both continuous and dichotomized probability distri-

butions on the data. Mutual information for the continuous distribution was computed by

first performing Gaussian kernel density estimation (using Scott’s method for bandwidth

PLOS ONE | https://doi.org/10.1371/journal.pone.0225353 January 27, 2020 5 / 15

selection) on the simulated dataset, and then approximating the following integral using Mar-

kov chain Monte-Carlo sampling:

IKDE½yjjx ¼ ZZpðx;yÞlog pðx;yÞ

pðxÞpðyÞdxdyð12Þ

Conversely, discrete MI was computed by first creating a 2-dimensional histogram by bin-

ning data based on a dichotomization threshold τ. Data that lie below the dichotomization

threshold are denoted 1, and those that lie above the threshold are represented as 0. Based on

Fig 1. Demonstration of the synthetic agreement data across differences in the parameter ranges and presence of

asymmetry. The x-axes all represent the ground truth value of the variable, and the y-axes represent the “observed”

values. Data are depicted based on different values of a uniform noise parameter (0 ω1) that governs what

proportion of the data is merely uniform noise over the interval [0, 10], and a disagreement parameter (σ0), which

governs the variance around the diagonal line. Panel A (upper three rows, shown in blue) depicts the synthetic data in

which there was asymmetrical levels of agreement across the score domain. Panel B (lower three rows, shown in red)

depict synthetic data in which there was symmetrical agreement over the score domain.

https://doi.org/10.1371/journal.pone.0225353.g001

PLOS ONE | https://doi.org/10.1371/journal.pone.0225353 January 27, 2020 6 / 15

this joint distribution, the dichotomized MI is

It½yjjx ¼ X

xX

y

pðx;yÞlog pðx;yÞ

pðxÞpðyÞ:ð13Þ

Note that continuous MI will remain constant across τ.

Statistical power of classical tests of association. Association between the observed (y)

and ground truth (x) data can be measured using Pearson’s correlation coefficient (ρ) when

data are left as continuous, or using Fisher’s exact test when data are dichotomized. The statis-

tical power of the hypothesis that ρ6¼ 0 given dataset DðkÞ

b;o;swith N

(k)

observations and two-

tailed statistical significance threshold α—which here is not the same αused as a Dirichlet con-

centration in Empirical Evaluation of the Alda Score of Lithium Response—can be easily shown

to equal

powerrðDðkÞ

b;o;s;a¼0:05Þ ¼ Fjz rð Þj F11a

2

;ð14Þ

where F() and F

−1

() are the cumulative distribution function and quantile functions for a

standard normal distribution, and z() is Fisher’s Z-transformation

zðrÞ ¼ 1

2log 1þr

1r:ð15Þ

Under a dichotomization of DðkÞ

b;o;swith threshold τassociation between the ground truth

and observed data can be evaluated using a (two-tailed) Fisher’s exact test, whose alternative

hypothesis is that the odds ratio (η) of the dichotomized data does not equal 1. The null-

hypothesis has a Fisher’s noncentral hypergeometric distribution,

Lo¼FisherHypergeometricDistributionNðkÞ

d½y<t;NðkÞ

d½x<t;NðkÞ;Z¼1ð16Þ

where N

(k)

is the total number of observations in sample k, and NðkÞ

d½x<tand NðkÞ

d½y<tare the num-

ber of ground truth and observed data, respectively, that fall below the dichotomization thresh-

old τ. Under the alternative hypothesis, this distribution has an odds ratio parameter estimated

from the data:

La¼FisherHypergeometricDistributionNðkÞ

d½y<t;NðkÞ

d½x<t;NðkÞ;^

Z:ð17Þ

The statistical power of Fisher’s exact test under this setup and a two-tailed significance

threshold of αis

fp DðkÞ

b;o;s;t;a

¼d½^

Z1SLaS1

Lo1a

2

þd½^

Z<11SLaS1

Lo

a

2

;ð18Þ

where SLaðÞ and S1

LoðÞ are the survival functions of the alternative hypothesis and the inverse

survival function of the null hypothesis, respectively.

Evaluation of mutual information. The central aspect of this analysis is comparison

of the dichotomized and continuous MI across values of the dichotomization threshold τ,

global noise ω, asymmetry parameter β, and diagonal spread σ. Under all cases, we expect that

increases in the global noise parameter ωwill reduce the MI. We also expect that with symmet-

rical reliability (i.e. β= 0), the dichotomized MI will be lower than the continuous MI across

all thresholds. However, as the degree of asymmetry in the reliability increases, we expect the

dichotomized MI to exceed the continuous MI (i.e. as σincreases when β= 1). Finally, as a

PLOS ONE | https://doi.org/10.1371/journal.pone.0225353 January 27, 2020 7 / 15

sanity check, we expect that both continuous and dichotomized MI will be approximately 0

when applied to a grid of points regularly spaced over the [0, 10] interval.

Evaluation of effects on statistical power of classical tests of association. Statistical

power of the Pearson correlation coefficient and Fisher’s exact test were computed across sym-

metrical (β= 0) and asymmetrical (β= 1) conditions of the synthetic dataset described above.

Owing to the greater computational efficiency of these calculations (compared to the MI), the

diagonal spread parameter was varied more densely (σ2{1, 2, . . ., 20}). The power of Fisher’s

exact test was evaluated at two dichotomization thresholds: a median split at τ= 5 and a “tail

split” at τ= 3. We evaluated three global noise settings (ω2{0.3, 0.5, 0.7}). At each experimen-

tal setting, we computed the aforementioned power levels for 100 independent synthetic data-

sets. Results are presented using the mean and 95% confidence intervals of the power estimates

over the 100 runs under each condition. We expect that the Fisher’s exact test under a “tail

split” dichotomization (not a median split) will yield greater statistical power in the presence

of asymmetrical reliability, greater diagonal spread, and higher global noise. However, under

the symmetrically reliable condition, we expect that the statistical power will be greater for the

continuous test of association.

Materials

Mutual information experiments were conducted in Mathematica v. 12.0.0 (Wolfram

Research, Inc.; Champaign, IL). Experiments evaluating the statistical power under classical

tests of continuous and dichotomous association were conducted in the Python programming

language. Code for analyses are also provided in S3 File (Alda score analyses), S4 File (theoreti-

cal analysis of MI under asymmetrical reliability), S5 File (theoretical evaluation of classical

associative tests). The Mathematica notebooks are also available in PDF form in S6 and S7

Files.

Results

Empirical evaluation of the Alda score of lithium response

Histograms of the observed Alda scores for each of the gold standard vignette values are

depicted in Fig 2. Resulting joint distributions of the gold standard vs. observed Alda scores

(in both the raw or dichotomized representations) are shown in Fig 3 (Panels A-C) across

varying levels of observation noise. Fig 3D plots the MI for the raw and dichotomized Alda

scores across increasing levels of the observation noise parameter α(recalling that ξ= 11α/2).

Beyond an observation noise of approximately α>3.52, one can see that the dichotomized

lithium response definition retains greater MI between the true and observed labels, compared

to the raw representation.

Discrete vs. continuous mutual information in asymmetrically reliable data

Fig 4 shows the results of the experiment on synthetic data. Under agreement levels that are

constant across the (x,y) domains, one can observe that MI of dichotomized representations

of the variables are generally lower than their continuous counterparts. However, under asym-

metrical reliability (i.e. where agreement between xand ydecreases as xincreases), we see that

MI is higher for the dichotomized, rather than the continuous, representations. In particular,

as the level of agreement asymmetry increased (i.e. for higher values of σ), the best dichotomi-

zation thresholds decreased.

PLOS ONE | https://doi.org/10.1371/journal.pone.0225353 January 27, 2020 8 / 15

Fig 2. Histograms of ratings for each value of the ground truth Alda score available in the first wave dataset from Manchia

et al. [1]. Each histogram represents the distribution of ratings (n

r

= 59) for a single one of twelve assessment vignettes. The gold

standard (“ground truth”) Alda score, obtained by the Halifax consensus sample, is depicted as the title for each histogram. Plots in

blue are those for vignettes with gold standard Alda scores less than 7, which would be classified as “non-responders” under the

dichotomized setting. Vignettes with gold standard Alda scores 7 are shown in red, and represent the dichotomized group of

lithium responders.

https://doi.org/10.1371/journal.pone.0225353.g002

Fig 3. Mutual information between gold standard and observed Alda scores in relation to the observation noise (α) and

whether the scale is in its raw or dichotomized form (lithium responder [Li(+)] is Alda score 7; non-responder [Li(-)] is Alda

score <7). Panels A-C show the inferred joint distributions of the observed (x

o

for raw, y

o

for discrete) and gold standard (x

for

raw, y

for discrete) values at different levels of observation noise (α2{0, 10, 100}). Panel D plots the mutual information for the raw

(red) and discrete (blue) settings of the Alda score across increasing values of α. Recall that here we set ξ= 11/2.

https://doi.org/10.1371/journal.pone.0225353.g003

PLOS ONE | https://doi.org/10.1371/journal.pone.0225353 January 27, 2020 9 / 15

Statistical power of classical associative tests

Fig 5 plots the statistical power of null-hypothesis tests using continuous and dichotomized

representations of the synthetic dataset. As expected, under conditions of symmetrical reliabil-

ity, the continuous test of association (Pearson correlation) retains greater statistical power as

the degree of diagonal spread increases, although this difference lessens at very high levels of

diagonal spread or overall (uniform) noise. However, under conditions of asymmetrical reli-

ability, dichotomizing data according to a “tail split” (here a threshold of τ= 3) preserves

greater statistical power than either a median split (τ= 5) or continuous representation; this

relationship was present even at high levels of diagonal spread and overall uniform noise.

Discussion

The present study makes two important contributions. First, using a sample of 59 ratings

obtained using standardized vignettes compared to a consensus-defined gold standard [1],

we showed that the dichotomized Alda score has a higher MI between the observed and gold

standard ratings than does the raw scale (which ranges from 0-10). Those data suggested that

the Alda score’s reliability is asymmetrical, with greater inter-rater agreement at the upper

extreme. Secondly, therefore, using synthetic experiments we showed that asymmetrical inter-

rater reliability in a score’s range is the likely cause of this relationship. Our results do not

argue that lithium response is itself a categorical natural phenomenon. Rather, using the

dichotomous definition as a target variable in supervised learning problems likely confers

greater robustness to noise in the observed ratings.

Some have argued that the existence of categorical structure in one’s data [9], or evidence of

improved reliability under a dichotomized structure [16], are potentially justifiable rationales

Fig 4. Mutual information (MI) for dichotomized (solid lines) and continuous (dashed lines) distributions on synthetic data with

asymmetrical (upper row, Panel A) and symmetrical (lower row, Panel B) properties with respect to agreement. X-axes represent the

dichotomization thresholds at which we recalculate the dichotomized MI. Mutual information is depicted on the y-axes. Plot titles indicate the

different diagonal spread (σ) parameters used to synthesize the synthetic datasets. Solid lines (for dichotomized MI) are surrounded by ribbons

depicting the 95% confidence intervals over 10 runs at each combination of parameters (τ,ω,β,σ).

https://doi.org/10.1371/journal.pone.0225353.g004

PLOS ONE | https://doi.org/10.1371/journal.pone.0225353 January 27, 2020 10 / 15

for dichotomization of continuous variables. These claims are generally stated only briefly, and

with less quantitative support than the more numerous mathematical treatments of the prob-

lems with dichotomization [9,10,16,17]. However, these more rigorous quantitative analyses

typically involve assumptions of symmetrical or Gaussian distributions of the underlying vari-

ables in the context of generalized linear modeling (although Irwin & McClelland [10] demon-

strated that median splits of asymmetric and bimodal beta distributions is also deleterious).

These analyses have led to vigorous generalized denunciation of variable dichotomization

across several disciplines, but our current work offers important counterexamples to this nar-

rative [10,11].

The Alda score is more broadly used as a target variable in both predictive and associative

analyses, and not as a predictor variable, which is an important departure from most analyses

against dichotomization. Since there is no valid and reliable biomarker of lithium response,

these cases must rely on the Alda score-based definition of lithium response as a “ground

truth” target variable. In the case of predicting lithium response, where these ground truth

labels are collected from multiple raters across different international sites, variation in lithium

response scoring patterns across centres might further accentuate the extant between-site

heterogeneity.

To this end, inter-individual differences in subjective rating scales may be more informative

about the raters than the subjects, and one may wish to use dichotomization to discard this

nuisance variance [8,9,16]. Doing so means that one turns regression supervised by a dubious

target into classification with a more reliable (although coarser) target. Appropriately balanc-

ing these considerations may require more thought than adopting a blanket prohibition on

dichotomization or some other form of preprocessing.

An important criticism of continuous variable dichotomization is that it may impede compa-

rability of results across studies, both in terms of diminishing power and inflating heterogeneity

Fig 5. Statistical power achieved with the Pearson coefficient (a continuous measure of association; blue lines) and Fisher’s

exact test (a measure of association between dichotomized variables; red lines) for synthetic data with symmetrical (upper row)

and asymmetrical (lower row) properties with respect to agreement. Columns correspond to the level of uniform “overall” noise

(ω) added to the data, representing prior uncertainty. X-axes represent the diagonal spread (σ), and the y-axes represent the test’s

statistical power for the given sample size and estimated effect sizes. Data subjected to Fisher’s exact test were dichotomized at either

a threshold of 5 (the “Median Split,” denoted by ‘+’ markers in red) or 3 (the “Tail Split,” denoted by the dot markers in red). For all

series, dark lines denote means and the ribbons are 95% confidence intervals over 100 runs.

https://doi.org/10.1371/journal.pone.0225353.g005

PLOS ONE | https://doi.org/10.1371/journal.pone.0225353 January 27, 2020 11 / 15

[17]. However, this is more likely a problem when dichotomization thresholds are established

on a study-by-study basis, without considering generalizability from the outset. These argu-

ments do not necessarily apply to the Alda score, since the threshold of 7 has been established

across a large consortium with support from both reliability and discrete mixture analysis [1],

and is the effective standard split point for this scale [18].

Our study thus provides a unique point of support for the dichotomized Alda score insofar

as we show that the retention of MI and frequentist statistical power is likely due to asymmetri-

cal reliability across the range of scores. Our analyses show that there is a range of Alda scores

(those identifying good lithium responders; scores 7) for which scores correspond more

tightly to a consensus-defined gold standard in a large scale international consortium. Con-

versely, this asymmetry implies that Alda scores at the lower end of the range will carry greater

uncertainty (Fig 2). This may be due to the intrinsic structure of the Alda scale, wherein a

score of 3 may result from 2159 item combinations, while only 79 combinations can yield a

score of 7. In particular, we showed that tail split dichotomization of the Alda score will be

more robust to increases in the prior uncertainty (i.e. the overall level of background “noise”

in the relationship between true/observed scores). This feature is important since the sample

of raters included in the Alda score’s calibration study [1] was relatively small and consisted of

individuals involved in ConLiGen centres. It is reasonable to suspect that assessment of Alda

score reliability in broader research and clinical settings would add further disagreement-

based noise to the inter-rater reliability data. At present, use of the dichotomized scale could

confer some robustness to that uncertainty.

More generally our study showed that if reliability of a measure is particularly high at one

tail of its range, then a “tail split” dichotomization can outperform even the continuous repre-

sentation of the variable. This presents an important counterexample to previous authors,

such as Cohen [5], Irwin & McClelland [10], and MacCallum et al. [9] who argued that “tail

splits” are still worse than median splits. While our study reaffirms these claims in the case of

measures whose reliability is constant over the domain (see Fig 4B and the upper row of Fig 5),

our analysis of the asymmetrically reliable scenario yields opposite conclusions, favouring a

“tail split” dichotomization over both median splits and continuous representations. Tail split

dichotomization was particularly robust when data were affected by both asymmetrical reli-

ability and high degrees of uniform noise over the variable’s range. Together, these results

suggest that dichotomization/categorization of a continuous measurement may be justifiable

when its relationship to the underlying ground truth variable is noisy everywhere except at an

extreme.

Our study has several limitations. First, our sample size for the re-analysis of the Alda score

reliability was relatively small, and sourced from highly specialized raters involved in lithium-

specific research. However, one may consider this sample as representative of the “best case

scenario” for the Alda score’s reliability. It is likely that further expansion of the subject popu-

lation would introduce more noise into the relationship between ground truth and observed

Alda scores. It is likely that most of this additional disagreement would be observed for lower

Alda scores, since (A) there are simply more potential item combinations that can yield an

Alda score of 5 than an Alda score of 9, for example, and (B) unambiguously excellent lithium

response is a phenomenon so distinct that some question whether lithium responsive BD may

constitute a unique diagnostic entity [19,20]. Thus, we believe that our sample size for the reli-

ability analysis is likely sufficient to yield the present study’s conclusions.

Our study is also limited by the fact that theoretical analysis was largely simulation-based,

and thus cannot offer the degree of generalizability obtained through rigorous mathematical

proof. Nonetheless, our study offers sufficient evidence—in the form of a counterexample—to

show that there exist scenarios in which dichotomization is statistically superior to preserving

PLOS ONE | https://doi.org/10.1371/journal.pone.0225353 January 27, 2020 12 / 15

a variable’s continuous representation. Furthermore, we used well controlled experiments to

isolate asymmetrical reliability as the cause of dichotomization’s superiority across simulated

conditions.

Conclusion

In conclusion, we have shown that a dichotomous representation of the Alda score for lithium

responsiveness is more robust to noise arising from inter-rater disagreement. The dichoto-

mous Alda score is therefore likely a better representation of lithium responsiveness for multi-

site studies in which lithium response is a target or dependent variable. Through both re-analy-

sis of the Alda score’s real-world inter-rater reliability data and careful theoretical simulations,

we were able to show that asymmetrical reliability across the score’s domain was the likely

cause for superiority of the dichotomous definition. Our study is not only important for future

research on lithium response, but other studies using subjective and potentially unreliable

measures as dependent variables. Practically speaking, our results suggest that it might be bet-

ter to classify something we can all agree upon than to regress something upon which we can

not.

Supporting information

S1 Fig. A-score reliability histograms. Histograms of ratings for each value of the ground

truth Alda A-score. This figure was generated identically to Fig 2, but using the A-score data

only.

(PDF)

S2 Fig. A-score mutual information results. Mutual information between gold standard and

observed Alda A-scores in relation to observation noise and the scale’s “raw” or dichotomized

form. This figure was generated identically to Fig 3, but using the A-score data only.

(PDF)

S1 File. Total Alda score ratings. Inter-rater reliability data for the total Alda score.

(CSV)

S2 File. Alda A-score ratings. Inter-rater reliability data for the Alda A-score.

(CSV)

S3 File. Alda score analysis code. Mathematica notebook containing the empirical evaluation

of the Alda Score of Lithium response. This notebook also contains additional analysis of the

A-score alone.

(NB)

S4 File. Theoretical analysis code. Mathematica notebook containing the theoretical analyses

of discrete vs. continuous mutual information in asymmetrically reliable data.

(NB)

S5 File. Code for statistical power tests. Jupyter notebook containing the theoretical analyses

of the statistical power of classical associative tests under asymmetrically reliable data.

(IPYNB)

S6 File. Alda score analysis code (PDF version). PDF version of S3 File for those without

Mathematica license.

(PDF)

PLOS ONE | https://doi.org/10.1371/journal.pone.0225353 January 27, 2020 13 / 15

S7 File. Theoretical analysis code (PDF version). PDF version of S4 File for those without

Mathematica license.

(PDF)

Acknowledgments

The authors wish to acknowledge those members of the Consortium on Lithium Genetics

(ConLiGen) who contributed ratings for the vignettes herein: Mirko Manchia, Raffaella

Ardau, Jean-Michel Aubry, Lena Backlund, Claudio E.M. Banzato, Bernhard T. Baune, Frank

Bellivier, Susanne Bengesser, Clara Brichant-Petitjean, Elise Bui, Cynthia V. Calkin, Andrew

Tai Ann Cheng, Caterina Chillotti, Scott Clark, Piotr M. Czerski, Clarissa Dantas, Maria Del

Zompo, J. Raymond DePaulo, Bruno Etain, Peter Falkai, Louise Frise

´n, Mark A. Frye, Jan

Fullerton, Se

´bastien Gard, Julie Garnham, Fernando S. Goes, Paul Grof, Oliver Gruber, Ryota

Hashimoto, Joanna Hauser, Rebecca Hoban, Ste

´phane Jamain, Jean-Pierre Kahn, Layla Kas-

sem, Tadafumi Kato, John R. Kelsoe, Sarah Kittel-Schneider, Sebastian Kliwicki, Po-Hsiu Kuo,

Ichiro Kusumi, Gonzalo Laje, Catharina Lavebratt, Marion Leboyer, Susan G. Leckband, Car-

los A. Lo

´pez Jaramillo, Mario Maj, Alain Malafosse, Lina Martinsson, Takuya Masui, Philip B.

Mitchell, Frank Mondimore, Palmiero Monteleone, Audrey Nallet, Maria Neuner, Toma

´s

Nova

´k, Claire O’Donovan, Urban O

¨sby, Norio Ozaki, Roy H. Perlis, Andrea Pfennig, James B.

Potash, Daniela Reich-Erkelenz, Andreas Reif, Eva Reininghaus, Sara Richardson, Janusz K.

Rybakowski31, Martin Schalling, Peter R. Schofield, Oliver K. Schubert, Barbara Schweizer,

Florian Seemu¨ller, Maria Grigoroiu-Serbanescu, Giovanni Severino, Lisa R. Seymour, Claire

Slaney, Jordan W. Smoller, Alessio Squassina, Thomas Stamm, Pavla Stopkova, Sarah K.

Tighe, Alfonso Tortorella, Adam Wright, David Zilles, Michael Bauer, Marcella Rietschel, and

Thomas G. Schulze.

Author Contributions

Conceptualization: Abraham Nunes.

Data curation: Martin Alda.

Formal analysis: Abraham Nunes.

Investigation: Abraham Nunes.

Methodology: Abraham Nunes, Martin Alda.

Resources: Martin Alda.

Software: Abraham Nunes.

Supervision: Thomas Trappenberg, Martin Alda.

Validation: Abraham Nunes.

Visualization: Abraham Nunes.

Writing – original draft: Abraham Nunes.

Writing – review & editing: Abraham Nunes, Thomas Trappenberg, Martin Alda.

References

1. Manchia M, Adli M, Akula N, Ardau R, Aubry JM, Backlund L, et al. Assessment of Response to Lithium

Maintenance Treatment in Bipolar Disorder: A Consortium on Lithium Genetics (ConLiGen) Report.

PLoS ONE. 2013; 8. https://doi.org/10.1371/journal.pone.0065636

PLOS ONE | https://doi.org/10.1371/journal.pone.0225353 January 27, 2020 14 / 15

2. Humphreys LG, Fleishman A. Pseudo-orthogonal and other analysis of variance designs involving indi-

vidual-differences variables. Journal of Educational Psychology. 1974; 66: 464–472. https://doi.org/10.

1037/h0036539

3. Humphreys LG. Doing research the hard way: Substituting analysis of variance for a problem in correla-

tional analysis. Journal of Educational Psychology. 1978; 70: 873–876. https://doi.org/10.1037/0022-

0663.70.6.873

4. Humphreys LG. Research on individual differences requires correlational analysis, not ANOVA. Intelli-

gence. 1978; 2: 1–5. https://doi.org/10.1016/0160-2896(78)90010-7

5. Cohen J. The Cost of Dichotomization. Applied Psychological Measurement. 1983; 7: 249–253. https://

doi.org/10.1177/014662168300700301

6. Royston P, Altman DG, Sauerbrei W. Dichotomizing continuous predictors in multiple regression: a bad

idea. Statistics in Medicine. 2006; 25: 127–141. https://doi.org/10.1002/sim.2331 PMID: 16217841

7. Altman DG, Royston P. The cost of dichotomising continuous variables. BMJ. 2006; 332: 1080. https://

doi.org/10.1136/bmj.332.7549.1080

8. Rucker DD, McShane BB, Preacher KJ. A researcher’s guide to regression, discretization, and median

splits of continuous variables. Journal of Consumer Psychology. 2015; 25: 666–678. https://doi.org/10.

1016/j.jcps.2015.04.004

9. MacCallum RC, Zhang S, Preacher KJ, Rucker DD. On the practice of dichotomization of quantitative

variables. Psychological Methods. 2002; 7: 19–40. https://doi.org/10.1037/1082-989x.7.1.19 PMID:

11928888

10. Irwin JR, McClelland GH. Negative Consequences of Dichotomizing Continuous Predictor Variables.

Journal of Marketing Research. 2003; 40: 366–371. https://doi.org/10.1509/jmkr.40.3.366.19237

11. Fitzsimons GJ. Death to Dichotomizing. J Consum Res. 2008; 35: 5–8. https://doi.org/10.1086/589561

12. Streiner DL. Breaking up is Hard to Do: The Heartbreak of Dichotomizing Continuous Data. Can J Psy-

chiatry. 2002; 47: 262–266. https://doi.org/10.1177/070674370204700307 PMID: 11987478

13. Nunes A, Ardau R, Bergho

¨fer A, Bocchetta A, Chillotti, Deiana V, et al. Prediction of Lithium Response

using Clinical Data. Acta Psychiatr. Scandinav. in press.

14. Grof P. Responders to long-term lithium treatment. In: Bauer M, Grof P, Muller-Oerlinghausen B, edi-

tors. Lithium in Neuropsychiatry: The Comprehensive Guide. UK: Informa Healthcare; 2006. pp. 157–

178.

15. Gershon S, Chengappa KNR, Malhi GS. Lithium specificity in bipolar illness: A classic agent for the clas-

sic disorder. Bipolar Disorders. 2009; 11: 34–44. https://doi.org/10.1111/j.1399-5618.2009.00709.x

PMID: 19538684

16. DeCoster J, Iselin A-MR, Gallucci M. A conceptual and empirical examination of justifications for dichot-

omization. Psychological Methods. 2009; 14: 349–366. https://doi.org/10.1037/a0016956 PMID:

19968397

17. Hunter JE, Schmidt FL. Dichotomization of continuous variables: The implications for meta-analysis.

Journal of Applied Psychology. 1990; 75: 334–349. https://doi.org/10.1037/0021-9010.75.3.334

18. Hou L, Heilbronner U, Degenhardt F, Adli M, Akiyama K, Akula N, et al. Genetic variants associated

with response to lithium treatment in bipolar disorder: A genome-wide association study. The Lancet.

2016; 387: 1085–1093. https://doi.org/10.1016/S0140-6736(16)00143-4

19. Alda M. Lithium in the treatment of bipolar disorder: pharmacology and pharmacogenetics. Molecular

Psychiatry. 2015; 20: 661. https://doi.org/10.1038/mp.2015.4 PMID: 25687772

20. Alda M. Who are excellent lithium responders and why do they matter? World Psychiatry. 2017; 16:

319–320. https://doi.org/10.1002/wps.20462 PMID: 28941103

PLOS ONE | https://doi.org/10.1371/journal.pone.0225353 January 27, 2020 15 / 15