Available via license: CC BY-NC 4.0
Content may be subject to copyright.
ORIGINAL RESEARCH
Analysis of lung cancer risk model (PLCOM2012 and
LLPv2) performance in a community- based lung cancer
screeningprogramme
Mikey B Lebrett ,1,2 Haval Balata,1,3 Matthew Evison,3 Denis Colligan,4,5
Rebecca Duerden,3,6 Peter Elton,7 Melanie Greaves,3,6 John Howells,8 Klaus Irion,6
Devinda Karunaratne,6 Judith Lyons,3 Stuart Mellor,9 Amanda Myerscough,4
Tom Newton,9 Anna Sharman,3,6 Elaine Smith,3,6 Ben Taylor,10 Sarah Taylor,4,11
Anna Walsham,12 James Whittaker,13 Phil V Barber,3 Janet Tonge ,14
Hilary A Robbins,15 Richard Booton,1,3,16 Philip A J Crosbie 1,2,3
Lung cancer
To cite: LebrettMB,
BalataH, EvisonM, etal.
Thorax Epub ahead of print:
[please include Day Month
Year]. doi:10.1136/
thoraxjnl-2020-214626
For numbered affiliations see
end of article.
Correspondence to
Dr Philip A J Crosbie, Division
of Infection, Immunity and
Respiratory Medicine, The
University of Manchester Faculty
of Biology Medicine and Health,
Manchester M13 9PL, UK;
philip. crosbie@ manchester.
ac. uk
Received 5 February 2020
Revised 8 April 2020
Accepted 20 April 2020
►http:// dx. doi. org/ 10. 1136/
thoraxjnl- 2020- 215156
© Author(s) (or their
employer(s)) 2020. Re- use
permitted under CC BY- NC. No
commercial re- use. See rights
and permissions. Published
by BMJ.
ABSTRACT
Introduction Low- dose CT (LDCT) screening of high-
risk smokers reduces lung cancer (LC) specific mortality.
Determining screening eligibility using individualised risk
may improve screening effectiveness and reduce harm.
Here, we compare the performance of two risk prediction
models (PLCOM2012 and Liverpool Lung Project model
(LLPv2)) and National Lung Screening Trial (NLST) eligibility
criteria in a community- based screening programme.
Methods Ever- smokers aged 55–74, from deprived areas
of Manchester, were invited to a Lung Health Check (LHC).
Individuals at higher risk (PLCOM2012 score ≥1.51%) were
offered annual LDCT screening over two rounds. LLPv2
score was calculated but not used for screening selection;
≥2.5% and ≥5% thresholds were used for analysis.
Results PLCOM2012 ≥1.51% selected 56% (n=1429) of
LHC attendees for screening. LLPv2 ≥2.5% also selected
56% (n=1430) whereas NLST (47%, n=1188) and
LLPv2 ≥5% (33%, n=826) selected fewer. Over two
screening rounds 62 individuals were diagnosed with LC;
representing 87% (n=62/71) of 6- year incidence predicted
by mean PLCOM2012 score (5.0%). 26% (n=16/62) of
individuals with LC were not eligible for screening using
LLPv2 ≥5%, 18% (n=11/62) with NLST criteria and 7%
(n=5/62) with LLPv2 ≥2.5%. NLST eligible Manchester
attendees had 2.5 times the LC detection rate than NLST
participants after two annual screens (≈4.3% (n=51/1188)
vs 1.7% (n=438/26 309); p<0.0001). Adverse measures of
health, including airflow obstruction, respiratory symptoms
and cardiovascular disease, were positively correlated with
LC risk. Coronary artery calcification was predictive of LC
(adjOR 2.50, 95% CI 1.11 to 5.64; p=0.028).
Conclusion Prospective comparisons of risk prediction
tools are required to optimise screening selection in
different settings. The PLCOM2012 model may underestimate
risk in deprived UK populations; further research focused
on model calibration is required.
INTRODUCTION
Lung cancer causes 1.6 million deaths each year,
representing 20% of total cancer mortality world-
wide.1 Survival is poor in both developed and low/
middle- income countries.2 3 Poor survival is driven
by late clinical presentation which is primarily a
consequence of early- stage lung cancer having no or
only mild non- specific symptoms.4 In the UK, half
of all patients have stage IV disease at diagnosis and
one- third present as an emergency, at which point
outcomes are particularly poor.5 6 The National Lung
Screening Trial (NLST) demonstrated that low- dose
CT (LDCT) screening of high- risk smokers detects
early- stage tumours and reduces lung cancer- specific
mortality by 20%.7 This finding has recently been
confirmed by the NELSON trial, which showed
a 24% reduction in lung cancer- specific mortality
in men and 33% reduction in women with LDCT
screening after 10 years of follow- up.8 The MILD
study also demonstrated a 39% reduction in lung
cancer- specific mortality over 10 years with LDCT
screening.9 Selection for these studies was based on
age and a specific threshold of smoking exposure;
in NLST, participants had a minimum of 30 pack-
years and had smoked within 15 years of trial entry.
Key messages
What is the key question?
►How well do different risk prediction models
perform in selecting a cohort for lung cancer
screening in a deprived population?
What is bottom line?
►There is significant variation in numbers
selected for screening and cancer detection
rate according to risk prediction model and
population screened; prospective analysis is
required to determine the best model and
threshold for screening selection.
Why read on?
►Our findings have direct relevance for screening
implementation within the UK, as the risk
models analysed (Liverpool Lung Project model
and PLCOM2012) are being used for screening
selection in the National Health Service England
Targeted Lung Health Check programme.
1LebrettMB, etal. Thorax 2020;0:1–8. doi:10.1136/thoraxjnl-2020-214626
on July 6, 2020 by guest. Protected by copyright.http://thorax.bmj.com/Thorax: first published as 10.1136/thoraxjnl-2020-214626 on 6 July 2020. Downloaded from
Lung cancer
Retrospective modelling of the NLST population demonstrated
that the benefits and harms of screening varied according to lung
cancer risk. Very few deaths were prevented among those in the
lowest risk quintile but screening related harms still occurred.10
More precise targeting of screening, based on individual lung
cancer risk, is proposed as a more effective approach.
The PLCOM2012 risk prediction model, which was derived
from the large PLCO study (including 80 375 ever- smoking
participants), estimates 6- year lung cancer risk.11 At a threshold
of ≥1.51%, PLCOM2012 showed improved sensitivity, specificity
and positive predictive value for lung cancer detection compared
with NLST criteria.12 The PanCan study used a precursor of the
PLCOM2012 model, at a 6- year lung cancer risk threshold ≥2%,
to select individuals for screening in a large Canadian screening
trial. This resulted in the detection of 172 lung cancers in 2537
eligible smokers, 77% at an early stage.13 In the UK Lung Cancer
Screening Trial (UKLS) the Liverpool Lung Project model (LLPv2)
was used; those with a 5- year lung cancer risk of ≥5% were
eligible for screening. After one screening round and nodule
surveillance, 2.1% were diagnosed with lung cancer, 86% of
which were early stage.14
The Manchester Lung Health Check (LHC) pilot offered
targeted LDCT screening to ever- smokers in deprived areas of
Manchester. Annual screening, over two screening rounds, was
offered to individuals with a 6- year lung cancer risk score of
≥1.51% calculated using the PLCOM2012 risk calculator.15 As
well as recording lung cancer- related outcomes, other metrics
were collected such as spirometry and 10- year cardiovascular
disease (CVD) risk (QRISK2 score). Here, we compare screening
performance according to population selection based on NLST
eligibility criteria or individual lung cancer risk, calculated using
PLCOM2012 and LLPv2 models, in participants of the Manchester
LHC pilot. This is of relevance for screening implementation in
the UK as National Health Service England (NHSE) have funded
an expansion of LHCs at 14 sites across the country, employing
both PLCOM2012 ≥1.51% and LLPv2 ≥2.5% risk thresholds to
select screening participants.16
The risk of lung cancer is strongly correlated with increasing
age and tobacco smoke exposure. Previous studies have shown
that increased risk of lung cancer is accompanied by an increased
risk of having comorbid conditions such as chronic obstruc-
tive pulmonary disease (COPD) and CVD.17 Individuals with
comorbid conditions may experience less benefit and more
harm from screening.18 This may negatively impact the efficacy
and efficiency of the screening programme.19 Therefore, we
also consider how comorbidity and respiratory symptoms are
correlated with lung cancer risk.
METHODS
The Manchester LHC pilot took place in three deprived areas
of Manchester (Harpurhey, Gorton and Wythenshawe) during
2016–2017. The design and approach used by the service
has previously been described.15 In brief, ever- smokers aged
55–74 from 14 general practices were invited to take part in
a community- based LHC. The LHC consisted of an assess-
ment of symptoms, measurement of spirometry, calculation of
6- year lung cancer risk using the PLCOM2012 model and smoking
cessation advice. Those at higher risk of lung cancer (defined
as PLCOM2012 score ≥1.51%) were eligible for annual LDCT
screening over two screening rounds.
Variables used in the PLCOM2012 risk calculator are age, race/
ethnicity, body mass index, detailed tobacco smoking history,
self- reported COPD status, personal history of cancer, family
history of lung cancer and educational attainment.11 US educa-
tional levels were adapted to the UK population in accordance
with the International Standard Classification of Education.20
Data were also collected to allow calculation of 5- year lung
cancer risk using the LLPv2, although this was not used for
screening selection. Variables included in LLPv2 include age, sex,
prior history of cancer, previous asbestos exposure, any first
degree relative with lung cancer (and any under the age of 60),
number of years smoked and previous history of one or more of
the following: pneumonia, emphysema, bronchitis, tuberculosis,
COPD.21 Spirometry was performed using a desktop spirometer
(Vitalograph ALPHA) to measure the forced expiratory volume
in 1 s (FEV1), forced vital capacity (FVC) ratio and respective
predictive values. Airflow obstruction was defined as FEV1/FVC
<0.7.22 All measures were prebronchodilator.
LHC participants completed a questionnaire related to symp-
toms. The Medical Research Council (MRC) dyspnoea scale was
used to categorise breathlessness (scale of increasing severity
from 1 to 5). Participants were classified as ‘symptomatic’ if they
reported one or more of the following symptoms at baseline:
breathlessness (MRC dyspnoea scale ≥2), cough ≥6 weeks in
duration, sputum production ≥teaspoon/day. The presence or
absence of emphysema was recorded by the reporting radiolo-
gist at the time of the baseline scan. Ten- year risk of CVD was
estimated, in those with no prior history of CVD, in those who
attended the second screening round using the QRISK2 calcu-
lator.23 Coronary artery calcification (CAC), another indepen-
dent predictor of CVD, was measured using the LDCT images
produced during the lung cancer screening. CAC was classified
subjectively by the reporting radiologist as: none, mild, moderate
or severe.24
We used hypothetical detection rates to approximate the
number of lung cancers that might have been detected in
Manchester if screening had been continued for a total of 6
years. In the NELSON trial, lung cancer detection rates at each
screen were 0.9% at baseline, 0.8% in the second screening
round (1 year after baseline), 1.1% in the third (3 years after
baseline) and 0.8% in the fourth screening round (5.5 years after
baseline). This is one- third and one half the detection rate seen
in Manchester at baseline (3.0%) and the second round (1.6%).
The projected detection rates used for analysis were therefore
adjusted twofold to 2.2% (3rd round) and 1.6% (4th round).
Lung cancer incidence data from the NELSON trial was used
as a basis for projections because it was undertaken in a compa-
rable European population, at a large scale and the duration of
screening was similar (5.5 years) to the 6- year risk calculated
by PLCOM2012. The denominator was total number eligible for
screening minus the number of participants diagnosed with lung
cancer prior to that screening round, thus the estimated number
of cases assumes complete adherence. Lung cancers diagnosed
outside of screening (interval cancers) in the NELSON trial were
not included in the analysis, and therefore, we assumed that no
interval cancers would occur. This is consistent with our own
data which showed no interval cancers between the first and
second screening round.25
Statistical analysis was carried out using IBM SPSS Statistics
V.25. Comparison of means was conducted by T test or one- way
analysis of variance (ANOVA). All p values for ANOVA were
adjusted for multiple testing by post hoc Bonferroni correc-
tion. P values for trend were calculated by fitting linear or
logistic regression models, using a single independent variable
with 1 degree of freedom. We tested a number of variables, not
included in the PLCOM2012 model, in a logistic regression analysis
to see if they were independently predictive of lung cancer after
2LebrettMB, etal. Thorax 2020;0:1–8. doi:10.1136/thoraxjnl-2020-214626
on July 6, 2020 by guest. Protected by copyright.http://thorax.bmj.com/Thorax: first published as 10.1136/thoraxjnl-2020-214626 on 6 July 2020. Downloaded from
Lung cancer
Table 1 Distribution of variables according to LC risk category (as calculated by PLCOM2012)
Variable
PLCOM2012 score
<1.51% 1.51%–2% 2%–3% 3%–4% 4%–6% 6%–9% ≥10% P value for trend
No participants (%) 1112 265 (19) 294 (21) 233 (16) 263 (18) 221 (16) 153 (11) –
No screened (%) n/a 263 (19) 291 (21) 231 (16) 256 (18) 218 (16) 151 (11) –
% with LC (n)
(%M/F)
– 1.5 (4)
(1.5/1.6)
1.4 (4)
(1.5/1.3)
3.0 (7)
(2.6/3.4)
5.5 (14)
(3.2/7.6)
7.8 (17)
(6.6/9.4)
10.6 (16)
(10.8/10.4)
<0.0001
NNS to detect 1 LC within each individual risk group (M/F) – 66
(67/65)
73
(66/80)
33
(38/29)
18
(31/13)
13
(15/11)
9
(9/10)
–
NNS to detect 1 LC at or above risk score (M/F) – 23
(26/20)
20
(23/18)
16
(19/14)
13
(16/11)
11
(12/10)
9
(9/10)
–
Mean age (±SD) 63.4±5.5 62.4±5.6 63.4±5.4 64.1±5.2 65.0±4.8 66.1±4.5 69.7±3.6 <0.0001
Sex % F (n) 52 (573) 49 (130) 55 (162) 51 (118) 52 (136) 44 (98) 52 (79) 0.388
Age start smoking (±SD) 17.6±6.3 16.3±5.1 16.2±4.3 16.0±4.0 15.8±4.0 15.3±4.0 15.0±3.1 <0.0001
Current smokers % (n) 12 (137) 39 (104) 41 (121) 51 (118) 57 (149) 66 (145) 77 (117) <0.0001
Mean years smoked (±SD) 22.8±12.6 37.5±8.0 40.2±6.8 42.6±6.9 45.7±5.6 48.5±6.0 53.5±4.6 <0.0001
Mean cigs/day (±SD) 16±12 21±10 22±11 24±12 26±17 26±12 27±14 <0.0001
LC relative* % yes (n) 15 (161) 17 (45) 24 (70) 19 (45) 30 (80) 39 (85) 44 (67) <0.0001
MRC dyspnoea scale 1 % (n) 78 (871) 63 (168) 72 (211) 70 (70) 66 (174) 57 (125) 52 (80) <0.0001
Symptomatic† % yes (n) 38 52 (136) 51 (149) 55 (129) 62 (163) 73 (162) 76 (116) <0.0001
FEV1 (±SD) 2.6±0.8 2.4±0.7 2.3±0.7 2.2±0.7 2.0±0.7 2.0±0.7 1.7±0.6 <0.0001
% pred FEV1 (±SD) 98±24 91±23 90±22 86±26 81±23 78±25 73±23 <0.0001
FVC (±SD) 3.4±1 3.3±0.9 3.2±0.9 3.2±0.9 3.0±1.0 3.1±1.0 2.9±0.9 <0.0001
% pred FVC (±SD) 107±25 102±25 103±22 101±27 97±24 95±24 96±24 <0.0001
FEV1: FVC ratio (±SD) 75±9 72±9 70±10 68±10 67±11 64±11 61±12 <0.0001
Airflow obstn % yes (n) 21 (232) 28 (75) 39 (114) 50 (114) 58 (152) 68 (149) 74 (112) <0.0001
Emphysema % yes (n) – 54 (142) 60 (171) 64 (146) 65 (164) 69 (146) 77 (114) <0.0001
CVD % yes (n) – 17.8 (33) 17.0 (35) 24.8 (38) 18.2 (31) 31.1 (42) 30.0 (27) 0.005
QRISK2 score‡ %±SD – 21±11 22±12 23±11 24±10 27±13 31±12 <0.0001
CAC% (n)
None – 34 (81) 26 (66) 27 (52) 29 (62) 23 (42) 20 (25) –
Mild – 54 (80) 40 (103) 38 (72) 37 (81) 39 (71) 39 (49)
Moderate- severe – 12 (75) 35 (90) 35 (67) 34 (74) 39 (71) 41 (52)
*LC diagnosed in a first degree relative.
†Defined as the presence of ≥1 symptom at baseline, including: breathlessness (MRC dyspnoea scale ≥2), cough ≥6 weeks in duration, sputum production ≥teaspoon/day.
‡QRISK2 score calculated in the second screening round only in those with no prior history of CVD.
CAC, coronary artery calcification; CVD, cardiovascular disease; FEV1, forced expiratory volume in 1 s; FVC, forced vital capacity; LC, lung cancer; NNS, number needed to screen.
controlling for PLCO risk score, as indicated by a statistically
significant p value for the variable in question (<0.05). These
variables included FEV1/FVC ratio, CAC and emphysema.
RESULTS
The results from both screening rounds have previously been
published.15 25 In brief, 2541 ever- smokers had an LHC. A
total of 1429 (56.2%) were eligible for screening based on a
threshold of PLCOM2012 ≥1.51%; 1384 had a baseline LDCT
scan and 26 had a scan at T1 only. The total number who had
at least one screening scan was 1410 or 98.7% of those eligible.
The characteristics of attendees are detailed in table 1 stratified
according to lung cancer risk (PLCOM2012). Variables included in
the PLCOM2012 risk model such as age and smoking exposure
associate closely with risk score; for example, mean age increases
from 62.4 in the lowest risk (PLCOM2012 score 1.51%–1.99%)
to 69.7 in the highest risk group (PLCOM2012 score ≥10%) and
mean smoking duration likewise increases from 37.5 to 53.5
years.
Lung cancer detection
Forty- two individuals were diagnosed with screen detected lung
cancer in the first screening round (3.0%) and 19 (1.6%) in the
second. One person was diagnosed with lung cancer during
nodule surveillance after the end of the second round and has
been included in the analysis. The number needed to screen
(NNS) to detect one lung cancer using PLCOM2012 risk threshold
of ≥1.51% was 23; this ranged from 66 to 73 in those with a
risk score 1.51%–2.99%, falling to nine for those with a score
of ≥10% (table 1). There was a difference in the NNS to detect
one lung cancer at or above each risk group (simulating different
screening thresholds) between male and female subgroups
(table 1). At the ≥1.51% threshold, used in the LHC pilot, the
NNS to detect one cancer was 26 for men and 20 for women.
3LebrettMB, etal. Thorax 2020;0:1–8. doi:10.1136/thoraxjnl-2020-214626
on July 6, 2020 by guest. Protected by copyright.http://thorax.bmj.com/Thorax: first published as 10.1136/thoraxjnl-2020-214626 on 6 July 2020. Downloaded from
Lung cancer
Figure 1 Number of participants eligible for screening by each risk
model: PLCOM2012 ≥1.51%, NLST and (A) LLPv2 ≥5% (B) LLPv2 ≥2.5%.
Percentage in () = lung cancer detection rate within segment. Number
in [] = number of individuals eligible for screening. Shaded = threshold
used for LHC screening selection. LHC, lung health check; LLPv2,
Liverpool Lung Project Model; NLST, National Lung Screening Trial.
Figure 2 Distribution of screen detected lung cancers according to
lung cancer risk (PLCOM2012 and Liverpool Lung Project model (LLPv2)
score).
The NNS to detect one cancer was lower for women at every
threshold until ≥6% at which point it was similar.
In UKLS, screening eligibility was defined as LLPv2 score
≥5%. In our population, 826 participants would have qual-
ified for screening based on this threshold, including 52.1%
(n=745/1429) of those eligible by PLCOM2012 (figure 1A). One-
quarter of those diagnosed with lung cancer would not have
been eligible for screening (25.8%, n=16/62). It is unknown
how many additional cancers would have been diagnosed
within the 81 individuals eligible using LLPv2 ≥5% criteria who
were not screened (PLCOM2012 <1.51%). With a lower LLPv2
threshold of ≥2.5% (figure 1B), 1430 of those assessed would
have been eligible for screening, including 93.5% (n=58/62)
of screen- detected lung cancers. Lung cancer outcomes are
unknown among the 272 participants who were eligible by
LLPv2 ≥2.5% but ineligible by PLCOM2012 ≥1.51%. This cohort
(n=272) was significantly older (66.7±4.9 vs 62.4±5.6 years,
p<0.0001), had a lower proportion of current smokers (19%
vs 39%, p<0.0001) and lower smoking exposure (pack- years
19.4±13.7 vs 37.8±15.5, p<0.0001) than the lowest risk
screened group (PLCOM2012 score 1.51%–1.99%; n=265). If
LHC attendees had been deemed eligible by either PLCOM2012
(≥1.51%) or LLPv2 (≥2.5%), the number screened would have
increased by 19% (n=272). We show the distribution of screen
detected lung cancers according to PLCOM2012 and LLPv2 risk
score (figure 2).
Had screening participants been selected according to NLST
eligibility criteria (aged 55–74; ≥30 pack- year smoking history;
smoker within 15 years), 14% fewer individuals would have
been offered screening (n=1188) and 18% (n=11) of individ-
uals diagnosed with lung cancer would not have been eligible
for screening. A PLCOM2012 ≥1.51% threshold selected 92.7%
of individuals eligible by NLST criteria. It is unknown how
many additional cancers would have been diagnosed within the
7.3% (n=94) of individuals eligible by NLST criteria but not
screened (PLCOM2012 score <1.51%) (figure 1A). If we conser-
vatively assume that none of the 94 individuals eligible by NLST
criteria but ineligible by PLCOM2012 would have been diagnosed
with lung cancer, then detection rates in the NLST eligible
Manchester population would be 4.3% (n=51/1188) over both
screening rounds or 2.8% at T0 (n=35/1188) and 1.5% at T1
(n=16/1094). These are significantly higher (p=0.0001) than
the equivalent figures in NLST which are 1.0% (n=270/26
309) at T0 and 0.7% (n=168/24 715) at T1 equating to 1.7%
(n=438/26 309) over both rounds.26 Therefore, despite the same
eligibility criteria, the Manchester population had more than
double the rate of lung cancer detection than those recruited to
NLST.
Based on the mean PLCOM2012 score among screened partici-
pants (5.0%), the model predicts 71 lung cancers over 6 years.
The actual number of lung cancers detected was 62 (detection
rate 4.3%) after approximately 15 months (two rounds of
annual screening and 3 months nodule follow- up). Assuming a
series of detection rates modelled on the NELSON trial27 (see
the Methods section), we hypothesise that 113 lung cancers
might be diagnosed after 5.5 years (figure 3). Even with a conser-
vative projection using unadjusted NELSON incidence, 87 lung
cancers might be detected after 5.5 years. If the annual detection
rate were to remain the same as T1 (1.6%) (approximately the
pattern seen in the Danish Lung Cancer Screening Trial28), then
an estimated 168 lung cancers might be detected after 6 years,
almost 2.5 times as many as predicted by the PLCOM2012 model.
Respiratory comorbidity
Respiratory symptoms were recorded at baseline as part of the
LHC and attendees classified symptomatic if they reported one or
more of the following symptoms: breathlessness (MRC dyspnoea
scale ≥2), cough ≥6 weeks in duration or sputum production
≥teaspoon/day. Overall, 50% of attendees reported at least one
symptom (n=1273). The proportion categorised as symptomatic
was highest in the highest risk group (PLCOM2012 ≥10%; 75.9%
symptomatic) double that in those at lowest risk (PLCOM2012
≤1.5%; 37.6% symptomatic). The proportion reporting breath-
lessness (MRC dyspnoea score ≥2) increased from 22% in the
lowest risk group to 52% in the highest (p<0.001).
4LebrettMB, etal. Thorax 2020;0:1–8. doi:10.1136/thoraxjnl-2020-214626
on July 6, 2020 by guest. Protected by copyright.http://thorax.bmj.com/Thorax: first published as 10.1136/thoraxjnl-2020-214626 on 6 July 2020. Downloaded from
Lung cancer
Figure 3 Projected cumulative number of individuals with lung cancers detected within the LHC cohort over 6 years of screening with detection
rates based on: NELSON actual detection rates, NELSON- adjusted detection rates (see the Methods section) and continuation of the 1.6% LHC T1
detection rate. LHC, lung health check.
Figure 4 FEV1/FVC ratio according to lung cancer risk group. Horizontal line represents 70% threshold indicative of airway obstruction. FEV1/FVC,
forced expiratory volume in 1 s/forced vital capacity. (o = outlier 1.5-3 interquartile ranges (IQRs) from nearest edge of box. * = outlier >3 IQRs from
nearest edge of box.)
The increase in symptom burden was mirrored by changes in
lung function. Airflow obstruction was present in 37% (n=948)
of LHC attendees overall, this ranged from 21% in those at
low risk (PLCOM2012 ≤1.5%) to 74% in those at highest risk
(PLCOM2012 ≥10%). The degree of airflow obstruction (repre-
sented by reduced FEV1/FVC ratio) was inversely correlated
to PLCOM2012 score, with mean ratio dropping from 74.7%
(PLCOM2012 ≤1.5%) to 61.3% (PLCOM2012 ≥10%) (table 1,
figure 4, p<0.001). The presence or absence of emphysema was
recorded in 98.4% of scan reports (n=1387/1410); overall, 64%
(n=883) of individuals had radiological evidence of emphysema,
this ranged from 54% in the lowest (PLCOM2012 1.51% to 1.99%)
to 77% in the highest risk group (PLCOM2012 ≥10%) (p<0.001).
Cardiovascular disease
Of 939 individuals who responded to a questionnaire in the
second screening round, 21.9% reported a prior history of CVD
(defined as a previous diagnosis of one or more of the following:
myocardial infarction, angina, stroke or transient ischaemic
attack). This ranged from 17.8% (PLCOM2012 ≤1.51%) to 30.0%
(PLCOM2012 ≥10%). The presence of CAC also increased with
5LebrettMB, etal. Thorax 2020;0:1–8. doi:10.1136/thoraxjnl-2020-214626
on July 6, 2020 by guest. Protected by copyright.http://thorax.bmj.com/Thorax: first published as 10.1136/thoraxjnl-2020-214626 on 6 July 2020. Downloaded from
Lung cancer
Table 2 Univariable and multivariable analysis of the risk of screen detected lung cancer stratified according to clinical variables and PLCOM2012
score
Variable No
Risk of lung cancer
Univariable Multivariable
OR (95% CI) P value adjOR* (95% CI) P value
Emphysema No 504 1 – – –
Yes 883 1.12
(0.65 to 1.92)
0.68 – –
CAC None 328 1 – 1 –
Mild 456 2.44
(1.03 to 5.75)
0.042 2.38
(0.996 to 5.67)
0.051
Mod- Sev 429 2.84
(1.21 to 6.65)
0.016 2.62
(1.11 to 6.20)
0.029
Symptomatic No 393 1 – 1 –
Yes 1007 2.37
(1.16 to 4.85)
0.018 1.55
(0.74 to 3.28)
0.25
FEV1/FVC ratio 1400 0.97
(0.95 to 0.99)
0.001 0.98
(0.96 to 1.01)
0.19
PLCOM2012 score 1410 1.13
(1.08 to 1.17)
<0.0001 1.11
(1.05 to 1.16)
<0.0001
*Adjusted OR—the multivariable model included: CAC (none/mild/moderate to severe), symptomatic (yes/no), FEV1/FVC ratio and PLCOM2012 score.
CAC, coronary artery calcification; FEV1/FVC, forced expiratory volume in 1 s/forced vital capacity .
PLCO risk; CAC was present in 66% of the lowest risk group
increasing to 80% in the highest risk. Almost one in five of those
with PLCOM2012 ≥10% risk of lung cancer had severe CAC.
Lung cancer risk
We assessed a number of clinical variables, not included in the
PLCOM2012 model, to see if any were independently predictive
of lung cancer after controlling for PLCOM2012 score. The pres-
ence of symptoms (as defined in the Methods), FEV1/FVC ratio
and the presence of CAC were predictive of lung cancer after
univariable analysis but the presence of emphysema on LDCT
was not (table 2). There was a borderline significant associa-
tion between decreasing FEV1/FVC ratio and lung cancer after
adjustment for PLCOM2012 score alone (adjOR 1.02, 95% CI 0.99
to 1.05; p=0.067). This analysis was undertaken because FEV1/
FVC ratio is available at the time of the LHC but radiological
measures are not. After multivariable analysis (which included:
CAC, presence of symptoms, FEV1/FVC ratio and PLCOM2012
score) only PLCOM2012 score and CAC were independent predic-
tors of lung cancer (table 2). The presence of CAC was asso-
ciated with lung cancer (n=885) (adjOR 2.50, 95% CI 1.11
to 5.64; p=0.028), this association was higher in those with
moderate and severe CAC (n=429) (adjOR 2.62, 95% CI 1.11 to
6.20; p=0.029) compared with those with mild CAC (n=456)
(adjOR 2.38, 95% CI 0.996 to 5.67; p=0.051).
DISCUSSION
Retrospective analysis of PLCOM2012, LLPv2 (at ≥2.5% and ≥5%
thresholds) and NLST criteria, applied to the participants of
the Manchester LHC pilot, demonstrated significant variability
between models in lung cancer detection and numbers selected for
screening. For example, one in four of those with screen detected
lung cancer in our cohort would not have qualified for screening
in UKLS, which used the LLPv2 model at a threshold ≥5%, and
one in five using NLST criteria. In the national protocol for NHSE
Targeted LHC programme, selection for lung cancer screening is
based on either PLCOM2012 (≥1.51%) or LLPv2 (≥2.5%) score.
In our cohort, this lower LLPv2 threshold would have classified
a similar number of people as screening eligible, and among the
individuals who were screened, it selected 93.5% of those with
cancer. However, if eligibility was allowed by either PLCOM2012 or
LLPv2, the number screened would have increased by almost 20%.
It is not possible from our data to determine how many of those
20% would have been diagnosed with lung cancer, and therefore,
the potential benefit or detriment of concurrently using these two
different criteria.
One possible interpretation of our results is that the PLCOM2012
model may underestimate risk in our population. Over only
15 months of screening and associated follow- up, 87% of the
predicted 6- year lung cancer incidence rate was reached. Using
hypothetical detection rates projected based on NELSON, we
estimated a 1.6- fold increase in lung cancer incidence over 5.5
years of screening in our population compared with incidence
predicted by PLCOM2012. However, it is important to recognise
limitations to this approach. The PLCOM2012 model was devel-
oped in the PLCO trial and therefore predicts risk in the absence
of LDCT screening, whereas we have observed detection in
the presence of screening. Screening increases incidence over
a short- term period by moving lung cancer diagnoses earlier
in time, and also via overdiagnosis, the magnitude of which is
debated in lung screening.29 30 Further, we do not take drop- out
and participation rates into account when calculating projected
cancer diagnoses, nor do we account for mortality from other
causes. However, if the PLCOM2012 model does underestimate
risk in our population, this would highlight the need for a recali-
bration of the risk model to ensure optimal performance. Formal
calibration of the risk model would require at least 100 events,
which is beyond the scope of this study.31
The high- risk nature of our cohort was also evident when we
limited analysis to only those eligible for screening based on NLST
criteria, as the lung cancer detection rate was more than double
that seen in NLST. The majority of lung cancers diagnosed through
screening were found in individuals occupying the higher risk
groups. This was true for both PLCOM2012 and LLPv2 risk models.
Moving the risk threshold dramatically changes the number of
individuals eligible for screening and therefore the number of CT
6LebrettMB, etal. Thorax 2020;0:1–8. doi:10.1136/thoraxjnl-2020-214626
on July 6, 2020 by guest. Protected by copyright.http://thorax.bmj.com/Thorax: first published as 10.1136/thoraxjnl-2020-214626 on 6 July 2020. Downloaded from
Lung cancer
scans requiring reports. For example, increasing the threshold to
2% by PLCOM2012 (as in the PanCan study) reduces those eligible
for screening by 20%, with only 6.5% fewer cancers detected.
Although such a modification could mitigate the impact of limited
CT scanner and thoracic radiologist capacity in the UK, it would
fail to take into account a number of complex subsidiary consider-
ations such as the potential reduction in screening efficacy through
the selection of a more comorbid population. Our data underline
this by showing how increased lung cancer risk is associated with
increased comorbidity, such as established CVD, the presence and
severity of airflow obstruction as well as respiratory symptoms.
Previous studies have highlighted that individuals with significant
comorbidities can reduce the cost- effectiveness and benefit of a
screening programme. For example, in colorectal cancer screening,
individuals with diabetes at age 50 derive approximately 1.8 times
fewer ‘life years saved per- person’ than their counterparts without
diabetes.32 Breast cancer screening for those aged over 79 may
only be only cost- effective for the 25% of individuals with the
highest life expectancy (based on chronological age and comorbid-
ities).33 Similar recommendations have been made for lung cancer
screening based on comparative modelling analysis.34 A recent
study found that selecting lung cancer screening participants using
a model which predicts life- years gained rather than disease risk
would increase the benefits derived from CT screening by selecting
a younger, less comorbid screening cohort.35 A possible drawback
of such an approach might be that it deprives the LHC programme
of the opportunity to address common and significant comorbid-
ities such as detection of undiagnosed COPD, evaluation of CV
risk to optimise primary prevention and immediate access to stop
smoking services. Including these in a screening programme may
improve cost- effectiveness, although quantifying such an improve-
ment will require measurement of these potential added benefits.36
We investigated a number of objective measures of ‘smoking
damage’ (FEV1/FVC ratio, CAC and presence of emphysema on
CT), not included in the PLCOM2012 model, to see if they were
independently predictive of lung cancer after controlling for PLCO
risk score. The presence and degree of CAC remained an indepen-
dent predictor of lung cancer. If this association is confirmed in
other datasets, the value of CAC to inform screening strategies,
after the baseline round, should be evaluated further. The ratio of
FEV1/FVC showed borderline statistical significance to predict lung
cancer after adjusting for PLCOM2012 score. Previous studies have
shown that the inclusion of lung function test results in lung cancer
risk prediction significantly improved its precision.37 38 As spirom-
etry tests are already conducted as part of the LHC programme
(and the results are immediately available), lung function measures
could be factored into lung cancer risk prediction and screening
selection with ease. Further research is warranted to establish
whether FEV1/FVC improves prediction over current risk predic-
tion models.
Although our cohort is relatively small, it does provide ‘real-
world’ data relevant to lung cancer screening implementation.
The population screened in the LHC pilot represents the target
population for a lung cancer screening programme, particularly
regarding measures of deprivation (75% in the lowest depriva-
tion quintile), smoking history and associated comorbidity. Our
data relating to lung cancer risk and comorbidities is of partic-
ular relevance to screening implementation; previous studies
relating to lung cancer risk and comorbidities have examined
participants in randomised control trials such as NLST, where
participation bias and the healthy volunteer effect may reduce
lung cancer risk and the prevalence of comorbidities in the study
population.39 40 Conversely our data, although not immune from
bias, is derived from a cohort at high risk of lung cancer resident
in deprived areas of Manchester. This population may more
accurately indicate the health and comorbidities of screening
attendees in a ‘real- world’ setting.
One limitation of our study is that not enough time has elapsed
to allow for collection of data related to the life expectancy,
future health events and ultimate cause of death of those within
the cohort. PLCOM2012 calculates lung cancer risk over 6 years,
while our data are limited to 15 months. Furthermore, we do
not have data related to quality of life, an important metric when
it comes to decisions related to screening inclusion and success,
particularly when examining the impact of screening on individ-
uals with co- morbid conditions.
The retrospective nature of our analysis means that we are
unable to determine the outcomes of those who were not eligible
for screening using the PLCOM2012 threshold but were eligible
using LLPv2 or NLST criteria. Further work is needed to prospec-
tively evaluate the performance of risk models in different popu-
lations. The randomised controlled Yorkshire Lung Screening
Trial (YLST) is prospectively evaluating PLCOM2012 (≥1.51%),
LLPv2 (≥5%) and NLST criteria (extended to age 80) and
will provide important information for the optimal approach
for screening selection in a UK population. Future screening
programmes may incorporate screening result as a way of opti-
mising lung cancer risk prediction. Examples of such models
include PLCO2012results
41 and LCRAT+CT.42 Population selection
could also take into account ‘life- years gained’ to reduce the
impact of comorbidity on screening efficacy.35 These approaches
would require external validation and evaluation prior to clinical
implementation.
Author affiliations
1Division of Infection, Immunity and Respiratory Medicine, The University of
Manchester, Manchester, UK
2Prevention and Early Detection Theme, NIHR Manchester Biomedical Research
Centre, Manchester, UK
3Manchester Thoracic Oncology Centre, Manchester University NHS Foundation Trust,
Manchester, UK
4South Manchester Clinical Commissioning Group, Macmillan Cancer Improvement
Partnership, Manchester, UK
5Manchester Health and Care Commissioning, Manchester, UK
6Department of Radiology, Manchester University NHS Foundation Trust, Manchester,
UK
7Greater Manchester and Eastern Cheshire Strategic Clinical Networks, Manchester,
UK
8Department of Radiology, Royal Preston Hospital, Preston, UK
9Department of Radiology, Royal Blackburn Hospital, Blackburn, UK
10Department of Radiology, Christie NHS Foundation Trust, Manchester, UK
11Manchester Health and Care Commissioning, Manchester, UK
12Department of Radiology, Salford Royal NHS Foundation Trust, Salford, UK
13Department of Radiology, Stockport NHS Foundation Trust, Stockport, UK
14Academic Unit of Primary Care, University of Leeds Leeds Institute of Health
Sciences, Leeds, UK
15International Agency for Research on Cancer, Lyon, Rhône- Alpes, France
16Manchester Academic Health Science Centre, Manchester, UK
Twitter Mikey B Lebrett @MLebrett, Janet Tonge @JetstreamSol and Philip A J
Crosbie @DrPhilCrosbie
Contributors Service concept: RB, PAJC, PVB and JT. Service development by
members of the Macmillan Cancer Improvement Partnership: RB, PAJC, PVB, JT,
DC, AM, PE, ST. Service operation and delivery by the Manchester University NHS
Foundation Trust lung cancer team: HB, ME, JL, RB, PAJC and AS. Radiology reporting
by the radiology consortium: RD, MG, JH, KI, DK, SM, TN, AS, ES, BT, AW and JW. Data
collection, analysis and drafting of manuscript: MBL, HB, HAR and PAJC. Guarantor
of overall content PAJC. Review, revision and agreement of final manuscript: all
authors.
Funding The Manchester ’Lung Health Check’ pilot was supported by funding from
Macmillan Cancer Support and the Macmillan Cancer Improvement Partnership
facilitated service design and development. PAJC is supported by the NIHR
Manchester Biomedical Research Centre. MBL is supported by the NIHR Manchester
Biomedical Research Centre. HAR is supported by the INTEGRAL project (USA
7LebrettMB, etal. Thorax 2020;0:1–8. doi:10.1136/thoraxjnl-2020-214626
on July 6, 2020 by guest. Protected by copyright.http://thorax.bmj.com/Thorax: first published as 10.1136/thoraxjnl-2020-214626 on 6 July 2020. Downloaded from
Lung cancer
National Cancer Institute U19 CA203654).
Disclaimer Where authors are identified as personnel of the International
Agency for Research on Cancer/ World Health Organization, the authors alone
are responsible for the views expressed in this article and they do not necessarily
represent the decisions, policy or views of the International Agency for Research on
Cancer / World Health Organization.
Competing interests None declared.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement Data are available upon reasonable request. We
have established a Lung Cancer Steering Committee who will consider applications
for data access.
Open access This is an open access article distributed in accordance with the
Creative Commons Attribution Non Commercial (CC BY- NC 4.0) license, which
permits others to distribute, remix, adapt, build upon this work non- commercially,
and license their derivative works on different terms, provided the original work is
properly cited, appropriate credit is given, any changes made indicated, and the use
is non- commercial. See:http:// creativecommons. org/ licenses/ by- nc/ 4. 0/.
ORCID iDs
Mikey BLebrett http:// orcid. org/ 0000- 0002- 5386- 9987
JanetTonge http:// orcid. org/ 0000- 0001- 9687- 9184
Philip A JCrosbie http:// orcid. org/ 0000- 0001- 8941- 4813
REFERENCES
1 Stewart BW, Wild C. International agency for research on cancer, world Health
organization. world cancer report, 2014. Available: http:// publications. iarc. fr/ Non-
Series- Publications/ World- Cancer- Reports/ World- Cancer- Report- 2014
2 Snell N, Strachan D, Hubbard R, etal. Burden of lung disease in the UK; findings
from the British Lung Foundation’s “respiratory health of the nation” project. In: 61
Epidemiology [Internet]. European Respiratory Society 2016.
3 Howlader N, Noone A, Krapcho M, etal. Cancer Statistics Review, 1975-2014 - SEER
Statistics, National Cancer Institute. SEER Cancer Statistics Review, 1975-2014 2016.
4 Birring SS, Peake MD. Symptoms and the early diagnosis of lung cancer. Thorax .
2005;60:268–9.
5 Cancer survival by stage at diagnosis for England (experimental statistics) - Office
for National Statistics. Available: https://www. ons. gov. uk/ peop lepo pula tion andc
ommunity/ healthandsocialcare/ cond itio nsan ddis eases/ bulletins/ canc ersu rviv alby stag
eatd iagn osis fore ngla ndex peri ment alst atistics/ adul tsdi agno sed2 0122 013a nd20 14an
dfol lowe dupt o2015
6 Lung cancer diagnosis and treatment statistics | Cancer Research UK [Internet].
Available: https://www. cancerresearchuk. org/ health- professional/ cancer- statistics/
statistics- by- cancer- type/ lung- cancer/ diagnosis- and- treatment# heading- Zero
7 National Lung Screening Trial Research Team, Aberle DR, Adams AM, etal. Reduced
lung- cancer mortality with low- dose computed tomographic screening. N Engl J Med
2011;365:395–409.
8 de Koning HJ, van der Aalst CM, de Jong PA, etal. Reduced lung- cancer mortality with
volume CT screening in a randomized trial. N Engl J Med 2020;382:503–13.
9 Pastorino U, Silva M, Sestini S, etal. Prolonged lung cancer screening reduced 10- year
mortality in the mild trial: new confirmation of lung cancer screening efficacy. Ann
Oncol 2019.
10 Kovalchik SA, Tammemagi M, Berg CD, etal. Targeting of low- dose CT screening
according to the risk of lung- cancer death. N Engl J Med . 2013;369:245–54.
11 Tammemägi MC, Katki HA, Hocking WG, etal. Selection criteria for lung- cancer
screening. N Engl J Med 2013.
12 Tammemägi MC, Church TR, Hocking WG, etal. Evaluation of the lung cancer risks
at which to screen ever- and never- smokers: screening rules applied to the PLCO and
NLST cohorts. PLoS Med . 2014;11:e1001764.
13 Tammemagi MC, Schmidt H, Martel S, etal. Participant selection for lung cancer
screening by risk modelling (the Pan- Canadian Early Detection of Lung Cancer
[PanCan] study): a single- arm, prospective study. Lancet Oncol . 2017;18:1523–31.
14 Field JK, Duffy SW, Baldwin DR, etal. The UK lung cancer screening trial: a pilot
randomised controlled trial of low- dose computed tomography screening for the early
detection of lung cancer. Health Technol Assess 2016;20:1–146.
15 Crosbie PA, Balata H, Evison M, etal. Implementing lung cancer screening: Baseline
results from a community- based “Lung Health Check” pilot in deprived areas of
Manchester. Thorax 2018.
16 Targeted Screening for Lung Cancer with Low Radiation Dose Computed Tomography
Standard Protocol prepared for the Targeted Lung Health Checks Programme
[Internet]. Available: www. england. nhs. uk/ cancer
17 Sigel K, Wisnivesky JP. Comorbidity profiles of patients with lung cancer: a new
approach to risk stratification? Ann Am Thorac Soc 2017;14:1512–3.
18 Fabrikant MS, Wisnivesky JP, Marron T, etal. Benefits and challenges of lung cancer
screening in older adults. Clin Ther 2018;40:526–34.
19 Terret C, Castel- Kremer E, Albrand G, etal. Effects of comorbidity on screening and
early diagnosis of cancer in elderly people. Lancet Oncol 2009;10:80–7.
20 ISCED mappings | UNESCO UIS. Available: http:// uis. unesco. org/ en/ isced- mappings
21 Field JK, Duffy SW, Baldwin DR, etal. Uk lung cancer RCT pilot screening trial:
baseline findings from the screening arm provide evidence for the potential
implementation of lung cancer screening. Thorax 2016;71:161–70.
22 Global initiative for chronic obstructive lung disease global strategy for the diagnosis,
management, and prevention of chronic obstructive pulmonary disease (2018 report)
[Internet], 2018. Available: www. goldcopd. org
23 Hippisley J. Predicting cardiovascular risk in England and Wales: prospective derivation
and validation of QRISK2. Available: www. qresearch. org
24 Balata H, Blandin Knight S, Barber P, etal. Targeted lung cancer screening selects
individuals at high risk of cardiovascular disease. Lung Cancer 2018;124:148–53.
25 Crosbie PA, Balata H, Evison M, etal. Second round results from the Manchester
“Lung Health Check” community- based targeted lung cancer screening pilot. Thorax
2018.
26 , Aberle DR, Adams AM, etal, National Lung Screening Trial Research Team. Reduced
lung- cancer mortality with low- dose computed tomographic screening. N Engl J Med
2011;365:395–409.
27 Yousaf- Khan U, van der Aalst C, de Jong PA, etal. Final screening round of the
Nelson lung cancer screening trial: the effect of a 2.5- year screening interval. Thorax
2017;72:48–56.
28 Wille MMW, Dirksen A, Ashraf H, etal. Results of the randomized Danish lung
cancer screening trial with focus on high- risk profiling. Am J Respir Crit Care Med
2016;193:542–51.
29 Heleno B, Siersma V, Brodersen J. Estimation of overdiagnosis of lung cancer in low-
dose computed tomography screening: a secondary analysis of the Danish lung cancer
screening trial. JAMA Intern Med 2018;178:1420–2.
30 National Lung Screening Trial Research Team. Lung cancer incidence and mortality
with extended follow- up in the National lung screening trial. J Thorac Oncol
2019;14:1732–42.
31 Collins GS, Ogundimu EO, Altman DG. Sample size considerations for the external
validation of a multivariable prognostic model: a resampling study. Stat Med
2016;35:214–26.
32 Dinh TA, Alperin P, Walter LC, etal. Impact of comorbidity on colorectal cancer
screening cost- effectiveness study in diabetic populations. J Gen Intern Med
2012;27:730–8.
33 Mandelblatt JS, Schechter CB, Yabroff KR, etal. Toward optimal screening strategies
for older women. costs, benefits, and harms of breast cancer screening by age,
biology, and health status. J Gen Intern Med 2005;20:487-96–96.
34 Ten Haaf K, Bastani M, Cao P, etal. A comparative modeling analysis of risk- based
lung cancer screening strategies. J Natl Cancer Inst 2019.
35 Cheung LC, Berg CD, Castle PE, etal. Life- Gained- Based versus Risk- Based selection
of smokers for lung cancer screening. Ann Intern Med 2019;171:623.
36 Ostrowski M, Marczyk M, Dziedzic R, etal. Lung cancer survival and comorbidities in
lung cancer screening participants of the Gdańsk screening cohort. Eur J Public Health
2019;29:1114–7.
37 Tammemagi MC, Lam SC, McWilliams AM, etal. Incremental value of pulmonary
function and sputum DNA image cytometry in lung cancer risk prediction. Cancer Prev
Res 2011;4:552–61.
38 Muller DC, Johansson M, Brennan P. Lung cancer risk prediction model incorporating
lung function: development and validation in the UK Biobank prospective cohort
study. J Clin Oncol 2017;35:861–9.
39 Howard DH, Richards TB, Bach PB, etal. Comorbidities, smoking status, and
life expectancy among individuals eligible for lung cancer screening. Cancer .
2015;121:4341–7 http://www. ncbi. nlm. nih. gov/ pubmed/ 26372542
40 Pinsky PF, Gierada DS, Hocking W, etal. National lung screening trial findings by age:
Medicare- eligible versus under-65 population. Ann Intern Med 2014;161:627–33.
41 Tammemägi MC, Ten Haaf K, Toumazis I, etal. Development and validation of a
multivariable lung cancer risk prediction model that includes low- dose computed
tomography screening results: a secondary analysis of data from the National lung
screening trial. JAMA Netw Open 2019;2:e190204.
42 Robbins HA, Berg CD, Cheung LC, etal. Identification of candidates for longer lung
cancer screening intervals following a negative low- dose computed tomography
result. J Natl Cancer Inst 2019;111:996–9.
8LebrettMB, etal. Thorax 2020;0:1–8. doi:10.1136/thoraxjnl-2020-214626
on July 6, 2020 by guest. Protected by copyright.http://thorax.bmj.com/Thorax: first published as 10.1136/thoraxjnl-2020-214626 on 6 July 2020. Downloaded from