There have been few attempts to implement and dis-
seminate programmes to address the psychological health
impact of unemployment despite the burden of this problem
upon public health and health services. One approach that
has demonstrated efficacy in promoting both psychological
health and employment for this group is based upon the
principles of cognitive behavioural therapy (CBT). We have
been involved in three interventions based upon CBT to
improve the psychological health of people who are unem-
ployed, delivered through existing service structures in
Australia: employment support agencies, general practice
and mental health services. In this paper, we examine our
experiences in conducting research within these service
organizations using a framework for collaboration between
researchers and services based upon intersectoral action.
While effective collaboration can facilitate the implementa-
tion of research within systems, poor collaboration can
impact upon the integrity of research designs. In our experi-
ence, it was the capacity of service organizations to address
the psychological health impact of unemployment in particu-
lar that had a significant effect upon adoption of the inter-
vention. Service organizations did not have structures to
support the rigorous evaluation of interventions nor did they
have funding arrangements that facilitated effective col-
laboration on research to address psychological issues. The
dissemination of evidence-based interventions like CBT to
populations of people who are unemployed in Australia is
hindered by the absence of an accessible and appropriate
system through which to address the psychological health
impact of unemployment.
Keywords: health services research, psychology, unemploy-
Unemployment is a public health issue. People who are
unemployed have significantly poorer psychological health and
well-being than people who are in the workforce,1–4and use
health services in an attempt to address these problems at a rate
approximately twice that of their employed peers.5–7Poor psy-
chological health acts as a barrier to people returning to work
through decreased motivation, lowered expectations in finding
employment, and ineffective job seeking.8–10Despite this, there
are few interventions to improve the psychological health of
people who are unemployed. One of the most promising inter-
ventions is based upon the principles of cognitive behavioural
therapy (CBT). CBT is a psychological intervention that pro-
motes the development of coping resources by challenging nega-
tive and unhelpful beliefs about a specific issue or situation.8A
major benefit of CBT is that it has demonstrated long-term
treatment effects, including the prevention of relapse after the
termination of therapy and the prevention of the development
of depression and anxiety.11,12The CBT approach has been
shown to be effective in improving psychological health when
administered individually or in a group format,13,14and has been
successfully applied in clinical and non-clinical settings.15–17
Two studies have demonstrated the efficacy of group-
training in CBT on improving the psychological health of
people who are unemployed.15,18Additionally, one study found
clear evidence of a beneficial effect of CBT on employment with
34 per cent of CBT participants achieving full-time employment
compared with 13 per cent of control group participants.15
Given that employment itself has been shown to improve
psychological health,2,19broad-level CBT interventions such
as these may be of public health benefit. How can successful
psychological interventions, such as CBT, be disseminated to
populations of people who are unemployed?
The effective and efficient transfer of evidence-based inter-
ventions from research settings to populations is a key challenge
for public health. One method of facilitating this transfer is to
demonstrate that interventions with efficacy in controlled
environments, such as CBT, are also effective in real-world
environments. Implementation research involves evaluating the
effectiveness of an evidenced-based intervention within the
social or service structure through which it is most likely to be
disseminated. This approach differs from evaluation in research
settings not in terms of methodological rigor but in the degree of
regulation over intervention. That is, research settings reflect
the conditions necessary to demonstrate or refute efficacy while
From efficacy to effectiveness: case studies in
Vanessa Rose and Elizabeth Harris
Journal of Public Health
Vol. 26, No. 3, pp. 297–302
Printed in Great Britain
Journal of Public Health 26(3) © Faculty of Public Health 2004; all rights reserved.
Centre for Health Equity Training Research and Evaluation, School of Public
Health and Community Medicine, University of New South Wales and Division
of Population Health, South Western Sydney Area Health Service, Locked Bag
7017, Liverpool BC 1871, Australia
Vanessa Rose, Research Officer
Elizabeth Harris, Director
Address correspondence to Vanessa Rose.
by guest on July 14, 2011
JOURNAL OF PUBLIC HEALTH
real-world settings require collaboration between the research
team and social or service structure to facilitate implementation
and demonstrate effectiveness. This process of collaboration is
complex and can require change in interventions, settings and
services at the level of practitioner, client and organization.20
In this paper, we describe our experience in implementing
three CBT-based interventions to promote psychological health
within organizations that deliver services to people who are
unemployed in Australia. We examine these experiences using
a framework for collaboration adapted from the intersectoral
action for health framework21to identify factors that are import-
ant to the effective administration of implementation research.
The CBT interventions were implemented within employment
support agencies, general practice and mental health services.
We do not intend to present the results of the effectiveness of
these interventions in promoting health or practice because
this is published elsewhere.22–24Our intention is to use these
interventions as examples in identifying barriers and facilitators
to effective collaboration that may prove useful to other imple-
A framework for collaboration in
Understanding how research teams and service organizations
work together to implement research is integral to identifying
mechanisms for effective dissemination. The intersectoral action
for health framework identifies mechanisms that facilitate the
ability of the Australian health sector to work with other sectors
in addressing health issues in a way that is more effective and
efficient than working alone.21This framework can be meaning-
fully applied to implementation research because of its focus
upon addressing issues of collaboration between organizations
or service structures that have different visions, priorities and
structures than researchers. Additionally, the components for
effective collaboration are practical and action-oriented. That
is, each component can be changed to better meet the needs of
the intervention and enhance the potential effectiveness of
implementation. The six components of effective collaboration
in implementation research are necessity, opportunity, capacity,
relationship, action and sustainability (Table 1).
Case studies of collaboration in
Case study 1: collaborating with employment support
In Australia, employment support agencies are administered
through the federal government’s Job Network system. This
system consists of different agencies and organizations that
have been funded through a competitive tendering process to
provide: job search; job matching; and intensive assistance
employment support services to people receiving unemploy-
ment benefits. Funding for these agencies was outcomes based,
meaning that it was contingent upon unemployed individuals
regaining and maintaining employment for a specified period of
The aim of this research was to evaluate the impact of a CBT-
based group training programme in promoting psychological
health and employment for people who were unemployed. The
research team conducted the evaluation and delivered the train-
ing programme. Employment support agencies provided a site
for training and screened clients for recruitment according to
study selection criteria. Five separate employment agencies
participated in this implementation research.
There was low perceived necessity for the intervention amongst
employment support agency staff because training was seen to
be incompatible with funding arrangements and the core busi-
ness of securing employment for people who were unemployed.
Training, particularly training focussing upon the development
of skills to improve psychological health was not perceived to
have a tangible benefit in terms of immediate employment out-
Table 1 Conditions for effective collaboration in implementation research
Necessity The perceived value of the intervention in assisting the service delivery structures to achieve their goals or ‘core
The potential of the intervention to build on existing policy or support within the service delivery structures to facilitate
Factors, such as provider knowledge and resources, that impact upon the ability of the service delivery structures to
How the service delivery structures and research team work together to implement the intervention and promote
The perceived appropriateness of the research design for both evaluating and implementing the intervention within the
service delivery structure
The likelihood that the intervention will continue to be supported and disseminated within the service delivery structure
Adapted from Harris et al.21
by guest on July 14, 2011
CASE STUDIES IN UNEMPLOYMENT RESEARCH
There was limited opportunity to draw upon employment sector
policy and support to implement the intervention. Training-
based employment structures were dismantled under the
outcomes-based funding scheme, meaning that employment
support agencies were unable to secure funding for clients to
attend training that did not lead to vocational qualifications.
Employment support agencies had limited capacity to dissem-
inate the CBT intervention. First, the de-emphasis of training as
a route to employment meant that many agencies no longer had
appropriate training facilities and the intervention had to be
delivered in areas not conducive to training; in one instance in
the space provided by a walkway. Secondly, employment sup-
port agencies were only 6 months into the establishment of the
new national outcomes-based employment network when this
study began and were struggling to come to terms with their
new role and funding arrangements. This meant that there was
very little opportunity for effective collaboration between the
research team and employment support agency because the vast
majority of resources were directed towards achieving stability
and security within the employment sector. Thirdly, there was
little incentive for agency staff to be trained in delivering the
intervention because the impact of unemployment upon psy-
chological health was perceived to be of little concern for
This research was driven by the research team with little support
from employment support agencies. Essentially, agencies were
prepared to support the research if it did not interfere with
priorities dictated by funding and required few if any resources.
This imbalance in commitment to the intervention led to a diffi-
cult relationship between the research team and agency staff
that affected recruitment of participants to the intervention;
including the persistent referral of clients with characteristics
that met study exclusion criteria.
Using a randomized controlled trial design to evaluate the
effectiveness of the intervention posed a significant problem for
employment support agencies. Agency staff perceived that they
acted as advocates for their unemployed clients and were unwill-
ing to commit their clients to a process with an uncertain
outcome. Some agency staff attempted to refer clients to the
programme that they believed would be of most benefit and
withdrew clients from participation when this could not be
guaranteed. This action had a significant impact upon the rep-
resentativeness of the study sample.
The sustainability of CBT interventions within employment
support agencies is unlikely given current funding arrangements
and the de-emphasis upon training to improve psychological
health within the employment sector.
Implications for future research with employment support
Research with Australian employment support agencies must
focus upon achieving recognition of the psychological health
impact of unemployment; and the negative effect that unem-
ployment, and long-term unemployment in particular, has upon
subsequent re-employment and job-search activity. The greatest
gains by researchers in this respect may be made by collaborat-
ing with the institutions that determine organizational priority
through funding and the development of employment support
Case study 2: collaborating with general practice
organizations and general practitioners
The majority of Australian general practitioners (GPs) are
private medical practitioners funded through the federal gov-
ernment programme Medicare, a universal system of health
insurance that provides rebates on a fee-for-service basis.
General practice organizations are funded by the federal
government to support the delivery of primary care by GPs
practising in a defined area. These organizations have many
roles including: disseminating and supporting federal govern-
ment primary care initiatives; providing training; supporting
GPs in practice management and organization; advocating for
GP interests; and delivering programmes to improve the health
of particular groups in the community.
The aim of this research was to train GPs in brief and prac-
tical CBT techniques to use with their unemployed patients who
were experiencing psychological health problems. The research
team delivered the training programme and evaluated the
impact of training upon GP skills. The general practice organi-
zation recruited GPs and provided catering and a training
There was a high level of perceived necessity for the intervention
amongst GPs and the general practice organization because of
the high rate of unemployment within the surrounding region
and GPs’ sustained exposure to the health problems of people
who were unemployed.
There was opportunity to build upon training structures within
general practice to collaborate with the general practice organi-
zation in implementing the intervention. The Royal Australian
College of General Practitioners (RACGP) continuing pro-
fessional development (CPD) programme requires GPs to
regularly register and update their skills through training
administered by general practice organizations.
by guest on July 14, 2011
JOURNAL OF PUBLIC HEALTH
There was limited ongoing capacity to disseminate this interven-
tion within general practice. The general practice organization
did not have a mechanism or structure for the continued sup-
port of training programmes or skill development for GPs who
participated in them. This meant that reported difficulties with
administering CBT and negative attitudes toward people who
were unemployed amongst GPs could not be addressed.
Another limitation to capacity involved the financing of general
practice. Under Medicare financing arrangements, GPs can
claim reimbursement for identified services or time in consulta-
tion: however, CBT is not an identified Medicare service nor can
it be reliably administered within the time-frame of a standard
consultation. Further, GPs practising in areas of high unem-
ployment and disadvantage spend less time on average in
consultation with patients than GPs practising in advantaged
areas;25suggesting that there is little opportunity for this inter-
vention to be disseminated within general practice in the areas
that need it most.
This intervention was driven by both the research team and
general practice organization, although we tended to work in
tandem rather than in collaboration. There was a real need for
the general practice organization to be involved in the training
to engender commitment to the intervention and address issues
of capacity. The RACGP training structures are both a facili-
tator and a barrier to this: while general practice organizations
are interested in developing relationships with researchers to
deliver training that attracts CPD points for their GP members
there is little incentive for this relationship to continue.
The RACGP CPD programme requires an evaluation of GP
learning as a result of training; making this mechanism useful
for researchers interested in impacting upon the delivery of
primary care to particular groups of patients. However, the
training structures do not necessarily support rigorous research
designs that incorporate the randomization of participants or
inclusion of control groups. A significant concern for general
practice researchers in Australia is the potential for sample bias.
GPs are regularly reimbursed for their participation in research
through CPD incentives or financial remuneration. This can
have a substantial impact upon who participates in what
It is likely that some GPs will continue to use CBT techniques
with their unemployed patients at their discretion; although
these applications will likely be non-systematic, unsupported
and show limited reach in addressing the psychological health
impact of unemployment.
Implications for future research within general practice
Research activity should focus upon demonstrating the effect-
iveness of general practice based CBT interventions in improv-
ing the psychological health of people who are unemployed.
Any such trial will need to address the issue of consultation
length, perhaps through demonstration projects of extended
CBT sessions or lobbying for the introduction of Medicare items
specific to psychological intervention. Alternatively, research
may best focus upon workforce substitution, such as evaluating
interventions where psychologists are employed to address
the psychological effects of unemployment in disadvantaged
Case study 3: collaborating with mental health services
In Australia, people with chronic and severe mental illness
receive free mental health care from regionally organized mental
health services administered by state government health depart-
ments. Mental health services provide clinical treatment and
rehabilitation services centred upon a client case management
system. Vocational rehabilitation is not identified as a core goal
of treatment or rehabilitation and is provided to mental health
services clients on an ad-hoc basis. Where they exist, vocational
rehabilitation services are small units or programmes within
regional divisions of mental health services that provide pre-
vocational skills training. Vocational rehabilitation units are
reliant upon the larger mental health services structure for
budget allocation, approval of activities and the referral of
The aim of this research was to pilot-test the utility of a CBT
group training programme in promoting the psychological
resources of people with chronic and severe mental illness who
intended to re-enter the workforce. In this research, we collab-
orated with a vocational rehabilitation unit in planning and
implementing the intervention.
This pilot research was initiated by occupational therapists
(OT) within the vocational rehabilitation unit, demonstrating a
high level of perceived necessity for the intervention. The OTs
perceived that existing vocational rehabilitation programmes
were unable to impact upon major psychological barriers to
gaining and maintaining employment: that is, a lack of confi-
dence in looking for work, poor problem solving ability, and a
lack of appropriate psychological skills to manage stress.
We had the opportunity to build upon a number of policy and
support structures within the vocational rehabilitation unit to
implement this intervention. First, structures for training were
well established with mechanisms for recruitment, organization,
access to training facilities and transport of clients to and from
the training venue. Additionally, the manager of the vocational
by guest on July 14, 2011
CASE STUDIES IN UNEMPLOYMENT RESEARCH
rehabilitation unit allocated staff time and resources to support
The capacity of the vocational rehabilitation unit to disseminate
the intervention was enhanced by involving OTs in the research
planning and delivery of training; these measures ensured that
the intervention could be accommodated and supported within
training structures. However, there was a high turnover of OTs
during the intervention and capacity in terms of staff knowledge
in the goals and administration of the programme was lost each
time. Further limitations to capacity included the small client
pool of the vocational rehabilitation unit from which to draw
recruits for the intervention. Accessing clients for vocational
rehabilitation within mental health services was difficult, even
for the OTs who were well connected within the system. Essen-
tially, vocational rehabilitation was seen to be peripheral to the
core functions of symptom management and treatment and gen-
eral mental health staff tended to perceive employment as a
stressor that could induce relapse amongst people with chronic
and severe mental illness.
This research was driven by the vocational rehabilitation unit
who identified the need for client training in psychological skills
and allocated resources to supporting the intervention. This
demonstrated a shared commitment to the goals of the inter-
vention and enabled the research team and vocational rehabili-
tation unit to collaborate effectively in piloting the training
Despite this high level of commitment from the vocational
rehabilitation unit, we had some difficulty convincing staff of
the necessity for an evaluation of CBT training upon the
psychological health of people with chronic and severe mental
illness. It was not standard practice within the unit to system-
atically evaluate programmes in terms of their effectiveness in
achieving health or employment for participating clients. In
general, there appeared to be a poor understanding of the pro-
cess and utility of evaluation within mental health services.
The potential sustainability of this programme within the voca-
tional rehabilitation unit was encouraging, although access to
the intervention for people with chronic and severe mental
illnesses would likely be limited by a lack of structural support
from the wider mental health system.
Implications for future research with mental health
This pilot project needs to be extended into a larger study to
assess the effectiveness of CBT in improving the psychological
health of unemployed mental health services clients. It may be
useful to approach this research in two stages. First, by demon-
strating the efficacy of CBT in a methodologically rigorous
study that has the support of mental health services staff but not
their involvement in recruitment. Secondly, by collaborating
with mental health services staff in implementing and evaluating
a system to support the referral of clients to vocational rehabili-
Unemployment remains a significant public health problem. We
have demonstrated some of the challenges in implementing
interventions to address the psychological health impact of
unemployment within organizations that provide services to
people who are unemployed in Australia. Some of these chal-
lenges can be overcome through focussed intervention arising
from an analysis of organizational priorities and capacity to
support the delivery of psychological services. Other challenges
will require researchers to adopt different approaches to
improving the health of people who are unemployed, including
advocating for structural changes within services that better
meet the needs of the unemployed population.
Psychological interventions can be notoriously difficult to
implement in non-research settings.26Even though CBT is one
of the most prolific therapies in psychological intervention and
is increasingly being promoted as the preferred treatment for
depression, both in terms of efficacy and cost-effectiveness,27,28
we were unable to demonstrate sustainability of CBT unem-
ployment interventions in practice. Simply having the evidence
for the efficacy of an intervention does not guarantee its effect-
ive dissemination to people who are unemployed.
In Australia, there are currently few mechanisms to support
the delivery of psychological services to disadvantaged popula-
tions who are unemployed. While there are structures for dis-
seminating interventions for general health, physical health and
mental health, there is currently no structure through which we
can promote psychological health and well-being. Where do
you develop a psychological health delivery system, especially
for people who are unemployed? In other words, whose core
business is it to address the psychological effects of unemploy-
ment? This presents a challenge to public health researchers and
practitioners in collaborating with service organizations to
reduce the psychological health impact of unemployment.
Funding and support for these studies was received from the
Health Research Foundation and Mental Health Promotion
Unit, South Western Sydney Area Health Service and the Fair-
field Division of General Practice. We would like to thank all
the people who participated in this research program: our col-
leagues, Job Network providers, GPs, and people who were
by guest on July 14, 2011
302 Download full-text
JOURNAL OF PUBLIC HEALTH
1 Comino EJ, Harris E, Chey T, et al. Relationship between mental
health disorders and unemployment status in Australian adults.
Aust NZ J Psychiatry 2003; 37: (2) 230–235.
2 Harris E, Morrow M. Unemployment is a health hazard: the health
costs of unemployment. Econom Labour Relations Rev 2001;
12: (1) 18–31.
3 Mathers CD, Schofield DJ. The health consequences of
unemployment: the evidence. Med J Aust 1998; 168: (4) 178–182.
4 Bartley M. Unemployment and ill health: understanding the
relationship. J Epidemiol Commun Hlth 1994; 48: (4) 333–337.
5 Comino EJ, Harris E, Silove D, Manicavasagar V, Harris MF.
Prevalence, detection and management of anxiety and depressive
symptoms in unemployed patients attending general practitioners.
Aust NZ J Psychiatry 2000; 34: (1) 107–113.
6 Harris E, Webster IW, Harris MF, Lee PJ. Unemployment and
health: the healthcare system’s role. Med J Aust 1998; 168: (6)
7 Studnicka M, Studnicka Benke A, Wogerbauer G, et al. Psychological
health, self-reported physical health and health service use. Risk
differential observed after one year of unemployment. Soc Psychiatry
Psychiatr Epidemiol 1991; 26: (2) 86–91.
8 Creed PA. Improving the mental and physical health of unemployed
people: why and how? Med J Aust 1998; 168: (4) 177–178.
9 Eden D, Aviram A. Self-efficacy training to speed reemployment:
helping people to help themselves. J Appl Psychol 1993; 78: 352–360.
10 Price RH, Choi JN, Vinokur AD. Links in the chain of adversity
following job loss: how financial strain and loss of personal control
lead to depression, impaired functioning and poor health. J Occup
Hlth Psychol 2002; 7:(4) 302–312.
11 Seligman MEP, Schulman P, DeRubeis RJ, Hollon SD.
The prevention of depression and anxiety. Prevention and Treatment
1999; 2: Article 8. Available at
12 Scott J, Palmer S, Paykel E, Teasdale J, Hayhurst H. Use of cognitive
therapy for relapse prevention in chronic depression.
Cost-effectiveness study. Br J Psychiatry 2003; 182:221–227.
13 Marques AC, Formigoni ML. Comparison of individual and group
cognitive-behavioral therapy for alcohol and/or drug-dependent
patients. Addiction 2001; 96: (6) 835–846.
14 Manassis K, Mendlowitz SL, Scapillato D, et al. Group and
individual cognitive-behavioral therapy for childhood anxiety
disorders: a randomized trial. J Am Acad Child Adolesc Psychiatry
2002; 41: (12) 1423–1430.
15 Proudfoot J, Guest D, Carson J, Dunn G, Gray J. Effect of
cognitive-behavioural training on job-finding among long-term
unemployed people. Lancet1997; 350: (9071) 96–100.
16 Scott C, Tacchi M, Jones R, Scott J. Acute and one-year outcome of a
randomised controlled trial of brief cognitive therapy for major
depressive disorder in primary care. Br J Psychiatry 1997; 171:
17 Hall PL, Tarrier N. The cognitive-behavioural treatment of low
self-esteem in psychotic patients: a pilot study. Behav Res Ther 2003;
41: (3) 317–332.
18 Creed PA, Machin MA, Hicks RE. Improving mental health status
and coping abilities for long-term unemployed youth using
cognitive-behaviour therapy based training interventions. J Org Behav
1999; 20: (6) 963–978.
19 Graetz B. Health consequences of employment and unemployment:
longitudinal evidence for young men and women. Soc Sci Med 1993;
36: (6) 715–724.
20 Schoenwald S, Hoagwood K. Effectiveness, transportability and
dissemination of interventions: what matters when? Psychiatr Serv
2001; 52: (9) 1190–1197.
21 Harris E, Wise M, Hawe P, Finlay P, Nutbeam D. Working together:
intersectoral action for health. Canberra: National Centre for Health
Promotion and Commonwealth Department of Human Services and
22 Harris E, Lum J, Rose V, Morrow M, Comino E, Harris M. Are CBT
interventions effective with disadvantaged job-seekers who are
long-term unemployed? Psychol Hlth Med2002; 7: (4) 401–410.
23 Rose V, Morrow M, Harris E, Harris M. Training GPs in cognitive
behaviour therapy for the unemployed. Aust Fam Phys 2004; 33:
24 Rose V, Harris E. Vocational rehabilitation within South West Sydney.
Sydney: Centre for Health Equity Training Research & Evaluation,
25 Furler J, Harris E, Chondros P, et al. The inverse care law revisited:
impact of disadvantaged location on accessing longer GP consultation
times. Med J Aust 2002; 177:(2) 78–79.
26 Grey S. If psychological therapy makes such good sense why isn’t
everybody doing it? J Ment Hlth 2002; 11: (5) 465–468.
27 Ward E, King M, Lloyd M, et al. Randomised controlled trial of non-
directive counselling, cognitive-behaviour therapy and usual general
practitioner care for patients with depression I: clinical effectiveness.
Br Med J 2000; 321: 1383–1388.
28 Bower P, Byford S, Sibbald B, et al. Randomised controlled trial of
non-directive counselling, cognitive-behaviour therapy and usual
general practitioner care for patients with depression II: cost
effectiveness. Br Med J 2000; 321:1389–1392.
Date Accepted 16.2.04.
by guest on July 14, 2011