Content uploaded by Cassyano Correr
Author content
All content in this area was uploaded by Cassyano Correr on Jun 23, 2015
Content may be subject to copyright.
REVIEW ARTICLE
Effectiveness of clinical pharmacy services: an overview
of systematic reviews (2000–2010)
Inajara Rotta
1
•Teresa M. Salgado
2
•Maria Lara Silva
3
•Cassyano J. Correr
4
•
Fernando Fernandez-Llimos
5
Received: 1 October 2014 / Accepted: 12 May 2015
Koninklijke Nederlandse Maatschappij ter bevordering der Pharmacie 2015
Abstract Background Multiple reviews have evaluated
the impact of pharmacist-delivered patient care on health-
related outcomes. However, it is unclear which of the
pharmacist-delivered interventions in these services are the
most effective. Aim of the review To gather the evidence of
the impact of clinical pharmacy services on the medication
use process or on patient outcomes using an overview of
systematic reviews. Methods PubMed was searched to re-
trieve systematic reviews published between 2000 and 2010
that assessed the impact of clinical pharmacy services on the
medication use process or patient outcomes. Two indepen-
dent reviewers evaluated the study eligibility and one ex-
tracted the description and results of the services. The
methodological quality of each review was assessed with the
R-AMSTAR tool. Results Of the 343 potentially relevant
records identified, 49 systematic reviews, comprising a total
of 269 randomized controlled trials, met the selection cri-
teria. Clinical pharmacy services that focused on specific
medical conditions, such as hypertension or diabetes melli-
tus, revealed a positive impact of pharmacists’ interventions
on patient outcomes. For other medical conditions, however,
the results were inconclusive (e.g., dyslipidemia or throm-
boprophylaxis). Interventions that targeted medication ad-
herence and assessed the impact of clinical pharmacy
services in prescription appropriateness also produced in-
conclusive results because of the variability of methods used
to assess both medication adherence and medication ap-
propriateness. Conclusions Systematic reviews that assessed
clinical pharmacy services targeting specific conditions
were more conclusive given that the intervention was well
defined, and the measured outcomes were unequivocal and
tangible. Conversely, the results were inconclusive for in-
terventions with a broader target and with monitoring pa-
rameters that were unclearly established or inconsistently
assessed across studies. These findings emphasize the need
to better define clinical pharmacy services and standardize
methods that assess the impact of these services on patient
health outcomes.
Keywords Clinical pharmacy services Pharmaceutical
care Pharmacists Systematic review
Impacts of findings on practice
•In order to create robust evidence, a better standard-
ization of interventions performed as part of clinical
pharmacy services across countries is required.
•Ambiguous terminologies for clinical pharmacy services
should be addressed through the creation of glossaries
and the achievement of international agreements on the
Electronic supplementary material The online version of this
article (doi:10.1007/s11096-015-0137-9) contains supplementary
material, which is available to authorized users.
&Fernando Fernandez-Llimos
f-llimos@ff.ulisboa.pt
1
Post-Graduate Program of Pharmaceutical Sciences, Federal
University of Parana, Curitiba, Brazil
2
Research Institute for Medicines (iMed.ULisboa), Faculty of
Pharmacy, University of Lisbon, Lisbon, Portugal
3
Post-Graduate Program of Pharmaceutical Care, Federal
University of Parana, Curitiba, Brazil
4
Department of Pharmacy, Federal University of Parana,
Curitiba, Brazil
5
Research Institute for Medicines (iMed.ULisboa) and
Department of Social Pharmacy, Faculty of Pharmacy,
University of Lisbon, Avda. Prof. Gama Pinto,
1649-019 Lisbon, Portugal
123
Int J Clin Pharm
DOI 10.1007/s11096-015-0137-9
definition and the components of each clinical pharmacy
service.
•Journal editors play an important role in ensuring the
rigorousness of the description of the interventions
performed and the outcomes measured in articles
accepted for publication.
Introduction
Over the past five decades, pharmacists have attempted to
extend their scope of activity beyond the traditional dis-
tributive and dispensing roles [1]. In 2000, the Institute of
Medicine recognized the critical role played by pharma-
cists in the areas of medication safety and management as
well as the value of pharmacist–physician collaboration in
patient care [2]. Pharmacists’ interventions were shown to
help optimize processes of care by improving the quality of
the medication use process and disease management
through effective interactions with both patients and other
health professionals [3,4]. Different terms have been used
to define pharmacists’ clinical activities; clinical pharmacy
services [5] and pharmaceutical care [6] have been two of
the most commonly used.
A vast set of published literature has assessed the impact
of clinical pharmacy services in different patient groups.
Randomized controlled trials (RCTs) demonstrated that
pharmacists have a positive impact on patient health out-
comes both in the community and hospital settings [7,8].
Several systematic reviews and meta-analyses showed that
pharmacist care was associated with improvements in
health outcomes of patients with heart failure [9], diabetes,
hypertension, or hiperlipidemia [10]. However, recent sys-
tematic reviews have raised reasonable doubts regarding the
actual impact of these pharmacist interventions [11,12].
Systematic reviews are usually performed to gather the
available evidence and develop guidelines for professional
practice. To ensure robust evidence, the quality of sys-
tematic reviews is required to be thoroughly evaluated.
Previous authors have assessed the methodological quality
of systematic reviews and meta-analyses addressing phar-
macist-led health interventions and demonstrated that the
quality of most reviews ranged from poor to moderate,
which could result in misinterpretations of results [13].
Additionally, few systematic reviews with meta-analyses
have been published on this topic due to the high hetero-
geneity of outcomes reported across primary studies [13].
Heterogeneity is not only an issue when including different
services in the review, but also when the meta-analysis
targets only one specific pharmacist service such as
Medication Therapy Management [14]. To analyze the
origin of this heterogeneity, a robust subgroup analysis
should be performed in systematic reviews [15]. However,
a limiting aspect of many systematic reviews and meta-
analyses is the poor and inconsistent description of the
pharmacist intervention across primary studies [16,17]. As
a means of addressing this issue a tool to characterize the
components of clinical pharmacy services—DEPICT (De-
scriptive Elements of Pharmacist Intervention Charac-
terization Tool)—was developed in 2013 and recently
refined as part of a larger project (http://depictproject.org)
[18,19]. This tool was designed so as each item reflects a
component of pharmacists’ interventions. This tool has
been successfully used to identify reproducible clinical
pharmacy services in the area of chronic kidney disease
based on the accuracy of the intervention description in
these studies [20]. Broader scope systematic reviews could
be used to identify the common components among dif-
ferent successful clinical pharmacy services.
Aim of the review
The aim of this study was to assess the impact of sub-
stantially different clinical pharmacy services on the
medication use process or on patient outcomes using an
overview of systematic reviews published in the first dec-
ade of the 2000s and to find common elements among these
services.
Methods
An overview of published systematic reviews was con-
ducted following the Cochrane Collaboration recommen-
dations and the PRISMA statement [21,22]. To identify
the articles published between 2000 and 2010, Medline
(PubMed) was searched in December 2012 employing the
following search strategy: systematic review*[TIAB]
OR meta-analysis[PT] OR meta-analysis[TIAB] OR sys-
tematic literature review[TIAB] OR ‘‘cochrane database
syst rev’’[JOURNAL] OR [search*[TIAB] AND (medline
OR embase OR peer-review* OR literature OR ‘‘evidence-
based’’ OR pubmed OR ipa or ‘‘international pharmaceu-
tical abstracts’’)] NOT [letter(PT) OR ‘‘newspaper arti-
cle’’(PT) OR comment(PT)] AND hasabstract AND
[pharmacist*(TIAB) OR pharmacists(MH)]. In addition,
reference lists of the systematic reviews ultimately in-
cluded were searched manually to retrieve any further
references.
Initially, two reviewers (I.R. and C.J.C.) independently
selected studies based on their title and abstract (screening
phase), with disagreement being adjudicated by a third
reviewer (F.F-L.). Articles that appeared to be potentially
relevant were fully analyzed by the same reviewers who
Int J Clin Pharm
123
considered the following inclusion criteria: systematic re-
views assessing the impact of a clinical pharmacy service
using either measures of the medication use process or
patient outcomes. Clinical pharmacy services were defined
as those where pharmacists provide patient care that opti-
mizes medication therapy and promotes health, wellness,
and disease prevention in all health care settings [5]. A
study was considered to be a systematic review if it satis-
factorily fulfilled the following three items of the PRISMA
Statement checklist: (1) item 4: a clear description of the
clinical question to be answered by the systematic review,
including participants, interventions, controls, outcomes
and study design (PICOS); (2) item 7: a description of all
data sources used to retrieve the literature and the search
period considered; and (3) item 9: a detailed description of
the studies’ selection process (number of articles included
and excluded in each step) [23].
The exclusion criteria used for our study included the
following: (1) systematic reviews in which the health in-
terventions involved pharmacists but their contributions to
the healthcare team were indistinguishable; (2) studies re-
viewing guidelines or other overviews of systematic re-
views; (3) systematic reviews analyzing non-clinical
activities, such as: drug compounding, storage, adminis-
tration (including vaccines) or other logistic activities; (4)
studies published in a language other than English, Span-
ish, Portuguese, French or German; and (5) reviews not
including at least one RCT. For systematic reviews pub-
lished in duplicate or updated versions of Cochrane re-
views, only the most recent publication was considered.
The quality of all systematic reviews was assessed using
the Revised Assessment of Multiple Systematic Reviews (R-
AMSTAR) checklist [24], which is a revised version of the
AMSTAR [25]. R-AMSTAR comprises 11 domains. Prior to
the start of the assessment, the items composing the checklist
were thoroughly discussed (I.R. and F.F-L.), and a manual to
guide the interpretation of R-AMSTAR items was created to
ensure consistency in the analysis.
Finally, the data were extracted from the systematic re-
views by one of the authors (I.R.), using a previously dis-
cussed table for extraction that included the following items:
year of publication, number of RCTs included in each re-
view, scope of the research, components of the clinical
pharmacy services described, and all the results reported
including economic, clinical and humanistic outcomes
(ECHO model) [26] and medication use process indicators.
Results
Of the 343 potentially relevant records initially identified,
228 were excluded after screening the title/abstract, and 69
were excluded after full-text analysis. Therefore, 46
systematic reviews were initially included, while three
others were identified through a manual search, resulting in
a total of 49 systematic reviews analyzed. An outline of the
selection process is presented in Fig. 1. These reviews
comprised a pool of 269 RCTs published between 1973
and 2009. The percentage of reviews who satisfactorily met
each R-AMSTAR criterion is described on Table 1. Ad-
ditionally, a detailed analysis of the quality of each review
is presented in Online Appendix 1.
The Online Appendix 2 shows the components and the
main findings of clinical pharmacy services reported in
systematic reviews. Based on these reports, clinical phar-
macy services were grouped into seven categories, which
are presented in Table 2. The main research questions ad-
dressed by systematic reviews could be grouped as follows:
interventions to improve disease or condition management
[3,27–39], patient adherence [9,40–52], appropriateness of
prescriptions [53–68] and miscellaneous interventions [4,
10,69–71].
Impact of clinical pharmacy services on disease
or condition management
The impact of clinical pharmacy services on hypertension
and diabetes management was assessed in six [3,29–33]
and four [27,34,36,39] systematic reviews, respectively.
All the studies included patient education and counseling
regarding disease, therapy and lifestyle modifications, and
all of them showed positive results. The reduction in sys-
tolic blood pressure ranged from 8 to 11 mmHg, and the
reduction in HbA1c ranged from 0.9 to 2.1 %. Drug ther-
apy adjustments performed after medication review and
medication follow-up were also reported in six of the re-
views [3,31–34,39]. Three reviews also described other
healthcare professionals’ education performed by pharma-
cists [29,30,36]. Wubben et al. [39] noted that there was a
greater effect of the pharmacist intervention when the
pharmacists were granted prescription autonomy.
Hyperlipidemia management by pharmacists was
assessed in two [28,35] systematic reviews. One of the
reviews included a meta-analysis and showed that total
cholesterol significantly improved after pharmacist inter-
vention [mean (SD) 22.0 (10.4) mg/dL, P=0.034], but
LDL-C, HDL-C, or triglycerides did not improve [35].
Despite the results not being statistically significant, the
other review showed that more patients in the intervention
group reached the total cholesterol goal, reduced their
LDL-C and triglycerides and improved their HDL-C [28].
Additionally, more patients in the intervention group (57
vs. 31 %) had a cholesterol panel ordered and changes in
the dose of their cholesterol-lowering medication [OR
3.0; 95 % confidence interval (CI) 2.2–4.1; P\0.001]
[28].
Int J Clin Pharm
123
Anticoagulation management was assessed in one sys-
tematic review [37]. Pharmacists’ interventions consisted
of warfarin dose adjustment, identifying potential drug–
drug interactions and educating patients or health profes-
sionals. These activities were shown to significantly im-
prove the prevention of total bleeding (RR 0.51; 95 % CI
0.28–0.94; P=0.019). Other warfarin-related complica-
tions, such as major bleeding, thromboembolic events, all-
cause mortality and warfarin-related mortality, did not
improve after pharmacists’ intervention [37].
Smoking cessation programs led by pharmacists were
the focus of two reviews [28,38]. Both these studies pro-
duced inconclusive results, possibly because both reviews
included only two RCTs that had contradictory findings
with regards to the prevalence of abstinence during the
follow-up period.
Impact of clinical pharmacy services on medication
adherence
Fourteen systematic reviews addressed clinical pharmacy
services that aimed to enhance patient medication adher-
ence [9,40–52]. In all 14 reviews, the pharmacist inter-
vention consisted of providing patient counseling.
Adherence rates improved when counseling was ad-
dressed to both the patient and the physician, but the rates
did not change when the target was the physician only
[47]. Providing medication follow-up in addition to pa-
tient counseling [9,43,49], self-monitoring blood pres-
sure devices [48] or both [42] produced mixed results.
Supplementing patient counseling with medication rec-
onciliation [40] or giving the pharmacists prescription
autonomy [46] did not improve results. Eight of the 14
adherence reviews that presented inconclusive findings
included a medication review, [40,41,44–46,50–52] and
in six of them [40,41,45,50–52] the pharmacy service
also comprised a comprehensive medication therapy
management program with different follow-up duration.
Three of these reviews concluded that the variability in
the adherence rates found across studies was due to the
variety of methods used to assess medication adherence [9,
44,52]. Additionally, another review pointed out that
studies in this area were heterogeneous in terms of quality,
patient population, duration, outcomes measured, and
lengths of follow-up [40].
The most successful pharmacist interventions included
the use of electronic devices [42], a system of reminders
and blister packs combined with [50] or without [46]
education and pharmacist follow-up, providing concurrent
oral and written information [43], and regular scheduled
consultations with the pharmacist at the time of prescrip-
tion refill [44].
Fig. 1 Flowchart illustrating
the selection process of
systematic reviews. RCTs
randomized controlled trials,
CPS clinical pharmacy services,
SR systematic reviews
Int J Clin Pharm
123
Table 1 Percentage of reviews that satisfactorily met each R-AMSTAR criterion
Criterion Description Yes
(%)
Q 1.a ‘‘A priori’’ design established 100
Q 1.b Statement of inclusion criteria 100
Q 1.c PICO/PIPO research question (population, intervention, comparison, prediction, outcome) 35
Q 2.a There were at least 2 independent studies selectors and data extractors as stated or implied 61
Q 2.b Statement of recognition or awareness of consensus procedure for disagreements 59
Q 2.c Disagreements among extractors resolved properly as stated or implied 8
Q 3.a At least 2 electronic sources were searched 90
Q 3.b The report includes years and databases searched 100
Q 3.c Key words and/or MESH terms are stated and the search strategy is provided 39
Q 3.d In addition to the electronic databases, the search was supplemented by consulting current contents such as reviews,
textbooks, specialised registers, or experts in the field of study and by reviewing the references in the studies found
55
Q 3.e Journals were ‘‘hand searched’’ or ‘‘manual searched’’ (i.e., identifying highly relevant journals and conducting a manual,
page-by-page search of their entire contents looking for potentially eligible studies)
16
Q 4.a The authors stated that they searched for reports regardless of publication type 4
Q 4.b The authors state whether or not they excluded any reports (from the SR), based on their publication status or language 80
Q 4.c ‘‘Non-English’’ papers were translated or readers sufficiently trained in foreign language 39
Q 4.d No language restriction or recognition of non-English articles 31
Q 5.a Table/list/or figure of included studies was provided; a reference list does not suffice 100
Q 5.b Table/list/or figure of excluded studies was provided either in the article or in a supplemental source (i.e., online). (Excluded
studies refers to those studies seriously considered on the basis of title and/or abstract, but rejected after reading the body
of the text)
29
Q 5.c Author satisfactorily/sufficiently stated the reason for exclusion of the seriously considered studies 71
Q 5.d Reader is able to retrace the included and the excluded studies anywhere in the article bibliography, reference, or
supplemental source
27
Q 6.a In an aggregated form such as a table, data from the original studies are provided on the participants, interventions AND
outcomes
88
Q 6.b Provide the ranges of relevant characteristics in the studies analysed (e.g., age, race, sex, relevant socioeconomic data,
disease status, duration, severity, or other diseases are reported)
24
Q 6.c The information provided appears to be complete and accurate (i.e., there is a tolerable range of subjectivity here. Is the
reader left wondering? If so, state the needed information and the reasoning)
98
Q 7.a ‘‘A priori’’ methods of assessment were provided (e.g., for effectiveness studies if the author(s) chose to include only
randomised, double-blind, placebo controlled studies, or allocation concealment as inclusion criteria); for other types of
studies alternative items will be relevant
76
Q 7.b The scientific quality of the included studies appears to be meaningful 69
Q 7.c Discussion/recognition/awareness of level of evidence 69
Q 7.d Quality of evidence was rated/ranked based on characterised instruments which is a created instrument that ranks the level
of evidence (e.g., GRADE)
4
Q 8.a The results of the methodological rigor and scientific quality were considered in the conclusions of the systematic review 65
Q 8.b The results of the methodological rigor and scientific quality were explicitly stated in formulating recommendations 8
Q 8.c To have conclusions integrated/drives towards a clinical consensus statement 0
Q 8.d This clinical consensus statement drives toward revision or confirmation of clinical practice guidelines 0
Q 9.a Statement of criteria that were used to decide that the studies analysed were similar enough to be pooled 3
Q 9.b For the pooled results, a test was performed to ensure the studies were combinable, to assess their homogeneity
(i.e., Chi square test for homogeneity, I
2
)
27
Q 9.c There was a recognition of heterogeneity or lack of thereof 27
Q 9.d If heterogeneity existed a ‘‘random effects model’’ was used and/or the rationale (i.e., clinical appropriateness) of
combining was taken into consideration (i.e., was it sensible to combine), or stated explicitly
18
Q 9.e If homogeneity existed, the authors stated a rationale or a statistical test 27
Q 10.a Recognition of publication bias or file-drawer effect 16*
Q 10.b Assessment of publication bias included graphical aids (e.g., funnel plot, other available tests) 16*
Int J Clin Pharm
123
Impact of clinical pharmacy services
on appropriateness of prescription
The objective of 16 systematic reviews was to assess the
impact of pharmacists on improving medication appropri-
ateness [53–68]. Seven of these reviews [56–60,63,67]
focused on elderly patients and their results were incon-
clusive. All but one [57] consisted of a medication review
service, and in one review [60] the pharmacist had pre-
scription autonomy. Another review, evaluating pharma-
cists’ interventions on the transition of elderly patients
between healthcare settings, reported positive results
namely: improvement of prescription appropriateness,
successful documentation rates, reduction of omitted
medications and a decrease in discrepancy-related adverse
drug events [59].
Table 1 continued
Criterion Description Yes
(%)
Q 10.c Statistical tests (e.g., Egger regression test) 12*
Q 11.a Statement of sources of support 88
Q 11.b No conflict of interest 59
Q 11.c An awareness/statement of support or conflict of interest in the primary inclusion studies 2
* Calculated only for the 14 meta-analyses
Table 2 Clinical pharmacy services categories identified from the literature
No. Clinical pharmacy services categories Study references
1 Patient education and counseling about medication, diseases and non-pharmacological
treatment. These services can (or not) be provided along with medication dispensing, with
the aim of promoting the correct use of medicines and adherence to treatment. The
pharmacist can also provide additional educational support, like printed materials or
multimedia and compliance aids, such as pillboxes, pill organisers, dispensers, dosage
systems, medication packs, medication diaries, reminder systems, beep-cards, among
others.
[3,4,9,10,27–57,61,63,66,68–71]
2 Structured programs for detection, prevention or control of specific risk factors (e.g.
smoking cessation, point-of-care testing, screening services), in which interventions are
usually focused on behavioral techniques and individual or group education.
[3,4,28,29,39,42,45,48,51,61]
3 Medication review and drug therapy adjustments, with or without direct contact with the
patient. The aim is to identify and correct any failures with the medication use process,
issues related to inappropriate prescribing, therapeutic regimen, treatment costs or adverse
effects. The pharmacist usually makes recommendations to the patient or physician and
can have major or minor autonomy to modify pharmacologic treatment.
[3,4,10,31–35,37,39–41,44–46,50–56,
58–69]
4 Elaboration or refinement of a complete and reliable medication history and therapeutic
reconciliation during hospital admission, transference between settings and after
discharge. These services can include provision of information to the physician and
patient, usually written, with the aim of correcting any discrepancies.
[4,40,53,59]
5 Medication therapy management and medication follow-up targeting health outcomes and
continuity of health care, by using several ways of contact with the patient and physician
(e.g. face-to-face, telephone, fax, web or email), different duration of follow-up and
number of appointments.
[3,4,9,10,27,28,31–35,39–43,45,49–
58,60,61,65–67,69,71]
6 Provision of information by the pharmacist to the physician and health care team, without
the need of direct patient care. It may include multidisciplinary case discussions, ward
rounds, development of clinical protocols, therapeutic formularies and closer relationships
with the team. It also includes services of academic detailing, in order to provide scientific
information and to promote good practices of prescription, usually focused on specific
clinical conditions or medications.
[29,30,36,37,47,50,59–61,67–69]
7 Services in which the pharmacist has the autonomy to manage or prescribe medicines to the
patient according to pre-defined clinical protocols or collaborative agreements between
providers or within ambulatory care settings. It also includes patient referral to the
community pharmacist for assessment and management of minor illness.
[46,55,60,61]
Int J Clin Pharm
123
Among the causes of variability in the results across
primary studies is the lack of agreement of the definition of
polypharmacy [63]. The use of different process indicators
such as the medication appropriateness index (MAI), the
Beer’s criteria, or ad hoc-created indicators was also re-
ported as a major cause of heterogeneity across studies [58,
60]. Even when the rate of inappropriate prescriptions was
reduced, no effect was observed in the patients’ clinical
outcomes such as morbidity, hospitalizations, mortality or
healthcare costs [56,57].
Most of the remaining nine reviews that were not fo-
cused on elderly patients showed positive results with re-
gards to improving medication appropriateness [54,55,61,
62,65,68]. Services such as medication review, medica-
tion follow-up, patient education and prescribing new
medications showed a positive impact on optimizing an-
timicrobial prescriptions [65,68], improving medication
use in children [54], enhancing patient safety [62], reduc-
ing the number of prescribed medications [61], and im-
proving prescribing practices, patient satisfaction and cost
avoidance [55]. However, a pharmacist-led medication
review was not effective in reducing hospital admissions
when clinical outcomes were considered [64]. Even when
the medication review was complemented with a follow-up
period, the results did not demonstrate a consistent im-
provement of patients’ quality of life or satisfaction nor did
it reduce adverse drug reactions or drug procurement costs.
These negative findings were attributed to an underlying
different research design of the studies included [66].
Other impacts of clinical pharmacy services
Five systematic reviews could not be grouped into the
previous categories because they described very heteroge-
neous outcomes. Stemer et al. [71] assessed the impact of
therapeutic drug monitoring and patient education on the
management of solid organ transplant recipients and found
positive perceptions of patients and healthcare profession-
als and high physicians’ acceptance rates of pharmacist’s
recommendations. Chisholm-Burns et al. [10] assessed the
integration of a pharmacist within a multidisciplinary team
and found favorable results in effectiveness and safety;
however, they also found less favorable results in human-
istic outcomes, particularly quality of life. Ellit et al. [69]
evaluated the pharmacist’s role in continuity of patient care
and also found positive results for economic, clinical and
humanistic outcomes. However, the authors criticized the
exclusion criteria used in 19 of the 21 included studies, and
these criteria may have biased their results. Kaboli et al. [4]
focused on pharmacists’ care to inpatients and found im-
provements in care in the reduction of the rate of adverse
drug events, medication errors and lengths of hospital stay,
even though the authors recognized several limitations to
their work. Naik Panvelkar et al. [70] showed high levels of
patient satisfaction with any type of community pharmacy
services, but they referred to the lack of consistent instru-
ments to measure this humanistic outcome.
Discussion
The present overview of systematic reviews was purpose-
fully conducted broad in scope in order to identify common
elements across substantially different clinical pharmacy
services. An in-depth analysis of 49 systematic reviews
revealed that clinical pharmacy services that focused on
specific medical conditions such as hypertension or dia-
betes mellitus showed a positive impact on outcomes, the
common element being the measurement of unequivocal
and tangible outcomes. However, interventions that tar-
geted medication adherence or prescription appropriateness
produced inconsistent results. Due to the small number of
systematic reviews addressing hyperlipidemia, warfarin
therapy management by pharmacists and smoking cessa-
tion programs, we could not draw a conclusion of the im-
pact of clinical pharmacy services on these conditions.
After applying the R-AMSTAR to the systematic re-
views, we identified that most were insufficiently reported,
and many did not employ methodological procedures that
are critical to reduce the risk of bias. For example, 30
reviews included only studies published in English, which
does not account for potential language or publication bias,
and study selection and data extraction were not performed
by two independent reviewers in 17 other reviews. In ad-
dition, for 33 % of the systematic reviews the authors did
not assess nor documented the methodological quality of
the primary studies included. These findings are in line
with those reported in a study that assessed the quality of
systematic reviews and meta-analysis on pharmacist health
interventions which concluded that the quality of published
reviews varied from moderate to poor [13]. Additionally,
some reviews have only reported the primary studies’ re-
sults individually without synthesizing the findings, as
opposed to what the PRISMA statement advocates: ‘‘au-
thors should give a brief and balanced summary of the
nature and findings of the review’’ [22]. Although the
R-AMSTAR was originally created with a scoring system,
we preferred not to use it similarly to what other authors
have done [72]. Scoring systems have been criticized for
their excessive rigidity in favor of more versatile systems
as it happens with the Cochrane’s ‘‘risk of bias’’ instrument
[21].
Meta-analyses were only performed in 14 of the 49 re-
views due to the heterogeneity of interventions described
and the outcomes reported across the primary studies. Poor
or inconsistent description of the pharmacists’ interventions
Int J Clin Pharm
123
was one of the main limiting factors to the quality and
reproducibility of the studies assessing the impact of clin-
ical pharmacy services [13,16,73,74]. Therefore, gener-
ating a definitive list of services from the available evidence
is not an easy task.
Among studies describing interventions that targeted
medication adherence and which were shown to produce
inconclusive results, several factors might have contributed
to the heterogeneity of the findings, including the wide
variability of methods used to assess medication adherence,
such as: self-report tools, pill counts, refill of prescriptions,
electronic medication monitors, i.e., MEMS, or medication
diaries. Adherence estimates when measured by different
methods varied across studies [75–78]. One study con-
cluded that pill-count was a superior method of medication
adherence assessment compared to 24-h recall and refill
history in both clinical practice and long-term medication
studies [10]. Another study, however, drew attention to the
fact that a summary measure combining several measures
was more strongly related to a clinical response [78]. Pa-
tient self-reported adherence and prescription refill records
were found to be poorly correlated [76] and patient self-
report appeared to overestimate adherence [46,75,77].
Other contributing factors to the variety of results could be
the selection of patients with different adherence rates at
baseline [79,80] and different cutoff points to classify
adherence behavior [46].
Conflicting evidence was also found across systematic re-
views that examined the impact of pharmacist interventions on
the quality of prescribing. Similarly to the adherence findings,
the multitude of instruments available to assess suboptimal
prescribing may be the underlying reason for these discrepan-
cies. Some authors reported different ability of different tools,
such as the Medication Appropriateness Index (MAI), Beers’
criteria 2003, the Improved Prescribing in the Elderly Tool
(IPET) and Health Plan Employer Data and Information Set
(HEDIS), to assess changes in medication appropriateness [81,
82]. Multidimensional approaches using different tools simul-
taneously will likely be necessary to robustly assess the quality
of prescribing [82]. Another important aspect is that the authors
used endpoints such as hospitalization, mortality or outpatient
visits as effectiveness indicators, but these endpoints require
longer follow-up periods to show a potential effect. Thus, in-
termediate or surrogate outcomes such as level of disease
control could be used as proxy indicators of pharmacists’ ser-
vice effectiveness to appropriately measure an intervention’s
short-term effect [12].
As practical implications of our study, we highlight the
need to better standardize the interventions performed as
part of clinical pharmacy services, especially in services
involving complex interventions which present a great
number of components and therefore more likelihood of
variability. This will require a close collaboration between
researchers and practitioners, and also more international
collaboration among pharmacy practice researchers. Am-
biguous terminologies should be eliminated, not only by
creating glossaries [83], but also through the achievement of
international agreements on the definition and the compo-
nents of each clinical pharmacy service [84]. Additionally,
journal editors should be very rigorous with regards to the
description of the interventions performed and the outcomes
measured in articles accepted for publication.
The main limitation of our study is the specific time
frame used (2000–2010). However, we believe that
analyzing the first decade of the 2000s would ensure that
the included studies reflect a higher position in the learning
curve of clinical pharmacy services development. Only
Pubmed was used to search the studies included since this
is one of the most comprehensive scientific databases [85]
and the overview method allowed for the collection of
systematic reviews whose primary studies were in turn
retrieved from several other databases.
Conclusions
In conclusion, clinical pharmacy services seem to be more
successful when they target specific medical conditions,
such as diabetes mellitus or hypertension, and when using
objective parameters to assess patient health status, such as
blood pressure or glycosylated hemoglobin. The results are
inconclusive for the pharmacists’ interventions that have a
broader target and whose monitoring parameters are not
clearly established or have an unstandardized assessment.
Although clinical pharmacy services seem to improve pa-
tients’ health, efforts should be done to prove the added
value of these services based on evidence-based practice
standards and an intensive analysis of components.
Funding Postgraduate Program in Pharmaceutical Sciences of the
Federal University of Parana, Brazil and Coordenac¸a
˜o de Aper-
feic¸oamento de Pessoal de Nı
´vel Superior—CAPES (Coordination for
the Improvement of Highly Educated Personnel).
Conflicts of interest None of the authors has any conflicts of in-
terest to disclose that could affect the study results.
References
1. Hepler CD. Clinical pharmacy, pharmaceutical care, and the
quality of drug therapy. Pharmacotherapy. 2004;24(11):1491–8.
2. Kohn LT, Corrigan JM, Donaldson MS. To err is human.
Washington, DC: National Academy Press; 2000.
3. Carter BL, Rogers M, Daly J, Zheng S, James PA. The potency of
team-based care interventions for hypertension: a meta-analysis.
Arch Intern Med. 2009;169(19):1748–55.
Int J Clin Pharm
123
4. Kaboli PJ, Hoth AB, McClimon BJ, Schnipper JL. Clinical
pharmacists and inpatient medical care: a systematic review.
Arch Intern Med. 2006;166(9):955–64.
5. American College of Clinical Pharmacy. The definition of clin-
ical pharmacy. Pharmacotherapy. 2008;28(6):816–7.
6. van Mil JW, Fernandez-Llimos F. What is ‘pharmaceutical care’
in 2013? Int J Clin Pharm. 2013;35(1):1–2.
7. Al-Rashed SA, Wright DJ, Roebuck N, Sunter W, Chrystyn H. The
value of inpatient pharmaceutical counselling to elderly patients
prior to discharge. Br J Clin Pharmacol. 2002;54(6):657–64.
8. Chen Z, Ernst ME, Ardery G, Xu Y, Carter BL. Physician-
pharmacist co-management and 24-hour blood pressure control.
J Clin Hypertens (Greenwich). 2013;15(5):337–43.
9. Koshman SL, Charrois TL, Simpson SH, McAlister FA, Tsuyuki
RT. Pharmacist care of patients with heart failure: a systematic re-
view of randomized trials. Arch Intern Med. 2008;168(7):687–94.
10. Chisholm-Burns MA, Kim Lee J, Spivey CA, Slack M, Herrier
RN, Hall-Lipsy E, et al. US pharmacists’ effect as team members
on patient care: systematic review and meta-analyses. Med Care.
2010;48(10):923–33.
11. Salgado TM, Moles R, Benrimoj SI, Fernandez-Llimos F. Phar-
macists’ interventions in the management of patients with chronic
kidney disease: a systematic review. Nephrol Dial Transplant.
2012;27(1):276–92.
12. Saez-Benito L, Fernandez-Llimos F, Feletto E, Gastelurrutia MA,
Martinez-Martinez F, Benrimoj SI. Evidence of the clinical ef-
fectiveness of cognitive pharmaceutical services for aged pa-
tients. Age Ageing. 2013;42(4):442–9.
13. Melchiors AC, Correr CJ, Venson R, Pontarolo R. An analysis of
quality of systematic reviews on pharmacist health interventions.
Int J Clin Pharm. 2012;34(1):32–42.
14. Viswanathan M, Kahwati LC, Golin CE, Blalock SJ, Coker-
Schwimmer E, Posey R, et al. Medication therapy management
interventions in outpatient settings: a systematic review and
meta-analysis. JAMA Intern Med. 2015;175(1):76–87.
15. Souza TT, Saez-Benito L, Correr CJ, Fernandez-Llimos F. Cri-
tical assessment of the systematic review on hospitalization re-
sulting from medicine related problems. Br J Clin Pharmacol.
2014;78(6):1465–6.
16. Kennie NR, Schuster BG, Einarson TR. Critical analysis of the
pharmaceutical care research literature. Ann Pharmacother. 1998;
32(1):17–26.
17. Charrois TL, Durec T, Tsuyuki RT. Systematic reviews of
pharmacy practice research: methodologic issues in searching,
evaluating, interpreting, and disseminating results. Ann Pharma-
cother. 2009;43(1):118–22.
18. Correr CJ, Melchiors AC, de Souza TT, Rotta I, Salgado TM,
Fernandez-Llimos F. A tool to characterize the components of
pharmacist interventions in clinical pharmacy services: the
DEPICT project. Ann Pharmacother. 2013;47(7–8):946–52.
19. Rotta I, Salgado TM, Felix DC, de Souza TT, Correr CJ, Fer-
nandez-Llimos F. Ensuring consistent reporting of clinical phar-
macy services to enhance reproducibility in practice: an improved
version of DEPICT. J Eval Clin Pract. 2015. doi:10.1111/jep.
12339. [Epub ahead of print].
20. Salgado TM, Correr CJ, Moles R, Benrimoj SI, Fernandez-Llimos
F. Assessing the implementability of clinical pharmacist inter-
ventions in patients with chronic kidney disease: an analysis of
systematic reviews. Ann Pharmacother. 2013;47(11):1498–506.
21. Higgins JPT, Green S. Cochrane Handbook for systematic re-
views of interventions version 5.1.0 [updated March 2011]. The
cochrane collaboration. 2011.
22. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC,
Ioannidis JP, et al. The PRISMA statement for reporting systematic
reviews and meta-analyses of studies that evaluate healthcare in-
terventions: explanation and elaboration. BMJ. 2009;339:b2700.
23. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting
items for systematic reviews and meta-analyses: the PRISMA
statement. Int J Surg. 2010;8(5):336–41.
24. Kung J, Chiappelli F, Cajulis OO, Avezova R, Kossan G, Chew
L, et al. From systematic reviews to clinical recommendations for
evidence-based health care: validation of revised assessment of
multiple systematic reviews (R-AMSTAR) for grading of clinical
relevance. Open Dent J. 2010;4:84–91.
25. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N,
Hamel C, et al. Development of AMSTAR: a measurement tool
to assess the methodological quality of systematic reviews. BMC
Med Res Methodol. 2007;7:10.
26. Kozma CM, Reeder CE, Schulz RM. Economic, clinical, and
humanistic outcomes: a planning model for pharmacoeconomic
research. Clin Ther. 1993;15(6):1121–32.
27. Babinec PM, Rock MJ, Lorenzetti DL, Johnson JA. Do re-
searchers use pharmacists’ communication as an outcome mea-
sure? A scoping review of pharmacist involvement in diabetes
care. Int J Pharm Pract. 2010;18(4):183–93.
28. Blenkinsopp A, Anderson C, Armstrong M. Systematic review of
the effectiveness of community pharmacy-based interventions to
reduce risk behaviours and risk factors for coronary heart disease.
J Public Health Med. 2003;25(2):144–53.
29. Fahey T, Schroeder K, Ebrahim S. Educational and organisa-
tional interventions used to improve the management of hyper-
tension in primary care: a systematic review. Br J Gen Pract.
2005;55(520):875–82.
30. Gallagher H, de Lusignan S, Harris K, Cates C. Quality-im-
provement strategies for the management of hypertension in
chronic kidney disease in primary care: a systematic review. Br J
Gen Pract. 2010;60(575):e258–65.
31. Glynn LG, Murphy AW, Smith SM, Schroeder K, Fahey T. Self-
monitoring and other non-pharmacological interventions to im-
prove the management of hypertension in primary care: a sys-
tematic review. Br J Gen Pract. 2010;60(581):e476–88.
32. Glynn LG, Murphy AW, Smith SM, Schroeder K, Fahey T. Inter-
ventions used to improve control of blood pressure in patients with
hypertension. Cochrane Database Syst Rev. 2010; (3):CD005182.
33. Machado M, Bajcar J, Guzzo GC, Einarson TR. Sensitivity of
patient outcomes to pharmacist interventions. Part II: systematic
review and meta-analysis in hypertension management. Ann
Pharmacother. 2007;41(11):1770–81.
34. Machado M, Bajcar J, Guzzo GC, Einarson TR. Sensitivity of
patient outcomes to pharmacist interventions. Part I: systematic
review and meta-analysis in diabetes management. Ann Phar-
macother. 2007;41(10):1569–82.
35. Machado M, Nassor N, Bajcar JM, Guzzo GC, Einarson TR.
Sensitivity of patient outcomes to pharmacist interventions. Part
III: systematic review and meta-analysis in hyperlipidemia
management. Ann Pharmacother. 2008;42(9):1195–207.
36. Renders CM, Valk GD, Griffin S, Wagner EH, Eijk JT, Assen-
delft WJ. Interventions to improve the management of diabetes
mellitus in primary care, outpatient and community settings.
Cochrane Database Syst Rev. 2001; (1):CD001481.
37. Saokaew S, Permsuwan U, Chaiyakunapruk N, Nathisuwan S,
Sukonthasarn A. Effectiveness of pharmacist-participated war-
farin therapy management: a systematic review and meta-analy-
sis. J Thromb Haemost. 2010;8(11):2418–27.
38. Sinclair HK, Bond CM, Stead LF. Community pharmacy per-
sonnel interventions for smoking cessation. Cochrane Database
Syst Rev. 2004; (1):CD003698.
39. Wubben DP, Vivian EM. Effects of pharmacist outpatient inter-
ventions on adults with diabetes mellitus: a systematic review.
Pharmacotherapy. 2008;28(4):421–36.
40. Banning M. A review of interventions used to improve adherence to
medication in older people. Int J Nurs Stud. 2009;46(11):1505–15.
Int J Clin Pharm
123
41. Bell S, McLachlan AJ, Aslani P, Whitehead P, Chen TF. Com-
munity pharmacy services to optimise the use of medications for
mental illness: a systematic review. Aust N Z Health Policy.
2005;2:29.
42. Cutrona SL, Choudhry NK, Fischer MA, Servi A, Liberman JN,
Brennan TA, et al. Modes of delivery for interventions to im-
prove cardiovascular medication adherence. Am J Manag Care.
2010;16(12):929–42.
43. Desplenter FA, Simoens S, Laekeman G. The impact of in-
forming psychiatric patients about their medication: a systematic
review. Pharm World Sci. 2006;28(6):329–41.
44. George J, Elliott RA, Stewart DC. A systematic review of inter-
ventions to improve medication taking in elderly patients pre-
scribed multiple medications. Drugs Aging. 2008;25(4):307–24.
45. Haynes RB, Ackloo E, Sahota N, McDonald HP, Yao X. Inter-
ventions for enhancing medication adherence. Cochrane Data-
base Syst Rev. 2008;2:CD000011.
46. Lindenmeyer A, Hearnshaw H, Vermeire E, Van Royen P, Wens J,
Biot Y. Interventions to improve adherence to medication in people
with type 2 diabetes mellitus: a review of the literature on the role of
pharmacists. J Clin Pharm Ther. 2006;31(5):409–19.
47. Morrison A, Wertheimer AI. Evaluation of studies investigating
the effectiveness of pharmacists’ clinical services. Am J Health
Syst Pharm. 2001;58(7):569–77.
48. Ogedegbe G, Schoenthaler A. A systematic review of the effects
of home blood pressure monitoring on medication adherence.
J Clin Hypertens (Greenwich). 2006;8(3):174–80.
49. Rueda S, Park-Wyllie LY, Bayoumi AM, Tynan AM, Antoniou
TA, Rourke SB, et al. Patient support and education for pro-
moting adherence to highly active antiretroviral therapy for HIV/
AIDS. Cochrane Database Syst Rev. 2006; (3):CD001442.
50. Schlenk EA, Bernardo LM, Organist LA, Klem ML, Engberg S.
Optimizing medication adherence in older patients: a systematic
review. J Clin Outcomes Manag. 2008;15(12):595–606.
51. Van Wijk BL, Klungel OH, Heerdink ER, de Boer A. Effec-
tiveness of interventions by community pharmacists to improve
patient adherence to chronic medication: a systematic review.
Ann Pharmacother. 2005;39(2):319–28.
52. Williams A, Manias E, Walker R. Interventions to improve
medication adherence in people with multiple chronic conditions:
a systematic review. J Adv Nurs. 2008;63(2):132–43.
53. Bayoumi I, Howard M, Holbrook AM, Schabort I. Interventions
to improve medication reconciliation in primary care. Ann
Pharmacother. 2009;43(10):1667–75.
54. Costello I, Wong IC, Nunn AJ. A literature review to identify
interventions to improve the use of medicines in children. Child
Care Health Dev. 2004;30(6):647–65.
55. Finley PR, Crismon ML, Rush AJ. Evaluating the impact of
pharmacists in mental health: a systematic review. Pharma-
cotherapy. 2003;23(12):1634–44.
56. Hanlon JT, Lindblad CI, Gray SL. Can clinical pharmacy services
have a positive impact on drug-related problems and health
outcomes in community-based older adults? Am J Geriatr Phar-
macother. 2004;2(1):3–13.
57. Holland R, Desborough J, Goodyer L, Hall S, Wright D, Loke YK.
Does pharmacist-led medication review help to reduce hospital
admissions and deaths in older people? A systematic review and
meta-analysis. Br J Clin Pharmacol. 2008;65(3):303–16.
58. Kaur S, Mitchell G, Vitetta L, Roberts MS. Interventions that can
reduce inappropriate prescribing in the elderly: a systematic re-
view. Drugs Aging. 2009;26(12):1013–28.
59. LaMantia MA, Scheunemann LP, Viera AJ, Busby-Whitehead J,
Hanson LC. Interventions to improve transitional care between
nursing homes and hospitals: a systematic review. J Am Geriatr
Soc. 2010;58(4):777–82.
60. Marcum ZA, Handler SM, Wright R, Hanlon JT. Interventions to
improve suboptimal prescribing in nursing homes: a narrative
review. Am J Geriatr Pharmacother. 2010;8(3):183–200.
61. Nkansah N, Mostovetsky O, Yu C, Chheng T, Beney J, Bond
CM, et al. Effect of outpatient pharmacists’ non-dispensing roles
on patient outcomes and prescribing patterns. Cochrane Database
Syst Rev. 2010; (7):CD000336.
62. Robertson J, Walkom E, Pearson SA, Hains I, Williamsone M,
Newby D. The impact of pharmacy computerised clinical decision
support on prescribing, clinical and patient outcomes: a systematic
review of the literature. Int J Pharm Pract. 2010;18(2):69–87.
63. Rollason V, Vogt N. Reduction of polypharmacy in the elderly: a
systematic review of the role of the pharmacist. Drugs Aging.
2003;20(11):817–32.
64. Royal S, Smeaton L, Avery AJ, Hurwitz B, Sheikh A. Interven-
tions in primary care to reduce medication related adverse events
and hospital admissions: systematic review and meta-analysis.
Qual Saf Health Care. 2006;15(1):23–31.
65. Tonna AP, Stewart D, West B, Gould I, McCaig D. Antimicrobial
optimisation in secondary care: the pharmacist as part of a mul-
tidisciplinary antimicrobial programme–a literature review. Int J
Antimicrob Agents. 2008;31(6):511–7.
66. Tully MP, Seston EM. Impact of pharmacists providing a pre-
scription review and monitoring service in ambulatory care or
community practice. Ann Pharmacother. 2000;34(11):1320–31.
67. Verrue CL, Petrovic M, Mehuys E, Remon JP, Vander Stichele R.
Pharmacists’ interventions for optimization of medication use in
nursing homes: a systematic review. Drugs Aging. 2009;26(1):37–49.
68. von Gunten V, Reymond JP, Beney J. Clinical and economic
outcomes of pharmaceutical services related to antibiotic use: a
literature review. Pharm World Sci. 2007;29(3):146–63.
69. Ellitt GR, Brien JA, Aslani P, Chen TF. Quality patient care and
pharmacists’ role in its continuity–a systematic review. Ann
Pharmacother. 2009;43(4):677–91.
70. Naik Panvelkar P, Saini B, Armour C. Measurement of patient
satisfaction with community pharmacy services: a review. Pharm
World Sci. 2009;31(5):525–37.
71. Stemer G, Lemmens-Gruber R. Clinical pharmacy services and
solid organ transplantation: a literature review. Pharm World Sci.
2010;32(1):7–18.
72. Kitsiou S, Pare G, Jaana M. Systematic reviews and meta-ana-
lyses of home telemonitoring interventions for patients with
chronic diseases: a critical assessment of their methodological
quality. J Med Internet Res. 2013;15(7):e150.
73. Farris KB, Kirking DM. Assessing the quality of pharmaceutical
care. II. Application of concepts of quality assessment from
medical care. Ann Pharmacother. 1993;27(2):215–23.
74. Mohler R, Bartoszek G, Kopke S, Meyer G. Proposed criteria for
reporting the development and evaluation of complex interven-
tions in healthcare (CReDECI): guideline development. Int J
Nurs Stud. 2012;49(1):40–6.
75. Burney KD, Krishnan K, Ruffin MT, Zhang D, Brenner DE.
Adherence to single daily dose of aspirin in a chemoprevention
trial. An evaluation of self-report and microelectronic monitor-
ing. Arch Fam Med. 1996;5(5):297–300.
76. Calvert SB, Kramer JM, Anstrom KJ, Kaltenbach LA, Stafford
JA, Allen LaPointe NM. Patient-focused intervention to improve
long-term adherence to evidence-based medications: a random-
ized trial. Am Heart J. 2012; 163(4):657–65 e1.
77. Grymonpre RE, Didur CD, Montgomery PR, Sitar DS. Pill count,
self-report, and pharmacy claims data to measure medication ad-
herence in the elderly. Ann Pharmacother. 1998;32(7–8):749–54.
78. Liu H, Golin CE, Miller LG, Hays RD, Beck CK, Sanandaji S,
et al. A comparison study of multiple measures of adherence to
HIV protease inhibitors. Ann Intern Med. 2001;134(10):968–77.
Int J Clin Pharm
123
79. Friedman RH, Kazis LE, Jette A, Smith MB, Stollerman J,
Torgerson J, et al. A telecommunications system for monitoring
and counseling patients with hypertension. Impact on medication
adherence and blood pressure control. Am J Hypertens. 1996; 9(4
Pt 1):285–92.
80. Volume CI, Farris KB, Kassam R, Cox CE, Cave A. Pharma-
ceutical care research and education project: patient outcomes.
J Am Pharm Assoc (Wash). 2001;41(3):411–20.
81. Gallagher P, Barry P, O’Mahony D. Inappropriate prescribing in
the elderly. J Clin Pharm Ther. 2007;32(2):113–21.
82. Steinman MA, Rosenthal GE, Landefeld CS, Bertenthal D, Sen S,
Kaboli PJ. Conflicts and concordance between measures of
medication prescribing quality. Med Care. 2007;45(1):95–9.
83. Carollo A, Rieutord A, Launay-Vacher V. European Society of
Clinical Pharmacy (ESCP) glossary of scientific terms: a tool for
standardizing scientific jargon. Int J Clin Pharm. 2012;34(2):263–8.
84. Pintor-Marmol A, Baena MI, Fajardo PC, Sabater-Hernandez D,
Saez-Benito L, Garcia-Cardenas MV, et al. Terms used in patient
safety related to medication: a literature review. Pharmacoepi-
demiol Drug Saf. 2012;21(8):799–809.
85. Shultz M. Comparing test searches in PubMed and Google
Scholar. J Med Libr Assoc. 2007;95(4):442–5.
Int J Clin Pharm
123
