Research Designs

Full text

Dr Jay Prasad, Dr Har Ashish Jindal
11
Chapter
 Introduction
 Characteristics of experimental research designs
 Threats to internal validity
 Threats to external validity
 Classicationofexperimentalresearchdesigns
 Advantages and disadvantages of experimental research designs
 Summary table of pre-experimental, quasi-experimental and true experimental research designs
 Summary and key concepts
 Assess yourself
Chapter Outline
At the completion of the chapter, the readers should be able to:
 understand concepts of research designs.
 acquire knowledge of different types of experimental research designs.
 discuss the difference between quasi-experimental and true experimental design.
 appraise relevance of internal and external validity of experimental research designs.
 devise an appropriate research design in conducting research.
Learning Objectives

2
Nursing Research in 21st Century
Secon V: Quantave Research Approaches and Designs
 Attrition
 Control
 Cross over
 Differential selection,
 Experimental
 External validity
 Instrumentation bias
 Interaction effect of testing,
 Internal validity
 Non equivalent control group design
 One group pre test post test design
 One shot case study
 Post test
 Pre experimental design
 Pre-test
 Regression effects
 Research Design
 Static group comparison study
 Testing effects
 Time series design
Key Terms
INTRODUCTION
An experimental research design is like a blueprint of the
procedures involved in answering a research question. It
is a detailed plan for conducting a scientific investigation.
A research design is defined as the series of those steps
taken to ensure that relevant data will be collected in a way
that permits objective analysis of the different hypotheses
formulated with respect to research problems. Researchers
conceive a research design for any investigation not merely
on premise of convenience or notion; Rather they consider
the specific purpose of research, the types of variables
involved and the research settings to evolve a research
design.1 However, any research design is considered fine
if it fulfils two basic functions:
Answers the research questions objectively, validly
and economically.
Enables the researcher to control undesirable variance
(extraneous or error variance) while conducting
experiment.
There are various methods and criteria’s to classify
research designs. However, based upon the criteria of
experiment, we may classify designs into two broad
categories: Non experimental and experimental designs.
Non- experimental designs involve the study of given
phenomenon, its description without any kind of
intervention or manipulation. Experimental designs
involve use of some form of experiment or intervention.
The current chapter is going to focus on various types
of experimental research designs
CHARACTERISTICS OF EXPERIMENTAL
RESEARCH DESIGNS
The experimental research designs involve manipulation
of certain stimuli, or treatments or environmental
conditions to make an observation on how such
manipulations affect the behaviour of the subject. While
doing such manipulation the researcher must be aware
of other factors which can affect the outcome, which he/
she either remove or control. Thus, the four basic essential
characteristics of an experimental research design can be
explained as:
a. Control: Those variables which are not of interest
to researcher and they can still affect the result also
called the extraneous variables are either removed
or the arrangements are made to minimize their
effects. These arrangements can be either the random
assignment of subjects to the groups, matching the
subjects on extraneous variables and at times keeping
groups homogenous or using some form of statistical
technique.
b. Manipulation: In an experimental research,
manipulation refers to a deliberate introduction or
operation of an independent variable on the subjects
in experimental group. This operated independent
variable is also called the treatment variable or an
experimental variable.
c. Observation: After introducing an independent
variable, the researcher observes its effect on
dependent variable. This task is called observation in
experimental research.
d. Replication: It refers to conducting sub experiments
under a large experimental design. The researcher
can combine a number of experimental and control
groups in singe design.
The experimental design can be further classified
into three major subtypes: True experimental, quasi
experimental and pre experimental (Figure1). Before
giving description of different kinds of research designs
it seems necessary to have an understanding about the
relevance of internal and external validity in various
research designs. The concepts of internal and external

3
Part A: Nursing Research
| 11 | Experimental Research Designs
validity pertaining to research designs bear a great
significance in any research proposal. So, it would be
worthwhile to discuss importance of internal and external
validity in research designs. The detailed description of
each design along with its evaluation in terms of internal
and external validity shall be discussed in later section.
INTERNAL AND EXTERNAL VALIDITY IN
RESEARCH DESIGN2-4
Any experimental research is conducted to see the impact
of independent variable upon a dependent variable. Thus,
an experimental investigation purports to answer the
basic question: Did the experimental treatment make any
difference in dependent variable? Internal validity refers
to the fact if the researcher has enough evidence to assert,
whether or not, experimental treatment/condition has
made the difference. If a research design has high internal
validity, there are enough reasons to conclude or attribute
the change in dependent variable to the introduction of
independent variable in the form of any intervention or
treatment. On the contrary, a design with low internal
validity shall have poor evidence of assuming such
causality. There are various threats to internal validity of
a research design that might compromise researcher’s
confidence in saying that a relationship is there between
the independent and dependent variables. The factors
that can confound the effects of experimental variable if
not controlled by the experimental design act as source of
threat to designs’ internal validity.
Another important fact that depicts the strength of an
experimental design is its external validity. The external
validity refers to the degree to which the results of the
experiment can be generalized to the entire population.
Thus, it is also commonly referred as the generalizability.
The next section shall offer a detailed description on
factors affecting the internal and external validity of
research designs.
THREATS TO INTERNAL VALIDITY
Internal validity of a research design can be affected by the
following factors:
a. History: Many a times, the change in scores on
dependent variable before and after the intervention
are due to the occurrence of some unanticipated
event which occur while the experiment is in progress
and the event has a potential to affect the dependent
variable. We can take an example, if a pre experimental
study is conducted to assess the impact of structured
teaching programme on tobacco cessation in reducing
prevalence of smoking. If the researcher finds a
significant reduction in smoking prevalence after the
intervention. There is a possibility that this occurred
due to an historical event wherein one of the persons
suffering from lung cancer died. Thus, attribution of
smoking cessation to intervention would be erroneous
without taking into account the event that occurred
during the experiment.
b. Maturation: The term maturation depicts all
biological or the psychological processes which have
a potential to vary systematically with passage of
time. Therefore, what researcher labels as “effect” of
intervention could be rather one of such maturational
processes. For example, a researcher assesses the
effect of playing video game on cognitive processing
of children. In such an experiment, the children are
in their growing age wherein there is likelihood of
natural improvement in cognition. Hence, attributing
all improvement in cognition to experiment would be
erroneous.
c. Testing effects: A pre-test may sensitize participant
in unanticipated ways and their performance on
the post-test may be due to the pre-test, not to the
treatment, or, more likely, and interaction of the
pre-test and treatment. Considering an example, a
study is conducted to assess the impact of structural
teaching programme on hand hygiene practices
in nursing personnel. The improvement in hand
hygiene practices may have been due to repeated
administration of the same test. That is, the nurses
simply learned to provide the right answers rather
than truly achieving improved hand hygiene practices.
d. Instrumentation bias: This threat which is also
referred as “instrument decay” occurs when the
changes in the outcome are due to the change in an
instrument used for the assessment. This can be
avoided by calibrating the instruments prior to the
data collection, so that they measure in the same way,
and to the same degree of accuracy.
e. Changes related to measurements: This bias may
also result when two examiners administer the same
test and are not trained adequately prior to the data
collection regarding certain measurements and there
are no written guidelines regarding the procedure.
Let us see an example, a study is conducted to
assess the cognitive impairment associated with
Electroconvulsive therapy. While first examiner duly
explains each test item to the subject satisfaction while
the other does not. This difference in the observer’s
approach to assessment can bring about a change in
the outcome. This could be controlled by training all
the data collectors, and testing them until they achieve
the inter rater reliability to a set standard.
f. Regression Effects: The regression effect is the
statistical effect of a propensity of regressing towards
the mean. The regression effects occur when the
subjects are recruited to an experiment on the basis

4
Nursing Research in 21st Century
Secon V: Quantave Research Approaches and Designs
of their extreme scores. This means the subjects with
extremely low or extremely high scores on study
variable are selected. When repeated testing is done
on such subjects, they reflect tendency to move closer
to the mean (i.e., regress). For example, if a group
of subjects was recruited on the basis of extremely
high anxiety scores and a psychological intervention
(progressive muscular relaxation) was done, any
post intervention improvement noted could be due
partly, if not entirely, to regression rather than to the
relaxation technique taught.
g. Differential Selection: This threat involves bias
resulting from differential selection when assigning
participants to groups. If the groups are different at
the outset or in the beginning of experiment, then they
are most likely to yield different outcomes following
the intervention/treatment to one of the groups,
regardless whether or not the treatment has any
effect. While randomization is a measure to eliminate
such threat, the design in which randomization is
not possible, differential selection bias weakens
the internal validity. For example, if a researcher is
conducting a study on effect of “structural teaching
programme” on attitude towards alcoholics. While
the experimental group involve caregivers of patients
undergoing de addiction treatment and control group
involves subjects of nearby residential area. In this
case, the two groups may be non - equivalent with
respect to their personal experience of staying with an
alcoholic as well as many other variables which can
influence their attitude.
h. Experimental Mortality/attrition: The experimental
mortality also called attrition bias which basically
refers to the loss or the dropout of subjects. There can
be multiple reasons for such loss or dropout including
death, lack of willingness to participate in follow ups,
non- availability owing to geographical migration or
at times the negative effect of treatment. In conditions
where the dropout rates are different in two groups, the
groups lose their compatibility at the time of post-test,
giving faulty results. For example, a researcher plans
a study to see impact of two specific diet regimens
on weight loss. While the first regime is very intense
and hard specific diet regimens on weight loss. While
the first regime is very intense and hard to follow, the
second one little flexible and easy to comply. There is
a high likelihood of greater dropout from first group,
especially of subjects with less mental hardiness
and higher weight. Thus, the two groups will not be
equivalent to be compared at end.
i. Selection- Maturation interaction: This effect
occurs when the rate of maturation/growth from
pre-test to post-test are different in experimental and
control group. Since the rate of maturation varies in
two groups, hence, it would not precise to attribute
change in outcome solely as an impact of treatment/
intervention.
Threats to external validity: Threats to external
validity compromise of the confidence in stating whether
the study’s results are applicable to other groups.
The common factors which affect the external validity of a
research design include the following2-4:
a. Reactive or interaction effect of testing: The term
interaction depicts that the effect is not merely due
to treatment rather an interaction between treatment
and pretesting. This effect occurs when the pre-
testing itself sensitizes or changes the behaviour of
experimental group. A pre-test might increase or
decrease the sensitivity and response of the subjects
with respect to the experimental variable. Thus, the
results drawn are no more representative of the entire
population which is un-pretested unlike the study
population. Let’s see an example, a study is conducted
to assess the effectiveness of ‘Structural teaching
programme’ on knowledge about infection control
in nursing personnel. The baseline knowledge is
assessed at the beginning followed by the intervention
and a post-test assessment is done at the end. In such
experiment, the post-test responses are likely to be
affected by exposure to pre-test itself independent of
the effect of intervention which will affect the external
validity of research design.
b. Interaction effect of selection biases and the
experimental variable: This is a compounded effect
resulting from the interaction between the selection
bias and the treatment. This means that some selection
factor interacts with the treatment which would not
have happened if the group was randomly selected.
Let’s see this effect in an example: suppose a study
is being conducted on effectiveness of ‘Assertiveness
training’ on nursing personnel. The researcher has
chosen a group of nurses working in a tertiary care
hospital. In this instance, the personal characteristics
of selected nurses including their knowledge, skill and
involvement in care are likely to make the treatment
more effective which cannot be generalized to all
nurses. Since the findings are not generalizable
across various settings, thus, there is threat to external
validity.
c. Reactive effects of experimental arrangements:
The subjects tend to behave differently just because
they are selected for a study. The experimental
arrangement of research makes it difficult to generalize
the results to non-experimental settings. Let’s see an
example; a study is being done on driving behaviour
of residents of a community. There is a chance that
subjects chosen might behave differently as they know

5
Part A: Nursing Research
| 11 | Experimental Research Designs
that all their driving related behaviour is under direct
surveillance. The “Hawthorne effect” is also included
in this which refers to change in behaviour of an
individual due to the attention they are receiving from
researchers rather than because of any manipulation
of independent variables.
d. Multiple-treatment interference: This effect is
evident in conditions when subjects are receiving
multiple treatments and it is at times difficult and
at other times impossible to eliminate the effect of
previous treatment when switching to new treatment
and thus a carryover effect persists. This makes
it difficult to generalize the outcomes to a single
treatment. Considering an example of a depressed
client who was given sessions of mindfulness therapy
which did not show any apparent results thus he
was started on another alternative therapy following
which the client begins to show improved mood.
Now it could be possible that the effect of first therapy
could be prodromal while the second therapy was
started. The researcher may erroneously attribute all
improvement in mood to second therapy which is in
true sense a combined effect of both therapies.
CLASSIFICATION OF RESEARCH DESIGNS
Fig. 1: Classicaon of Experimental Research Designs
After getting familiar with what a research design is
and what are the possible threats to external and internal
validity of given research design, let us see the detailed
explanation on various research designs
While studying the various research designs, the readers are
expected to be familiar with some symbols originally given by
Campbell & Stanley (1963)
R: Random selection of subject or random assignment of
treatment to experimental groups
X: Treatment or the experimental variable or the independent
variable. More than one treatment condition is labelled as X1, X2,
X3 and so on.
O: Observation: O1 ,O2,O3 represents subsequent observations.
I. PRE EXPERIMENTAL DESIGN
The pre experimental designs are considered as most
simplest and basic among all research designs. The suffix
‘experimental’ affirms to the fact that the experimenter
manipulates an independent variable to see its impact
upon a dependent variable. However, the prefix ‘pre’
affirms the fact that these designs fail to include a control
group.1
The pre experimental designs can be further
classified into three distinct types:
i. The One-Shot Case Study
ii. One Group Pretest - Posttest Study.
iii. The Static Group Comparison Study.
Each of these types is explained as follows:
i. One-Shot Case Study: This form of design involves a
study of single group of subjects at a single occasion. As
depicted in the figure 2, One-Shot Case Study design
has two notations, X & O, which means that there is
a single intervention followed by single observation.
The assessment of a dependent variable is made
subsequent to exposure to some form of treatment.
The treatment used is presumed to be the cause for the
change in dependent variable. However, this is done
in absence of any control group. A single instance is
carefully compared to general expectations of what
the case would have looked like had the treatment not
occurred and to other events casually observed. There
is very high likelihood of misplaced precision. This
research design has some inherent limitations. Firstly,
absence of a comparison group makes it impossible
to determine if the outcome scores are any higher
than they would have been without the treatment.
Secondly, without any pre-test scores, it is impossible
to determine if any change within the group itself has
taken place. These factors limit the use of one shot case
studies.3 The simplest example could be the subjects
undergo a training to improve emotional intelligence
following which a test of emotional intelligence is
performed on same subjects. The one shot case study
is diagrammatically depicted in Figure 2.

6
Nursing Research in 21st Century
Secon V: Quantave Research Approaches and Designs
X O
time
Fig. 2: One shot case study
Evaluating the design in terms of threats to internal
validity, this design has major weakness owing to history,
maturation, selection and mortality. In terms of external
validity, there is threat of interaction of selection and
treatment.
ii. One Group Pre-test Post-test Study: This form
of research design involves the assessment on
dependent variable before and after the treatment.
However, like one shot case study, this is done in
absence of any control group. The design is superior
to the One-Shot Case Study as this design enables the
comparison of the dependent variable(s) before and
after treatment. The One Group Pre-test Post-test
Study is diagrammatically depicted in Figure 3. In this
figure O1 is the pre testing or assessment at baseline
which is followed by treatment over time depicted as
X which is further followed by another observation O2
i.e. post test assessment.
X O2
O1
time
Fig. 3: One Group Pre-test Post-test Study
An example would be, a study conducted to assess
effect of Emotional intelligence training, for which the
researcher first assess the baseline EI, followed by training,
which is followed by retesting of emotional intelligence.
Owing to absence of control group, the design cannot
rule out the plausible rival hypotheses (explanations) for
differences between pre- and post-test (e.g., maturation).
This very fact explains various threats to the internal
validity of the design including history, maturation, testing,
instrumentation as well as selection and maturation
etc. The threats posed to the external validity are the
interaction of testing and treatment and the interaction of
selection and treatment.
iii. The Static Group Comparison Study: This design
takes two groups into account. One group is exposed
to a treatment and the results are tested while a control
group is not exposed to the treatment and similarly
tested in order to compare the effects of treatment.
The experimenter compares the two groups on
dependent variable for the purpose of establishing
the effect of experimental treatment. The design is
diagrammatically depicted in Figure 4.
Experimental group X O1
Control group No treatment O2
time
Fig. 4: The Stac Group Comparison Study
As depicted in Figure 4, the experimental group
receives a treatment ‘X’ while control group does not
receive any treatment. However, both groups are similarly
tested afterwards. Considering an example, a study is
done to assess the effectiveness of expressive emotional
writing on stress levels. To conduct this study, the two
groups are chosen. The first group undergoes the sessions
of emotional writing while second does not undergo any
such sessions. To assess the effectiveness of Emotional
writing, both groups are similarly assessed on stress at
the end. The inclusion of one control group removes
the threats of internal validity in terms of history, testing
and instrumentation. The threat related to the selection,
mortality and interaction of selection and maturation still
persists.
II. QUASI-EXPERIMENTAL DESIGN
Like true experimental design, the term experimental
in this design validates the fact that quasi experimental
designs involve the manipulation of independent variable.
However the prefix ‘quasi’ which means ‘false’ reflects the
fact that these experiments lack the key feature of random
assignment. Moreover, the researcher lacks full control
over the experimental conditions. The most common of
all the quasi experimental designs includes the following5
i. Non–equivalent (Nonrandomized) control group
design:
The non-equivalent control group design is a
very commonly employed design. It comprises of an
experimental group and a control group. The experimental
group receives treatment and control group do not receive
any treatment. The assignment to groups is not through
the process of randomization; rather, the researcher use
the groups that s/he thinks are similar as the treatment
and control groups. For example choosing two classes, two
similar schools or may be two similar villages. Although
the researcher tries to choose the groups as similar as
possible so as to have a fair comparison yet, the absolute
comparability is not guaranteed. There is a likelihood
of non- equivalence of groups and thus the term non-
equivalent control group design. The diagrammatical
depiction of non-equivalent control group design is shown
in Figure5.

7
Part A: Nursing Research
| 11 | Experimental Research Designs
Experimental group O1 X O2
Control group O3 O4
time
Fig. 5: The non-equivalent control group design
Let’s consider an example: A study to assess effect
of dance therapy on stress levels of nursing students.
The researcher chooses the students of first year in
experimental group and the students of second year in
control group. In such an experimental condition, there is
a chance that owing to new environment, homesickness
and other interpersonal factors, the first year students may
have greater levels of stress. This difference in the baseline
stress levels can largely affect the outcome
Evaluating the design in terms of threat to internal
and external validity, the design has a selection bias. For
instance, if a researcher is interested in assessing the impact
of planned peer interaction among first year students
in a college on the adjustment to new environment.
Such planned interaction is more likely to be successful
in students who are good at expression, have adequate
communication skills and are high on openness. There is
a possibility that the researcher chooses such subjects in
experimental group, while the other subjects who are poor
in such skills end up being in control group.
The second important threat in this design is the
selection –maturation interaction. Many a times, the
groups selected are such that maturation may proceed at
a faster rate in that group than the other. We might expect,
for instance, that person’s abstraction ability improves
with age (maturation) and that more intelligent person
will display problem solving skills more quickly and
easily. If the treatment group has subjects proportionately
with high I.Q. the group differences are likely to emerge
on such differential rates of maturation rather than any
intervention planned to improve abstract thinking. Such
an effect can be referred to as a selection–maturation
interaction. The history also poses an important threat in
this type of design.4
ii. Time series Design
These designs are also referred as the trend designs
in which repeated observations are made on dependent
variable before and after the treatment on one group
of subjects.7 The design is used to assess the change
in behaviour over a large scale. For example if an
organisation launches a new policy of annual appraisal in
terms of number of educational seminars attended by an
employee. Now, if we are interested to know if such launch
would bring about any change in employees behaviour
towards attending such programmes, we may effectively
utilize such design. For this purpose, we might obtain
retrospective information on number of educational
programmes attended in previous year, and then compare
it with the similar data in year following launch of such
programme. The Time series design is diagrammatically
depicted in Figure 6. As shown in figure 6, the group
undergoes periodic observations before and after the
intervention.
O1 O2 O3 X O4 O5 O6
Time
Fig. 6: The me series design
The time series design effectively rules out the threat
of maturation, testing and regression. Maturation is ruled
out as it is imperative to assume that the maturational
changes are gradual, not abrupt and sporadic. Secondly, if
there are any testing effects, they would be evident in the
initial set of observations which are made before the true
intervention is performed .Thirdly, like the testing effects
the regression to mean effect shall also come into sight in
initial set of observations9
The time series non-equivalent control group
design: The researcher also has an option of keeping
a control group concurrent to an experimental group,
thus employing a Time Series with Non-equivalent
Control Group Design. Since the control group is not
chosen through random technique, hence, the term
non- equivalent is employed. A Time Series with non-
equivalent Control Group Design is diagrammatically
depicted in figure 7.
Experimental group O1 O2 O3 X 04 O5 O6
Control group O1 O2 O3 04 O5 O6
Time
Fig. 7: A Time Series with non- equivalent Control Group Design
Let us see this design in line of an example, Lai et
al (2014)10 conducted a study to see the effectiveness of
a ‘Facebook-assisted teaching method’ on knowledge
and attitudes about ‘cervical cancer prevention and HPV
vaccination intention’ among female adolescent students.
The quasi experimental time series deign was employed in
the investigation. The researcher did purposive sampling
of a control group and an experimental group followed
by pre testing three times T1, T2 & T3 over a period of
time. Thereafter, both the groups were delivered lecture
on cancer prevention. The experimental group received
6 hour discussion sessions which were face book assisted
while control group had in person discussions after class.
The post test 1 was performed at T1, 2 weeks after T0, and
post test 2 was performed at T2, 8 weeks after T0. Time

8
Nursing Research in 21st Century
Secon V: Quantave Research Approaches and Designs
series is the only design to furnish a continuous record of
fluctuations in the experimental variables over the entire
course of the program.
The time series with multiple institution of treatment,
time series with intensified treatment, and time series with
withdrawn and reinstituted treatment (Fig.8 )11 are another
types of times series design,
O1 O2 XO3 O4 XO5 O6 XO7 O8
O1 O2 XO3 O4 X+1 O5 O6 X+2 O7 O8
O1 O2 XO3 O4 (- X) O5 O6 XO7 O8
Fig: 8: Time series with mulple instuon of treatment
In the figure 8, the first row is representing a time
series design with multiple institution of treatment. Lets
us see an example a researcher is interested in observing
the impact of an educational programme(X) for nurses
on patient satisfaction. Thus educational sessions are
provided at specified interval while patient satisfaction
(O1-O8) is assessed at intervals over a period.
In second row of figure 8, time series with intensified
treatment is depicted. For instance, a researcher intends
to see the impact of a new drug in diabetes on glycemic
control. In such scenario the two readings of Hba1c at
taken before initiating a drug, then two readings after
initiating a drug. Now the researcher increase the dose
and further makes two set of observations and thereafter a
further dose is increased and observations are made.
In third row, figure 8, time series with withdrawn and
reistituted treatment. Lets us understand with an example.
A researcher is interested to see the impact of a mobile
based personal coach application on weight reduction.
Thus two observations are made prior to the intervention,
then such phone based application is introduced
followed by two observations, Thereafter the phone based
application is withdrawn and further two observations
are made. And lastly the intervention is re introduced and
observation are made post intervention.
Advantages of Quasi Experimental Research Design
Most practical and feasible research design especially
in nursing.
Less time consuming.
Resources needed for experimentation are also
reduced.
Enable to compare with other groups.
Disadvantages of Quasi Experimental Research Design
It provides comparatively weaker evidence of effect of
an intervention.
Unexpected factors might affect the results.
TRUE EXPERIMENTAL DESIGNS1,5
True experimental design is regarded as the most
accurate and strongest form of experimental research.
The true experiment pose a strong control over the
known confounding variables, hence, they provide more
reliable evidence about the causes and effects. A True
experimental design is named so by the virtue of three
important properties:
1. Manipulation: The basic aim of conducting an
experiment is to see the effect of an independent
variable upon the dependent variable. This is possible
by the manipulation. The term manipulation in
context of experiment basically refers to doing
something to some of the subjects while withholding
the same in others. This ‘something’ which is done
to some of the subjects can be a special intervention
or treatment is the independent variable. Thus, an
independent variable is consciously manipulated
by the researcher to see its effect upon dependent
variable. For example if we are interested in evaluating
the impact of meditation in performance in a task.
Now, to find answer to this problem, the researcher
shall make a group of subjects perform meditation
before task while others shall not. In this situation,
the meditation acts as an independent variable while
performance is a dependent variable. Thus, the
researcher manipulates the independent variable to
test the hypothesis by comparing the performance of
subjects who did meditation to those who did not.
2. Control: Achieving a control is second important
characteristic of true experimental design. The
experimenter exercise control over various extraneous
variables. The randomization and manipulation
are some of the ways to control such variables. The
other way of control is the use of a control group.
The term control group refers to a group of subjects
whose performance on dependent variable is used to
evaluate the performance of the experimental group.
For example, if we are interested to see the effect of 30
minute aerobic exercise on weight. Now, if we have
a single group of subjects who take a specific diet for
one month period and also show reduction in their
weight at the end of one month. However, based on
the weight reduction seen in group, can we conclude
that specific diet has caused reduction in weight. There
might be a possibility that the subjects were following
a special exercise regimen as well. Hence, keeping a
control group who is similar to experimental group in
all aspects except that they do not take the specific diet
becomes necessary to conclude the results.
3. Random Assignment12-14: Random assignment is
“the process of assigning individuals at random to
different groups in an experiment”. The purpose of
random assignment is to insure that groups receiving
different treatments are as reasonably equal or similar
in any way that could possibly impact the outcome
(or dependent variable). This would mean that the

9
Part A: Nursing Research
| 11 | Experimental Research Designs
characteristics of the subjects which can have a direct
or indirect impact upon the outcome of study are
evenly distributed in two groups.
The true experimental designs can be employed in
various ways. Some of the most common true experimental
designs are being discussed in forthcoming section:
Types of True Experimental Research Designs
i. Pre test- Post test Control group Design
This design is considered to be the most effective
design in testing a cause effect relationship. The design
comprise of two groups which are equivalent as they have
been chosen through randomization. One of the groups
that receive the treatment is the experimental group while
the other which does not receive any treatment is the
control group. The diagrammatic depiction of design is
shown in Figure 8.
Fig. 8: Pre test- Post test Control group Design
As depicted above, the pre- test and post test is given to
both the groups. Such design provides a simple way to test
the effectiveness of treatment on the dependent variable
which can be analysed by making a comparison of gain
scores in experimental and control group respectively
i.e. comparing O2- O1 (mean gain score in experimental
group) with O4-O3 (mean gain score in control group).
Evaluating this design in terms of external and internal
validity, the design controls for most of the threats. The
threat for history is controlled as if, any historical event has
affected the outcome, it is likely to produce similar effects
in the control group as well provided both the groups are
assessed simultaneously. The threat of maturation and
testing are also controlled as they manifest equally in both
groups. The threat of regression is too controlled because
the subjects in experimental and the control group have
been randomly picked up from same population. So if any
regression to mean occurs, it is likely to occur in both groups
irrespective of the treatment received by the group. While
the group controls for most of the sources causing threats
to internal validity except the testing effect. The testing
effect is the gain in post test scores due to the experience
of undergoing pre- test itself. Secondly, the researcher
has no control over the sensitization to treatment i.e. the
interaction of testing and treatment is not controlled.
Let us consider the Pre test post test Control group
Design in an example: A study is conducted a study to
assess Effectiveness of Psycho educational programme
in reducing the caregiver’s burden. To conduct the
experiment participants are randomly allocated to two
groups: experiment, 25 patients; and control, 25 patients.
All the study subjects are pre tested using Zerit Burden
Interview Schedule. While the experimental group shall
get Psycho-education Programme (4 sessions) while
control would not receive any treatment. In this particular
example, the author would manipulate the independent
variable ‘Psycho-education Sessions’ on one group to see
its effect on dependent variable ‘Caregiver’s burden’. At the
end of the experiment, both the groups will be tested again
on the ‘Caregiver’s burden.’15
Evaluating this design in terms of internal validity,
the design controls for most of the threats. The threat for
history is controlled as if any historical event has affected
the outcome, it is likely to produce similar effects in
the control group as well provided both the groups are
assessed simultaneously. The threat of maturation and
testing are also controlled as they manifest equally in both
groups. The threat of regression is too controlled because
the subjects in experimental and the control group have
been randomly picked up from same population. So if
any regression to mean occurs, it is likely to occur in both
groups irrespective of the treatment received by the group.
However, there remain threats to the external validity
in this form of design. There is a threat for interaction
of testing and treatment, interaction of selection and
treatment as well as reactive arrangements.2
ii. Post-test only control group Design
This is a two group design including an experimental
group and a control group, where assignment to groups
occurs through randomization. The experimental group
is exposed to the manipulation; the control group is not.
The assessment on dependent variable is made post
intervention only, as the design does not use a pre-test
measure to establish a baseline.16 This design can be
diagrammatically depicted as shown in Figure 9. Figure 9
clearly depicts that subjects are assigned to either of the
two groups i.e. experimental or control groups through
random assignment. The experimental group receives
a treatment ‘X’ followed by post- test (O1). The control
group does not receive any treatment while post test is
conducted (O2).
Fig. 9: Post-test only control group Design
Let us consider the post test only group design in
an example: Batra (2014) conducted a study entitled
‘Application of ice cube’ prior to subcutaneous injection of
heparin on ‘pain perception and ecchymosis’ of patients
with cardiovascular problems.’ The independent variable

10
Nursing Research in 21st Century
Secon V: Quantave Research Approaches and Designs
for the study was the ice cube application for 3 min and
the dependent variables were pain perception and
ecchymosis. To answer the research question of whether
application of ice cube prior to subcutaneous injection
of heparin shall bring any change in pain perception
and ecchymosis, a total of 60 subjects were randomized
to experimental and control groups. The researcher
manipulated the independent variable in experimental
group while assessment on pain score and ecchymosis
was performed on both groups.17
Evaluating this design in terms of internal and
external validity, since there is no pre testing so this design
eliminates the threats associated with pretesting effect and
pre-test-manipulation interaction bias. This type of design
also minimizes the experimental mortality by reducing the
overall duration of study.
iii. Solomon Four group Design18
Also referred as the hybrid experimental design, this
design makes use of four groups. It is a combination of
standard pre-test-post-test two-group design and the post
test only controls design. To have a better understanding,
consider the diagrammatic depiction of the Solomon four
group design as shown in figure 10.
Fig. 10: Solomon four group design
As depicted in Figure 10, the random assignment is
used to make four separate groups. The group 1 undergoes
a pre-test followed by treatment, further followed by a post
test. Group 2 is a control group that undergoes both pre-
test as well as post, but there is no treatment. The group
3 is also an experimental group that receives a treatment
followed by post -test only. The group 4 is a control group
that has post- test only, in absence of either pre -test or
treatment. This particular design enables the researcher to
make multiple comparisons. The researcher can measure
the direct effect of treatment by comparing group 1 and
group 2. The comparison of post test results of group 1
with the post test results of group 3 enables to researcher
to conclude the testing effects i.e. the condition where
the act of taking a test itself affects the performance on a
retest or post-test. The comparison of pre- test post- test
scores of group 2 with that of group 4 helps to infer if there
are any external factors beyond treatment itself that are
responsible for change in post test scores. Lastly, if the
difference between the post-test results of Group 3 and
Group 4 is different from the group 1 and group 2, then
the researcher can presume that the pre-test has had some
effect upon the results.19-20
Let us consider an example : Chang et al (2014)
conducted a RCT to assess the effect of an asynchronous
e-learning curriculum on the knowledge level of medical
residents.16 The Solomon four group deign was employed
for the same. Block randomization was done to assign
each participant to either of the four groups. The Group
A received the pre test followed by access to the e learning
modules which was followed by the post test. While group
B was pre tested at beginning of rotation and post tested
at end without receiving any access to e learning module.
The group C was given access to e learning and post test
was done, however, pretesting was not done. Lastly, group
D was post tested only, there was neither pre-test nor
intervention. The results of the study revealed no pre-test
sensitization while use of e learning modules improved
the knowledge significantly.21
Despite of strengths in terms of internal and internal
validity, the Solomon four group design is not often
used owing to requirement of large sample, along with
the difficulty faced by researcher in introducing all the
treatments to four groups simultaneously. The researchers
also find it cumbersome to perform randomization in four
groups and lastly the burden of performing the intricate
statistical analysis. 22
iv. Factorial Design: There are conditions in which the
research question is not limited to see the effect of only
one independent variable. The researcher might be
interested in manipulating two or more independent
variables labelled as factors. Hence, a factorial
research design is utilized in such investigations
where researcher aims to assess the effect of two or
more independent variables upon a single dependent
variable. In conventional terms, the independent
variables are labelled as factors. The number of ways
in which a factor varies is labelled as the levels. An
experimental involving two independent variables
(factors) with two levels for each factor would be
represented as: 2X2 factorial design. The experiment
is planned in such a way that all levels of independent
variable are combined with all levels of another. This
enables the researcher in observing the direct effect
of independent variable on dependent variable as
well as the interaction effects of the various levels
of independent variable on dependent variable.
Lets consider an example: A study on the effects
of nicotine gum and counselling among African
American light smokers using a 2×2 factorial design.
In this particular example, the independent variables
included the use of 2mg nicotine gum and the use of
motivational interviewing and health education while
the dependent variable was 7 day quit rates. Here, there
are two factors i.e type of gum and type of counselling

11
Part A: Nursing Research
| 11 | Experimental Research Designs
and each factor has two levels. Thus factor refers to
the number of independent variables and term level
refers to the subdivision of a factor. So if it is a 3X 3
factorial design, this shall mean that there are three
independent variables and each independent variable
has further three levels. Such a factorial design shall
generate a total of 9 groups. (3X3=9)
Type of gum
Type of counselling
Health
Education
B1
Motivational
Interviewing
B2
2mg Nicotine gum
A1
A1B1 A1B2
Placebo gum
A2
A2B1 A2B2
Fig. 11: Factorial designs
The figure above depicts a 2x2 factorial design which
can help the researcher to test multiple hypotheses, by
manipulating more than one independent variable. In
this study the subjects shall be assigned to one of the four
arms:
a. The first group will receive 2mg nicotine gum and
health education
b. The second group will receive 2mg nicotine gum with
motivational interviewing.
c. The third group will receive placebo gum with health
education
d. The fourth group will receive placebo gum with
motivational interviewing.
The researcher shall test various research questions with
this design :
a. Is 2mg nicotine gum beneficial in tobacco cessation
than placebo gum
b. Is the health education with placebo gum effective
in tobacco cessation compared to 2mg nicotine gum
with heath education
c. Is motivational interviewing more effective than
health education in tobacco cessation
d. Is motivational counselling with 2mg nicotine gum
more beneficial than motivational counselling with
placebo gum.
Thus, the design shall test the main effect of using a
nicotine gum and placebo gum, the main effect of the type
of counselling used. The superiority of the design lies in
the fact that it shall assess the interaction effects of type
of gum and type of counselling used. An interaction effect
exists when differences on one factor depend on the level
you are on another factor. The factorial designs are highly
flexible designs with great utility in examining treatment
variations.
v. Randomised Block Design24-26 While conducting an
experiment, if the researcher feels that any known
variable is likely to produce some degree of variation in
the findings, then, in order to minimize that variance,
randomised block design can be employed. With a
randomized block design, the experimenter divides
subjects into subgroups called blocks, such that the
variability within blocks is less than the variability
between blocks. This implies that each block
introduced is a homogenous subgroup. The researcher
controls the variability by blocking. Thereafter
subjects within each block are randomly assigned to
all treatments. Each block contains a complete set of
treatments, therefore differences among blocks are not
due to treatments. This variability can be estimated as
a separate source of variation.
See this example
A study was carried out to evaluate the effect of
exercises on shoulder dysfunction and lymphedema
amongst the patients undergoing breast surgery. The
patients were randomized into three groups. Group I
received the intervention Presurgery; Group II received
the intervention postsurgery, and Group III served as
a control group. Block randomization was done. Each
block containing one A, B and C. Single sequence of block
randomization was used. Order of these was randomly
permuted (rearranged) so that all possible permutations
are created. Thus, in total we had 6 blocks.
1. “ABC
2. ACB
3. BCA
4. BAC
5. CAB
6. CBA”
A random number sequence was used to choose a
particular block, which sets the allocation order for the
first three subjects. Similarly, new number is selected-
corresponding block was allocated to the next three
subjects. The process was then repeated. From the
Random number Table a number was chosen (e.g. first no.
which we got was: 4-BAC, then 3- BCA, then 6- CBA and
so on…. till we reach at our targeted 330). Permuted block
randomization ensured intervention groups (A and B) and
control group (C) numbers are evenly balanced.
vi. Crossover designs 27
The crosses over designs are also commonly referred
as the repeated measures design. There is more than
one independent variable to be manipulated. However,
the subjects who are exposed to different treatment
conditions remain the same. Such designs are thus also
called the within subject designs. The most distinct feature
of crossover design is that each patient serves his/her own
control. This also ensures the equivalence among the

12
Nursing Research in 21st Century
Secon V: Quantave Research Approaches and Designs
groups exposed to treatment conditions. And hence, the
problems of comparability of experimental versus control
group with regard to various confounding variables are
resolved. The two terms used in cross over design are the
sequence and period. The sequence refers to the order
of treatment administration and the period is the time of
treatment administration. The design can be represented
diagrammatically as shown in Figure 12.
Fig. 12: Cross over design
Let us consider an example of crossover design:
Khalkhali et al (2014) conducted a study on effect of
applying ‘cold gel pack’ on the pain associated with ‘deep
breathing and coughing’ after open heart surgery. To
assess the effect of using ‘cold gel therapy’ on postoperative
pain associated with deep breathing a coughing, the
researcher used the cross over design ad recruited 50
subjects28. The subjects were divided into two groups
using random allocation. As depicted, the subjects in both
groups were pre assessed for pain. In group 1, the cold gel
application was done for 15 minutes following which the
deep breathing and coughing was performed and pain
assessment was done. While in group 2, the subjects were
prepared for deep breathing and coughing without any
cold gel application. Group 2 subjects were also assessed
for pain while deep breathing and coughing exercises
were performed. This was followed by a washout period
of 2 hours. After the washout period, there was a cross over
which means that in period 2 , the group 1 performed deep
breathing and coughing without cold gel application while
group 2 performed it with cold gel application , and pain
was assessment was done.
Thus, crossover designs are highly beneficial designs
provided utilized rationally. It is very essential that the
researcher ensures a washout period between two trials
being compared to rule out the carryover effect in such
design. The carryover effect refers to the impact of first
treatment condition while subject is being assessed for
the second. In many situations, it might be tedious to
test the assumption of carryover effects or it might not
be possible to determine the precise length of washout
period. Such situations limit the use of crossover designs.
Also mentioned the another limitation of crossover design
revealing the “period” effects in addition to carryover
effects viz. Progression of disease and dropouts.
Advantages of Experimental Research Design
Experimental research designs are the most powerful
methods to establish the causal relationship between
variables.
These designs provide the highest quality evidence
regarding the effects of specific interventions.
There is better extent of purity in observation as the
study is conducted under controlled environment.
Conditions that are not found in natural setting can
be created in experimental setting in a short period of
time that may take years to naturally occur (therefore
very useful in genetic studies).
Because the study is carried out in experimental
setting the problems of real life situations and the
personal problems of the researcher is eliminated.
The environment in which the research takes place
can often be carefully controlled. Thus, it becomes
easier to assess the true effect of the variable of interest
on the outcome of interest.
Disadvantages of Experimental Research Design
Though there are number of benefits of experimental
research design, it has certain limitations also. These
are
Sometimes, there could be a problem if the subjects
themselves have the discretion about participation in
the treatment.
If the experiment is conducted in the lab, and the
intervention is being carried out by the person other
than the researcher himself, then it may be difficult
to determine that if the subjects in the experimental
group have actually received the intervention.
In experimental studies conducted in natural settings
like a hospital or community, it may not be possible to
impose control over extraneous variables.
Experiments are often more impractical when the
effect of independent variable may require a lengthy
period of time before it can emerge as a response on
the criterion measures.
It is very difficult to obtain permission from the
participants.
Because the size of the sample is kept small especially
studies involving humans, the representativeness of
the findings of such study is questionable.
Experimental situation may not relate to the real
world.
It may be unethical or impossible to randomly assign
people to group e.g. deliberately depriving the children
off the sleep.
Summary table of pre experimental, quasi
experimental and true experimental research designs
After a detailed view on pre experimental, quasi
experimental and true experimental research designs, let
us have a brief summary table on unique characteristics of
each type of design.

13
Part A: Nursing Research
| 11 | Experimental Research Designs
Characteristics Pre experimental design Quasi experimental design True experimental design
1. Introduction of independent variable
(Manipulation)
2. Inclusion of a control group (Control)
3. Participants are randomly assigned
to either the treatment or the control
group.
(Random Assignment)
*Green indicates presence while red indicates absence.
SUMMARY AND KEY CONCEPTS
The chapter gave a detailed description on various types of research designs on the basis of type of experiment. The
chapter entailed upon the most basic forms of research beginning from the pre experimental to quasi experimental to
the true experimental designs. While each research design has its own inherent strengths and weaknesses, however,
choosing the right design for any investigation is largely affected by the nature of research question and the resources
available to conduct study. It is imperative to critically appraise a design opted for any study in terms of its external and
internal validity to get most precise results.

14
Nursing Research in 21st Century
Secon V: Quantave Research Approaches and Designs
Long and short answer type questions
1. Describe the classification of research designs. Discuss
in detail the true experimental research designs.
2. What are the various threats to external and internal
validity of a research design?
3. What are the characteristics of true experimental
research designs? Describe the cross over design with
appropriate example.
4. Define internal validity of research design.
5. Define external validity of research design.
6. Enlist the different types of quasi experimental types.
7. Describe briefly a randomised block design.
8. Explain a time series design.
Multiple choice questions
1. The evidence to assert, whether or not, experimental
treatment/condition has made the difference refers
to:
a. External validity
b. Internal Validity
c. Criterion validity
d. Reliability
2. The degree to which the results of the experiment can
be generalized to the entire population refers to:
a. External validity
b. Internal Validity
c. Criterion validity
d. Reliability
3. The designs in which repeated observations are made
on dependent variable before and after the treatment
on one group of subjects are referred as:
a. Time series design
b. The non- equivalent control group design
c. Pretest- Post test Control group Design
d. Post-test only control group Design
4. Which of the following is not a property of true
experimental design:
a. Manipulation
b. Control
c. Random Assignment
d. Stratification
5. Which of the following is not a property of Pretest-
Post test Control group Design :
a. The threat for history is controlled.
b. The threat of maturation and testing are also
controlled.
c. Threat of regression towards mean is controlled.
d. Threat of testing effects is controlled.
6. A 2X3 factorial design depicts that:
a. There are two independent variables and each
independent variable has further three levels.
b. There are three independent variables and each
independent variable has further two levels.
c. There are total five groups.
d. There are total two groups.
7. A research design in which each patient serves his/her
own control are commonly referred as
a. Cross over design
b. Factorial design
c. Non equivalent control group design
d. Solomon four group Design
8. The bias resulting from difference in the rate of
maturation/growth from pre-test to post-test are in
experimental and control group is called:
a. Selection- Maturation interaction
b. Maturation
c. History
d. Interaction effect of testing
9. The given icon represents which research design:
Experimental group O1 O2 O3 X 04 O5 O6
Control group O1 O2 O3 04 O5 O6
Time
a. Time Series with non- equivalent Control Group
Design
b. Factorial design
c. The time series design
d. The non- equivalent control group design
10. Which of the following are the threats to internal
validity of research designs:
a. Multiple-treatment interference
b. History
c. Maturation
d. Experimental mortality
Asse ss Your s elf
Ans.
1. b 2. a 3. a 4. d 5. d 6. a 7. a 8. a 9. a
10. a

15
Part A: Nursing Research
| 11 | Experimental Research Designs
1. Singh AK. Tests, Measurements and Research Methods in
Behavioural sciences. 5th edition .Patna: Bharti Bhavan,
2010.
2. Ohlund B & Yu Chong-Ho. Threats to validity of Research
Design. Cited [ 2015 March1].AvailableFromhttp://web.pdx.
edu/~stipakb/download/PA555/ResearchDesign.html
3. Campbell, D. & Stanley, J. (1963). Experimental and quasi-
experimental designs for research. Chicago, IL: Rand-
McNally.
4. Fife-Schaw C. Quasi experimental Designs 2012 Cited
[2015 March 4]Available from http://www.sagepub.in/upm-
data/46877_Breakwell_Ch04.pdf
5. Polit DF & Hunglar BP. Nursing research: Principles and
methods. 5th edition. Philadelphia: Lippincott. 1997
6. Yoo MS, Park JH. Effect of case-based learning on the
development of graduate nurses’ problem-solving ability.
Nurse Educ Today. 2014;34(1):47-51. doi: 10.1016/j.
nedt.2013.02.014
7. Gottman, JM, Mcfall RM & Barnett JT. Design and analysis of
research using Time series. Psychological Bulletin 1969, 72(
4) , 299-306
8. Lai CY, Wu WW, Tsai SY, Cehng SF , Lin KC & Liang SY. The
Effectiveness of a Facebook-Assisted Teaching Method
on Knowledge and Attitudes About Cervical Cancer
Prevention and HPV Vaccination Intention Among Female
Adolescent Students in Taiwan. Health Educ Behav.2014; pii:
1090198114558591. [Epub ahead of print]
9. PsychoMetrics. Threats to Internal Validity. Cited [2015
March 4]Available from http://www.psychmet.com/id12.
html
10. Nancy D & Miller- Albers. Posttest Only, Control Group. Cited
[2015 March 3] Available from http://facultyweb.berry.edu/
nmiller/classinfo/323/2.4/true_control_after_only.htm
11. Polit DF, Beck CT, Nursing Research: Generating and assessing
evidence for nursing research. Wolters Kluwer (India) Pvt.
Ltd., New Delhi. Eighth Edition. Page 267-271.
12. Creswell J. Educational research: Planning, conducting,
and evaluating quantitative and qualitative research. New
Jersey: Pearson: Merrill Prentice Hall. 2008
13. Slavin R. Educational research in an age of accountability.
Boston: Pearson Education. 2007
14. Characteristics of Experimental Research. Cited [2015 June
10]. Available from https://www.aea267.k12.ia.us/system/
assets/uploads/files/1468/experimental_research.pdf
15. Kaur M, An experimental study to assess the effect of
caregiver support interventions on the subjective burden
of caregivers of patients suffering from severe mental
illness attending psychiatry OPD and admitted in psychiatry
Ward. [M.Sc. Nursing Thesis]. National Institute of Nursing
Education :Post graduate Institute of Medical Education &
Research, Chandigarh,India;2015
16. Nancy D & Miller – Albers. Post test only control Group . Cited
[ 2015 March 3]Available from http://facultyweb.berry.edu/
nmiller/classinfo/323/2.4/true_control_after_only.htm
17. Batra G. Application of ice cube prior to subcutaneous
injection of heparin in pain perception and ecchymosis
of patients with cardiovascular problems. Nurs J India.
2014;105(4):155-9.
18. William MK. Hybrid Experimental Designs. Cited [March
2015]. Available from http://www.socialresearchmethods.
net/kb/exphybrd.php.
19. Mchae TW & Tingen MS. The Use Of The Solomon Four-
Group Design In Nursing Research. Southern online journal
of nursing research .9(1) http://www.resourcenter.net/
images/snrs/files/sojnr_articles2/Vol09Num01Art14.pdf
20. Lavanyakumari P. Significance of Solomon four group
pretest-posttest method in True Experimental Research- A
Study. IOSR-JAVS 2013; 5(2):51-58.
21. Chang et al. Paediatric Emergency medicine acynchronous
E- Learning. A multicenter randomized controlled
Solomon four group study. Academic Emergency
Medicine.2014;21(8);912-919.
22. Mcghaee TW and Tingen MS. The Use Of The Solomon Four-
Group Design In Nursing Research. SOJNR 2009; 9(1): 1-7.
Available from http://www.resourcenter.net/images/snrs/
files/sojnr_articles2/Vol09Num01Art14.htm
23. Alhuwalia JS. The effects of nicotine gum and counseling
among African American light smokers: a 2 x 2 factorial
design.Addiction. 2006 Jun;101(6):883-91
24. Randomised Block Design. Cited [April 2017]
Available at http://stattrek.com/statistics/dictionary.
aspx?definition=Randomized%20block%20design
25. Dallal GE. Randomized (Complete) Block Designs. Available
at http://www.jerrydallal.com/lhsp/ranblock.htm
26. Willia, MK. Randomized Block Designs. Available at https://
www.socialresearchmethods.net/kb/expblock.php
27. Blettner M & Welleck S. On the Proper Use of the Crossover
Design in Clinical Trials: Part 18 of a Series on Evaluation of
Scientific Publications. Dtsch Arztebl Int 2012; 109(15): 276-
81; DOI: 10.3238/arztebl.2012.0276
28. Khalkhali H R, Tanha, ZER, Feizi A & Ardabili SS. Effect of
applying cold gel pack on the pain associated with deep
breathing and coughing after open heart surgery. Iran J Nurs
Midwifery Res. 2014 Nov-Dec; 19(6): 545–549
Refe Ren c es