Content uploaded by Paul Jansen
Author content
All content in this area was uploaded by Paul Jansen on Dec 22, 2013
Content may be subject to copyright.
Multi-rater
feedback
methods
455
Journal of Managerial Psychology,
Vol. 14 No. 6, 1999, pp. 455-476.
#MCB University Press, 0268-3946
Multi-rater feedback methods:
personal and organizational
implications
Paul Jansen
Free University, Amsterdam, The Netherlands and
DanieÈl Vloeberghs
University of Antwerp, Antwerp, Belgium
Keywords Feedback, Assessment centres, Total Quality Management, Corporate communications
Abstract The conditions and implications for the use of multi-rater feedback or ``360-degree''
methods are studied from both a psychological and an organizational perspective. Psychological
research indicates that multi-ratings cannot be put on the same footing as ratings obtained from
more ``objective'' sources as for instance the assessment center. Multi-rater feedback can be
interpreted either as standing in the tradition of assessment centers, or originating from ``total
quality management'', but it seems more congenial to the TQM method of organizing ``customer
feedback''. Since multi-rater feedback presupposes social interaction and consequently a work
organization, the communicative conditions for such a feedback system are subsequently
investigated from the perspective of organization science. It is investigated how various structural
and cultural changes offer a fertile soil for the introduction of such kinds of feedback. However,
the authors warn of the terror of ``total feedback''.
Introduction
Increasingly, so-called ``multi-rater feedback methods'' are used in work
organizations. According to the guest editors of the special issue of Group &
Organization Management that was devoted to 360-degree feedback, these
feedback methods have become one of the most popular and least understood
management and organizational interventions in recent years (Church and
Bracken, 1997). The multi-rater feedback method typically consists of a
questionnaire containing instances of work behavior. Examples of such
``behavioral items'' are:
(1) He/she is someone who is open for questions.
(2) He/she tries new ideas first with his/her employees.
(3) He/she keeps firm in difficult matters.
(4) He/she tries to be as independent as possible.
(5) He/she has a large social network.
(6) He/she is easy to understand.
``Bystanders'' who are directly involved in the work of the employee who is to
be assessed (the so-called ``focal person''), are asked to give their observations of
this employee using a long (typically over 100 items) list of such situationally
phrased behaviors. So by a multi-rater feedback questionnaire, work behavior
The current issue and full text archive of this journal is available at
http://www.emerald-library.com
Journal of
Managerial
Psychology
14,6
456
is assessed on account of a multitude of work situations which are to be
controlled or managed by the person who is the focus of the feedback.
With this type of ``multi-source feedback'' (London and Smither, 1996) a
``circle'' of bystanders indicate the degree to which, in their view, specific
behaviors apply to the focal person. Instead of multi-rater feedback therefore
also the denotation ``360-degree feedback'' is used. The person whose behaviors
are assessed ± the midpoint (``focus'') of the circle of bystanders, him or herself ±
also fills out the list of work situatons.
Bystander ratings are averaged and compared with the self-ratings of the
person assessed. Self-ratings of the focal person are compared with the average
of what his or her social environment thinks of him/her. The focal person
should ``do something'' with the data thus obtained; multi-rater feedback is an
instrument for personal development and performance improvement. However,
London and Smither (1996) argue that multi-rater feedback has different effects
on the focal person. Generally, the following reactions can be observed:
.the employee completely neglects the feedback;
.the employee only takes positive feedback into account;
.the employee is only motivated by negative feedback;
.the employee is only interested in feedback which is given by someone
who is considered ``really important'' (e.g. his or her supervisor).
In order to counter such undesirable effects, multi-rater feedback sometimes is
combined with feedback from sources which have a more ``objective'' appeal,
such as an assessment center. Research indicates that feedback obtained in an
assessment center has a positive effect on job performance, provided that it is
detailed, behaviorally specific, and perceived to be useful (Engelbrecht and
Fischer, 1995; Francis-Smythe and Smith, 1997).
More generally, use of multi-rater feedback methods gives rise to the
following questions:
.What are the differences between multi-rater feedback and assessment
center methods? For, in the latter case the same ``focal'' person is also
rated by multiple assessors. Do assessment centers and multi-rater
feedback methods measure the same constructs or phenomena?
.On what kind of personal attributes is the focal person actually
assessed: dispositions or abilities or preferences or attitudes? That is, is
it possible to use multi-rater feedback to obtain information about the
focal person's mastery of ``competences'' (just as is the case in an
assessment center)?
.What to do when feedback ratings from bystanders are not consistent?
Research indicates that in practice inconsistent ratings are often found
(London and Smither, 1996, p. 809). In case of inconsistency, should the
``focal employee'' try to adapt his/her behavior in order to make one and
the same ``impression'' in all critical job situations, or should observation
Multi-rater
feedback
methods
457
of inconsistencies be considered an indication that the employee's
behavior is dominated by demands or constraints which are very
specific to the job situation?
.Does multi-rater feedback work? Stated diferently, what effect does
feedback have on the employee in this case?
.What is the impact of feedback on the commitment with collegues and
with the organization?
.What is the effect of repeated use of multi-rater feedback methods on
work interactions? For instance, do organizational communication
patterns become more open?
.What implications does a repeated, mutual and systematical
``bombardment'' of social feedback have for the organization as a context
of cooperation?
.What are the implications of multi-rater feedback for personnel
management?
These questions will be treated in this study. Initially, important themes from
recent research into multi-rater feedback will be discussed. It will appear that
published research has become more frequent in last years, but that it is
presently dominated by an individual-oriented, employee-centered perspective.
The organization in the context of feedback generally is neglected in multi-rater
feedback theory and applications. Use of multi-rater feedback in an actual
work-organisation, and its impact on patterns of cooperation and
communication, are almost totally neglected. Therefore, in the second part of
the paper, the ``wider circle'' that can be drawn around multi-rater feedback
systems and applications, will be discussed. We will discuss conditions and
effects of multi-rater methods at the organizational level, and the implied
redesign of patterns of organizational communication.
In sum, use of multi-rater feedback is increasing, but it is still used rather
uncritically and studied at a basic, ``individual-psychological'' level. It is our
intention to give a critical account, both using results from recent published
research, and stressing the wider organizational context. The aim of this study
is to obtain further understanding of the conditions and implications of multi-
rater feedback methods, both at the individual and at the organizational level.
We are interested in the proper place of this method both among other methods
for ``organizing feedback'' and in the context of organizational communication.
In particular, effects of multi-rater feedback at an organizational level are
hardly investigated. This means that we aim at conceptual integration: both of
various research findings (next section) and of various levels of research
(section on organizational conditions and effects). Beforehand we remarked
that, whereas empirical research at the individual-psychological level abounds,
it is scarce or almost absent at the organizational level. As a consequence, the
``tone'' of this study will change accordingly from a rather detailed discussion of
concrete research findings, to a more general conceptual reflection.
Journal of
Managerial
Psychology
14,6
458
Research into multi-rater feedback
Although on an informal basis multi-rater feedback is something which has
been around already for a long time (e.g. the employment of juries in the case of
a beauty contest, or in the case of the Eurovision Song Contest, or for the award
of a literature prize), the formal application, in the context of appraisal of work
behavior in organizations, is of a more recent date. While previously the
denotation ``multi-rater systems'' was used in the literature, the denotation ``360-
degree feedback'' was introduced by the American consultancy firm Teams Inc.
around 1985, and was also registered as a legal trade mark. Through the Wall
Street Journal,Fortune and Jack Welch (the method was generally
implemented with General Electric), the multi-rater feedback method had a
quick spread starting from 1994. At present, about 10 percent of American
organizations appears to use such a kind of personal feedback system
(Antonioni, 1996). Corresponding figures for Europe are lacking, but anecdotal
evidence suggests that use in countries such as the UK and The Netherlands is
increasing exponentially (in The Netherlands, conferences about this method
are very popular and two books have appeared this year). Apart from that,
introducing multi-rater feedback can also be considered a logical step in the
development of techniques which have been introduced already decades ago,
such as organizational surveys (mostly directly at organizational climate or
culture), assessment centers, client investigations stimulated by total quality
management (TQM), developmental feedback, and ``peer reviews''. Although at
first view multi-rater feedback seems to have something in common with all
these methods, we will argue in this study that its roots are more in the TQM-
approach and much less in the assessment center tradition.
Since multi-rater feedback is a relatively new method, studies into its content
and effects have only begun to appear in the last few years. Results of that
research can be arranged according to six major themes: relationship between
ratings from different external rating sources (e.g. from different bystanders),
relationship between self-rating and bystander-ratings, relationship between
the agreement between self-ratings and bystander-ratings on the one hand and
the focal person's job effectiveness on the other hand, development in self-
ratings on account of feedback obtained at different time-points, relationship
between multi-rater feedback and such other feedback methods as the
assessment center and TQM.
What is the relationship between the ratings in general?
According to the review study by Mabe and West (1982) intercorrelations
between ratings from different sources generally are low, that is: different types
of bystanders such as the supervising manager, colleagues, and customers,
only moderately agree with each other on their assessment of the same focal
person. There are a number of reasons for this low agreement.
First, raters who are natural bystanders generally are not trained in
assessment. Therefore they are apt to succumb to various natural rating errors
which are well-known from psychological literature (see, for example, Cooper,
Multi-rater
feedback
methods
459
1981; Murphy and Cleveland, 1995), such as the use of categories which are
irrelevant to the aim of the rating (e.g. gender), halo (the ``good guy''), horn (the
``bad guy''), recency (basing the assessment on recent behaviors only), primacy
(basing the assessment on the first behaviors that are observed), extremity
(giving only very extreme ratings: two examples are being too lenient or too
severe), and finally central tendency (giving only average ratings).
Second, raters are (too) committed to the focal person, and consequently do
not ``keep enough distance''. Their assessment is partly determined by
sympathy or antipathy, by irrelevant previous experiences, and by common or
conflicting interests. Bystanders are invited (often by the focal person him or
herself) to participate in multirater feedback on account of the work relation
they have with that focal person. This relation can be the cause of biased
ratings.
And third, raters only relate with the focal person from one perspective, and
accordingly rate him or her only with respect to one, specific aspect of behavior.
They are the supervisor of the focal person, or his or her customer, or employee,
or colleague. It is precisely the aim of the multi-rater approach to elicit, collect
and integrate the different bits of information which are obtained by persons
who take part in the various work roles the focal person fulfills. A clear
example of the influence of the ``structural role'' of the rater on his/her ratings is
given by the study of Salam et al. (1997) in which leaders were rated differently
by heir subordinates and their superior managers. Leaders who challenged the
status quo of the organization were rated lower by their superiors but higher by
their subordinates. Another example is the study by Church (1997a) in which
different raters appeared to have different ``perceptual frames of reference''.
What is the relationship between the focal person's ratings on the one hand, and
bystanders' ratings on the other hand, that is between self-rating and bystander-
ratings?
Essentially, this question pertains to the degree to which the focal person both
knows and accepts how his or her environment perceives and evaluates him or
her. According to the overview by Mabe and West (1982) this type of accuracy
of self-evaluations depends on three attributes of the focal person:
(1) (social) intelligence;
(2) internal locus of control; and
(3) performance in the past.
In short: first, the focal person should be able to imagine him or herself in his or
her bystanders (social intelligence). But second, he or she should also be
prepared to do this. This willingness to be open to feedback from bystanders is,
according to Mabe and West, reinforced by a past positive track record that, in
the view of the focal person, really can be attributed to his or her behavior
(internal locus of control).
Journal of
Managerial
Psychology
14,6
460
In this respect, three categories of focal persons can be distinguished (see, for
example, Fleenor et al., 1996):
(1) ``over-estimators'': self-ratings consequently are larger (more positive)
than bystanders' ratings;
(2) ``under-estimators'': self-ratings invariably are lower (more negative)
than bystanders' ratings;
(3) ``in-agreement'': self-ratings consequently are about equal to bystanders'
ratings.
Generally, overestimation is the case (Atwater and Yammarino, 1992). In fact,
over-estimation by focal persons with respect to bystander ratings is found so
frequently, that it seems plausible that over-estimation should be considered as
something ``normal''. Maciel et al. (1994, p. 82) conclude on the basis of a
literature review that ``normal as opposed to depressed individuals usually
over-estimate their capacities'', and that such kind of ``positive illusions''
accordingly should be considered ``creative self-deceptions of the healthy
mind''. So research indicates that persons generally are too optimistic about
their competences. The tendency to rate own achievements higher than the
average of the ``reference group'', that is of relevant bystanders, is denoted as
the ``better-than-average effect'' (Alicke et al., 1995). On the other hand, persons
who rate themselves consequently too low are much less frequent.
A possible explanation for the ``positively biased'' self-ratings in case of
multi-rater feedback, is the process of social comparison (Farh and Dobbins,
1989). With multi-rater feedback the focal person's competence profile is
assessed through relevant bystanders. Mostly it is even the focal person him or
herself who selects ``his or her'' bystanders. The latter have to appraise the focal
person as they experience him or her in the work role they have with him or her.
This means that raters in multi-rater feedback of necessity relate to the focal
person, thus ``have something to do with him or her''. But, multi-rater feedback
then becomes a situation of social judgment. When persons appraise other
persons in a social context of actual and committed interaction, ``social
comparison'' may take place.
According to social comparison theory initially formulated by Festinger
(1954), persons want to be sure about their individual and ``subjective'' opinions
and abilities. Since there does not exist some objective standard in this field, the
certainty is found in comparisons with ``relevant others'' (Wheeler, 1991;
Wrightsman, 1972). A premise of social comparison theory is that every social
comparison evokes competitive tendencies, that is it reinforces the tendency to
make a positive distinction between oneself and relevant others (Wills and Suls,
1991). Eventually, this causes personal over-estimation. An illustration of
competitive social comparison is given by the study of Salam et al. (1997)
discussed above, in which leaders who challenged the status quo of the
organization were rated lower by their superiors but higher by their
subordinates.
Multi-rater
feedback
methods
461
What is the relation between the degree of agreement of self-ratings and
bystander-ratings on the one hand, and the focal person's job effectiveness on
the other hand?
It appears that, generally, ``in-agreements'' are considered more effective by
bystanders than ``over-estimators'' (Atwater and Yammarino, 1992). Since, as was
discussed above on account of Mabe and West (1982), ``in-agreements'' generally
are more open to reactions of others, of their various role partners or ``customers'',
they are able to take these into account when acting and therefore are both more
empathetic and more sensitive. Defining ``self-awareness'' as the congruence
between self-ratings and bystander-ratings, Church (1997b) found that high-
performing managers were more self-aware than low-performing managers. It
subsequently appeared that self-awareness correlated with self-monitoring, that is
the ability to assess and adapt own behaviors vis-aÁ-vis the need of ``what the works
needs''. This corroborates that in-agreements are better able to assess what other
people think, are more accessible to feedback from others and more prone to use it
to improve own behavior (Ashford, 1984). Accurate self-raters are better able to
perceive both themselves and others with a high degree of accuracy, and to adjust
their behaviors accordingly (Wicklund, 1979).
It also appears that bad self-raters are considered less effective by their
bystanders (Atwater and Yammarino, 1992; Bass and Yammarino, 1991). For
this category there are two deficiences in a feedback process which lacks
bystander ratings:
(1) bad self-raters are not motivated to seek and apply bystander feedback;
(2) for bad self-raters the information to correct their self-image downwards
(in case of over-estimators) or upwards (in case of under-estimators) is
not available.
The first ``motivational'' deficiency in particular holds for over-estimators.
Since in their own view they ``are doing just fine'', and since that is the only
information they have, there is no reason for them to change. In contrast, under-
estimators have in their low self-ratings a clear motive to change, but on lack of
concrete and informational behavioral feedback, they have no clue for the
direction of the change. Under-estimators do not get feedback because in the
eyes of their bystanders there are no deficiences in their performance. So
whereas over-estimators are not motivated to look for feedback, under-
estimators do not get feedback.
Obviously, these deficiencies can be ``repaired'' by increasing motivation or
information. Below we will discuss the multi-rater feedback method with
respect to these feedback conditions.
Suppose bystander ratings collected at time-point 1 are fed back to the focal
person. What will then be the self-rating at a later time-point 2, that is, what will
the focal person do with the bystander-feedback obtained at time-point 1?
From a review study by Prue and Fairbank (1981) it appears that ``objective''
feedback generally has a positive effect on behavior. ``Objective'' in this case
Journal of
Managerial
Psychology
14,6
462
means: concrete, specific, timely and behavioral feedback. In a study by
Atwater et al. (1995) bystander-ratings of subordinates were fed back to their
respective managers. Some examples of items from this multi-rater method are:
(1) This leader treats me with respect.
(2) This leader deals with me in a consistent way.
(3) This leader treats me as an individual person instead of a number.
The effect of bystander-feedback given at time-point 1 on the manager's self-
ratings and the bystander-ratings at a later time-point 2, was:
.In general, managers' behavior improved after feedback. Thus, in
agreement with the general finding discussed before, feedback by others
stimulated actual changes in behavior.
.The number of in-agreements increased. The self-evaluation of the
manager became closer to the ratings of his or her subordinates.
According to the discussion above this implies that the effectiveness of
the manager increased.
In particular, the effect of multi-rater feedback in this study depended on the
group to which the focal person belonged at time-point 1, that is, on the
differences between self-ratings and bystander-ratings at time-point 1. In
accordance with the distinction made above between motivational and
informational feedback conditions, the effect of bystander feedback differed
between over-estimators and under-estimators.
Over-estimators
Results from research into the effects of bystander-feedback on behavior and
self-ratings for over-estimators are not equivocal.
Fleenor et al. (1996) found that, even when over-estimators adjusted their
behavior and consequently became more competent as judged by bystanders,
their self-ratings remained relatively unaffected. So over-estimation remained
the case ± although it had got smaller since competence as rated by bystanders
had increased. It seems that the general tendency to over-estimation discussed
above prevails even in case of feedback by multiple bystanders. Self-ratings do
not change although over-estimators do adjust (``upwards'') their actual
behavior as assessed by bystanders.
In contrast, in the study by Atwater et al. (1995), over-estimators appeared to
exhibit both more adequate behavior as assessed by bystanders and lower (i.e.
more realistic) self-ratings at the later time-point 2. So in that study self-images
were adjusted downwards at time-point 2.
Under-estimators
Atwater et al. (1995) found that bystander impact had more impact on the self-
evaluation of under-estimators than of over-estimators. In particular, self-
evaluation by under-estimators increased at the later time-point 2 after positive
Multi-rater
feedback
methods
463
bystander feedback at the earlier time-point 1. So self-evaluations are adjusted
upwards on account of feedback. This increase in self-image was not
accomplished by a corresponding increase in behavioral efficacy as observed
and assessed by bystanders. While self-images at time-point 2 were adjusted
positively, bystander feedback did not increase markedly at time-point 2
implying no real behavioral changes. It has to be noted, however, that for
under-estimators actual behavior already is better than the image they have of
themselves.
Concluding (see also Table I), research indicates that the effect of bystander-
feedback is that over-estimators improve their behavior (as rated by
bystanders) but either keep over-estimating (Fleenor et al., 1996) or become
more in-agreement (Atwater et al., 1995), and that under-estimators do not
change behavior but bring their self-ratings more in-agreement. All this points
to a relative independence between self-ratings (indicating self-image) and
behavior as rated by bystanders. And that implies that the best indicators of
job effectiveness, that is of actual competence, are bystander ratings and much
less the degree of discrepancy between self-ratings and bystander-ratings
(Fleenor et al., 1996, p. 502). Since over-estimators and in-agreements with the
same bystander-ratings are equally effective in practice, multi-rater feedback
systems should not be applied in organizations in order to reduce over-
estimation. An over-optimistic self-image should not be considered per se a sign
of practical ineffectiveness (above it appeared that it even can be the sign of a
``healthy mind''). For developmental purposes primarily the differences between
bystander-ratings at different time-points count.
The research discussed above implies that it is not bystander feedback per
se but the difference between self-ratings and bystander-ratings that is
motivational for the focal person: either to improve behavior (in case of over-
estimation) or to improve self-evaluation (in case of under-estimation). For over-
estimators, a large discrepancy can be an incentive to adjust own behaviors
(but not per se the self-ratings), and thereby improve competences in such a
way that bystander-ratings will increase. For example, Reilly et al. (1996) found
that repeated (four times during two-and-a-half years!) ``upward feedback'' to
managers by subordinates (using a questionnaire with 33 items) had a lasting
Table I.
Summary of research
findings on causes,
effects and impact of
multi-rater feedback in
case of over-estimation
or under-estimation
Over-estimation Under-estimation
Cause (Too) positive self-evaluation No information on behavior
Effect No feedback seeking behavior (Too) negative self-evaluation
Effect of multi-rater
feedback method
Conscious of difference
between self-evaluation and
bystander evaluation
Conscious of behavioral job
proficiency
Effect on behavior (as
rated by bystanders)
Improves Unaffected
Effect on self-evaluation Unaffected or less positive Improves
Journal of
Managerial
Psychology
14,6
464
positive effect on performance. Bystander-ratings improved with respect to the
earlier time-point 1 at the later time-point 2, and it stayed that way at the even
more later time-points 3 and 4.
What is the relation between multi-rater feedback and assessment center
feedback?
In multi-rater methods an inventory is made of concrete work behaviors by
relevant others, that is through ``committed bystanders''. Bystanders rate their
experiences with the focal person with respect to a number of work situations.
Obviously (as was pointed out above), those raters have had to be able to have
those experiences, and as a consequence have had to be able to interact to a
reasonable degree with the focal person. Thus, multi-rater feedback
presupposes raters to be part of the focal person's network of communication.
Comparing multi-rater feedback with feedback from assessment centers
(Jansen and de Jongh, 1997), we see that the assessors in the latter method:
.generally are not involved in daily work interactions with assessment
center candidates;
.are even selected in such a way that they do not have any prior
experiences with candidates;
.are presented candidate behaviors which are relevant to the aim of the
assessment center since these are evoked in controlled conditions;
.are able to share and discuss their observations with fellow assessors
because the assessment center is designed in such a way that a
candidate is always observed in the same situation by several assessors
at the same time (and so different assessors have a common
observational basis);
.appraise candidates for their competence in mastery of that set of
behaviors that determines effective control of critical job situations
simulated in the assessment center exercises;
.have clear and shared norms of (in)effective behaviors with respect to
those situations.
Thus, bystanders of a multi-rater focal person and assessors of an assessment
center candidate have quite different functions.
While an assessment center results in an inventory of a specific and limited
set of effectively mastered job situations, that is in a specific profile of critical
competences, multi-rater feedback yields a competence profile of the focal
person that is much more general. The end result is more general because
bystanders' ratings are averaged:
.both across a lot of different job situations, that is across the (many)
items from the multi-rater instrument; and
.across persons, that is across the set of bystanders who rate the focal
person from different perspectives.
Multi-rater
feedback
methods
465
By averaging across work situations or ``items'' and across role partners or
bystanders, an ``abstraction'' takes place where the focal person is, so to say,
lifted from his or her job context. In our view, this is the reason why there is a
tendency to conceive and apply multi-rater profiles as patterns of personal
dispositions which are exclusively tied to the person. Since in this way job and
organizational contexts evaporize, multi-rater feedback only creates individual
developmental tasks for the focal person, and not for bystanders or the
organization. At present, multi-rater feedback threatens to deteriorate to a
clinical instrument for diagnosing and curing employees for their individual
lack of abilities or ``diseases''.
Another difference with assessment center methods is that the use of multi-
rater feedback creates the necessity to be involved in and part of the
communication pattern of the person who is to be assessed. Per definition the
rater should be ``partner of the candidate'', should ``have something to do'' with
the focal person. Whereas an assessment center presupposes some distance
between rate and rater and, to keep that distance, assessors are trained in how
to observe and evaluate, multi-rater feedback requires interaction and personal
involvement of the rater. Consequently, inter-rater reliabilities generally are
acceptable to good for assessment centers (Latham et al., 1975; Lorenzo, 1984),
and low to almost absent for multi-rater methods as was mentioned before. But
low inter-rater agreement implies that averaging across raters or bystanders in
order to get a more complete and nuanced image of the focal person's
functioning (Landy and Farr, 1983) makes no sense. Averaging across raters in
order to ``average out'' measurement errors is not applicable any more.
One solution to this problem is to strive for a larger agreement between
raters; bystanders in multi-rater feedback should become ``trained assessors'',
so to say. Another solution would be to abstain from averaging and drawing a
general profile based on all bystanders, and to limit feedback to ratings from
individual bystanders. Disadvantages of that approach are, first, that
reliabilities of such kind of single point ratings are low, and, second, that
conflicting results may arise: whereas the one rater assesses a competence to be
present, another may diagnose it to be absent.
What is the relation between multi-rater feedback and TQM?
In the previous section, we compared the multi-rater feedback method with the
assessment center method. But as was stated in the introduction, the roots of the
multi-rater method also are in methods for client investigations in the context of
``total quality management''. Viewed from this perspective, the multi-rater method
is a system for organizing and collecting feedback from (internal) ``customers'' of
the focal person. However, there is a difference between feedback pertaining to
whether the ``customer is satisfied'' or feedback on whether the focal person is
competent. The following example may elucidate this (see also Table II).
When a pianist gives a recital, the feedback of his/her audience will be based
on their experience of the musical performance, that is on the ``satisfaction'' they
get out of it. This ``feedback'' certainly can be motivating for the piano player
Journal of
Managerial
Psychology
14,6
466
(every performer loves long and warm applauses . . .). Negative customer
feedback therefore can give a very strong impetus for improvement. Customer
feedback even is more stimulating in this respect than for instance more
detached assessment center feedback since satisfying the audience is the
ultimate aim of the pianist (without it he/she soon would be ``out of business'').
However, it is important to realize that feedback from an audience generally
does not contain concrete clues for behavioral improvement. Actual
improvement of behavior requires behavioral feedback, that is feedback in
terms of competences, and such feedback is not obtained from an audience of
bystanders but from an expert in the behaviors at issue (e.g. a piano teacher).
So we end up with two types of bystander feedback:
(1) feedback on account of whether ``the customer is satisfied'', that is
whether bystanders are served in their interests by the focal person;
(2) feedback on whether the focal person is competent with respect to a
certain job situation.
The first type of feedback typically is related to TQM methods, the second to
assessment center methods. Obviously, in multi-rater applications it should be
ascertained whether bystanders rate the focal person as ``customers'' or as
``assessors''. This is especially important because bystanders are both related to
and selected by the focal person.
Although multi-rater feedback is related to TQM feedback, there are also a
number of important differences. In case of TQM feedback:
.the (mostly) external customer (= producer of feedback) counts;
.consequently the focal person is considered as player of only one role;
customer feedback reaches as far as the service goes and the circle
mostly will be incomplete (only ``90 degrees''); and
.the aim is improvement of customer service.
But in case of multi-rater feedback:
.the internal focal person (= receiver of feedback) counts;
.consequently the focal person is considered the ``midpoint'' of a large
number of work roles; therefore feedback is total, the circle is complete
(``all'' 360 degrees); and
.the aim is personal development.
Table II.
The multi-rater method
and two types of
feedback
Position of rater Object of feedback
Relation between rater
and focal person Typical application
Expert in job Competence of focal
person
Distance Assessment center
Customer Satisfaction of customer
needs
Involvement TQM
Multi-rater
feedback
methods
467
Multi-rater feedback can be considered to follow naturally from TQM when
multi-rater feedback is conceived as a kind of TQM in which all internal and
external work roles are explored, and by averaging a complete ``impression'' of
the focal person with respect to ``customer satisfaction''.
Organizational conditions and effects of multi-rater methods
Until now the study on conditions and implications of multi-rater feedback
methods was restricted to an individual-oriented approach. That an
organizational approach is needed is indicated by the studies by Funderburg
and Levy (1997) and Waldman (1997) in which it was found that contextual
factors are more important in determining attitude toward and use of 360-
degree feedback than personality variables.
In the present study, we traced two ``roots'' of multi-rater feedback (see Table
II):
(1) the assessment center in which trained and not-involved assessors
observe and rate critical behaviors as demonstrated in specially
developed situational exercises, on prespecified and well-defined
competence dimensions;
(2) the tradition of TQM in which committed bystanders who are involved
in the actual work processes of the focal person, rate that person to the
degree to which they feel ``served'' by his or her daily, ``normal'' work
behavior.
When considered as a modified type of assessment center, the aim of the multi-
rater method is to enhance the focal person's personal development. The focus
is on the behaviors of the ratee, that is on the ratee's ``throughput''. But, when
considered as a TQM instrument, the final goal of the multi-rater method is to
increase satisfaction of internal customers. Emphasis therefore is on the ratee's
``output''.
In order to understand what happens as a result of (repeated) multi-rater
feedback in the context of a work organization, both interpretations should be
taken into account. In particular the second mode of use: multi-rater feedback
``as TQM'' underlines that multi-rater methods do not take place in a vacuum.
Therefore, in this section we zoom out and look at the wider circle of the work
context. In particular, we will discuss multi-rater feedback with respect to its
requirements of and impact on organizational communication. Changes in
organization will be discussed as well as their impact on communication and
feedback systems. Although these changes mostly amount to an ``opening up
and making more transparent'' of communication and feedback, we finally will
argue in favor of a new kind of ``feedback regulation'' at organizational level,
and investigate accordingly the proper place of multi-rater methods.
Beforehand, it should be emphasized that, whereas at the individual-
psychological level empirical research focussing on multi-rater feedback
Journal of
Managerial
Psychology
14,6
468
mechanisms and effects is available (as was discussed above), at the
organizational level we have to rely on more general, mostly only descriptive
approaches.
Changes in communication patterns in organizations
Antonioni (1996) points out that multi-rater feedback systems do not take place
in an organizational ``vacuum''. People work and are evaluated in the context of
an organizational communication system, consisting of such processes as
information gathering and sharing, performance feedback and personal
recognition. Also Lepsinger and Anntoinette (1997), providing organizational
guidelines for effective multi-rater degree feedback, stipulate that a core
requirement is clear and frequent communication. The use and effectiveness of
multi-rater feedback in work organizations therefore cannot be separated from
actual developments in communication.
Bartlett and Ghoshal (1994) (see also Ghoshal and Bartlett, 1995) describe a
number of changes in organizations. These can be summarized as: from
strategy to purpose, from structure to processes, and from systems to people.
Important is the parallelism between changes concerning transitions in
organizations, and changing approaches toward the functioning and the role of
communication in organizations. All kind of new work designs ± whether they
are called ``project management'', ``self-steering teams'', ``empowerment'' or
``knowledge-creating organizations'' ± presuppose both an increase in the
number of communication partners as well as an augmentation in the
complexity of the communication processes. A different way of cooperating
assumes a different design of communication patterns. Traditional
communication can be characterized as:
.top-down considerations prevail;
.the hierarchical line and communication lines overlap almost completely;
.one-way communication, with sender-dominance;
.information is an important power base, often leading to protection and
secrecy;
.bottom-up communication is mostly limited to reporting, to ``account''
for results;
.communication strictly conforms to imposed procedures;
.the mere fact of having a certain position implies the ``right'' to evaluate
subordinates, and leads to the automatic attribution of the necessary
competences and skills for this.
On the other hand, the evolving new communication pattern is characterized
by:
.information moves in all directions (vertically, horizontally, diagonally);
.it is rather a question of ``too much'', of an overload of information;
Multi-rater
feedback
methods
469
.a so-called ``holographic'' organization (by starting from any piece of a
holographic picture, one can reconstruct the whole picture) arises where
on each point all information is completely present (the Internet is a good
example of this, but the holographic character also holds for Intranet or
linked Databases (see Pasmore, 1995, p. 9));
.available information is passed on to all employees, as far as possible, so
that it is turned from a purely individual to a more organizational
property, and from implicit to explicit ± and thus debatable ±
knowledge;
.new technological possibilities, such as videoconferencing, telecomputer
conferencing and electronic mail, have resulted in a decrease of social
contacts so that an organization has to formally ``organize'' informal
contacts, for instance by providing ``obligatory'' breakfast or lunch
meetings, compulsory Friday afternoon drinks or evening parties;
.from the viewpoint of influencing organizational culture, more emphasis
is put on concrete day-to-day communication processes, on the way
people ``do things here'', in particular on the relations between manager
and subordinate, between subordinates and between departments
(Blumer, 1969; Larkin and Larkin, 1994).
These now more and more dominating communication patterns ask for a
different role and design of organizational feedback processes.
Multi-rater feedback in the ``spherical organization''
Miles and Snow (1995) present the metaphor of the ``spherical'' organization to
characterize an organization in which the new communication processes
prevail. This kind of organization has a very flexible structure in which
competent self-managing teams and other resources rotate around a common
knowledge-base. Such teams, capable of quick action, provide a distinctive
competitive advantage. These authors describe the importance of a matching
``human investment philosophy'' but do not elaborate this further in detail. In
our view, the way in which communication functions plays a crucial role in this
kind of organization.
In the ``spherical organization'' more openness in communication between
individuals and between groups is crucial. There are no obstacles, barriers or
filters which obstruct communication. Primary emphasis is put on direct
contacts between managers and subordinates, not only during crisis situations
but also during day-to-day, normal business situations. Only then does it make
sense to talk about concepts as ``empowerment'' (Spreitzer, 1995) and self-
steering teams. For, when the internal information lines have cleared,
employees can start to function as antennas for scanning the market and the
environment. Information can flow in and out freely. Essentially this means
Journal of
Managerial
Psychology
14,6
470
admitting again or even actively organizing feedback on all organization levels
and from all stakeholders in the company (as happens in multi-rater feedback
at the individual level).
Referring again to the work of Antonioni (1996), this author applies an open-
system model on the multi-rater feedback appraisal process, and specifies the
different components of such a system: inputs, processes, and (possible)
outputs. On account of longitudinal research in midsize American companies,
he argues that multi-rater feedback process should insure an increase in both
an appraisee's awareness of others' perceptions of him/her and the focal
person's commitment to improve targeted areas. Focal persons should be
trained in how to share and discuss their multi-rater appraisal results with their
``bystanders''. From his research data it follows that about 20 percent of the
feedback is of the following type: appraisees are surprised by feedback which
indicates that improvements are needed. This is the most difficult type for
appraisees to handle. More information, good coaching and fresh feedback are
needed in those cases. London et al. (1997) argue on account of a review of the
literature, that in particular organizational mechanisms should be activated to
ensure that feedback providers, and feedback receivers, and managers of the
latter, feel ``accountable'' for the outcome of the feedback process. In particular
managers should provide resources to support and reinforce real behavioral
change.
Antonioni observes the following outcomes of multi-rater feedback:
increased awareness of appraiser's expectations, improvement in appraisee's
work behavior, increase in informal multi-rater degree performance reviews,
and increase in the giving and receiving of constructive feedback on a more
regular basis. Note that in these results both uses of the multi-rater method
return (see Table II): multi-rater as performance feedback aiming at competence
improval of the focal person (``improvement in appraisee's work behavior''), or
multi-rater as feedback on satisfaction of customer needs (``increased
awareness of appraiser's expectations''). It would be a task for future research
to disentangle both applications of multi-rater feedback and, in particular, to
explore the long-term effects of such systems in a work organization. This
would not only require a longitudinal research design but also a multilevel
approach in which both psychological and organizational conditions and
effects are investigated. In particular, it is important to study how multi-rater
methods can be integrated in existing systems for personnel appraisal,
development and reward.
Discussion: the need for a new regulation of communication
From TQM to ``total feedback''?
Developments such as the growing importance of direct communication, more
openness in communication, and widening the feedback circle around
employees in a spherical organization, also may have disadvantages. A
consequence of these trends may be that the ``circle around the employee closes''
since employees ``cannot hide anymore''. They have to present themselves not
Multi-rater
feedback
methods
471
only with their competences and career expectations, but also with their
limitations (psychological, family, social), which until now have been
considered strictly private but which nevertheless influence their performance.
The boundary between work and the personal field (family, leisure time,
hobbies) has become more blurred. As a result, a continuous stream of feedback
is unleashed on the organization members, which could lead not only to a more
``transparent'' and responsible person, but also a kind of ``bombing of feedback''.
In the future special attention will have to to be paid to the redelineation of
these frontiers.
Legge (1995, p. 316) describes very sharply the nightmare of ``total
feedback'', using the proposal for a ``Panopticon prison'' by Jeremy Bentham:
(note that some prisons were built following this turret concept): ``Essentially, it
was (...) a building on a semicircular pattern with an inspection lodge at the
center and cells around the perimeter. Prisoners, in individual cells (...) were
open to the surveillance of guards, but, by a careful contrived system of lights
and wooden blinds, the guard would be invisible to the inmates. Control was to
be maintained by the constant sense that prisoners were watched by unseen
eyes''.
These considerations lead us toward the question whether or not in the
future we will need more regulated forms and patterns of communication. Even
though we notice for the moment an orientation toward more transparent, flat,
horizontal and open communication systems, which are able to cope in a
flexible and fast way with continuously changing market demands and
customer needs, we expect that a new need will arise for methods to steer
information flows again.
An illustration can be found already now in the position of middle-
management. In recent years the middle-management layer, considered as a
potential source of resistance and red tape, has been forced out of organizations
through practices such as delayering, flattening, business process
reengineering and horizontalization, and this on a scale never seen before. Even
more recently, however, there is a tendency to reinstall the middle-manager
somewhat. For instance, Nonaka and Takeuchi (1995), based on research in
mostly Japanese companies, conclude that middle-management can fulfill the
role of essential communication node and filter. Middle-management is able to
generalize concrete, operational experiences at the bottom of the organization
and make the connection with the more abstract vision and strategic concepts
in the upper levels of the organization. They relate this practice to a so-called
``middle-up-down''-movement, where the middle management plays a vital role,
up as well as down.
But also well-applied multi-rater feedback systems could play a role in the
re-organization of feedback. Through a reliable and continuous stream of multi-
rater feedback, steering information is made available, which makes it possible
to translate both the vision at the upper level to the concrete level of the
executing employee, and experiences based on daily work practices from
the bottom of the organization up to the upper levels. Multi-rater feedback, on
Journal of
Managerial
Psychology
14,6
472
the condition of proper use and translation to the meso- and macrolevel of the
organization, offers a chance for both stimulation and regulation of
communication. However, in order that multi-rater feedback can contribute to
effective organizational communication practices, a number of conditions
should be taken into account.
Organizational conditions for multi-rater methods
For an effective implementation of multi-rater feedback systems the following
conditions have to be fulfilled (see Lepsinger and Anntoinette, 1997; see also the
discussion above):
.use should be connected to the business;
.upper management should be involved, both in the role of rater and also
in that of focal person:
.users are involved in the design of the system;
.the profile produced by the focal person and his/her bystanders, has to
be complemented by additional data, originating from assessment
center methods;
.it is the focal person who determines which bystanders will be involved
in the process;
.anonymity and confidence of the feedback have to be guaranteed, so
that the focal person remains the final owner of these data;
confidentiality creates an open atmosphere and diminishes defensive
behavior, enlarges the willingness to listen and increases the chances of
honest feedback (Westerman and Rosse, 1997);
.prudence in coupling multi-rater feedback systems with regular
appraisal systems;
.focus on individual development (see Bettenhausen and Fedor, 1997);
only those who belong to the ``circle of influence'' of the focal person are
expected to take an active part in his/her personal development;
.the instrument (the questionnaire) is tested for coherence, and has to be
scored, interpreted and reported following a research based procedure;
.the focal person can make an appeal to a qualified career consultant in
interpreting and dealing with his/her feedback.
Common to these conditions is the notion that there are two ``parties'' in multi-
rater feedback: the focal person and the set of bystanders. Depending on the
intention of a multi-rater application, interests of one of these two will
dominate. In an ``assessment center-use'' of multi-rater feedback the focal
person is the primary focus of attention since his/her personnel development is
at stake, but in a ``TQM-application'' bystanders are the most important since
they are considered the primary internal customers (see Table II).
Multi-rater
feedback
methods
473
Conclusion
New developments, especially in the personnel field, tend to become a hype
very soon. That this danger is also not fictive in the context of multi-rater
feedback can be illustrated by the following citation, coming from a recent
American publication (Edwards and Ewen, 1996, pp. 47-50). These authors
state that multi-rater feedback:
is likely to prompt constructive action on the employee's part, creates credible behavior
feedback, creates high quality assessments for internal and external customers, yields rich
information for creating organization intelligence that provide a broad picture of training and
development needs, helps an employee answer ``How am I doing?'', pinpoints the employee's
areas of strength and areas in need of development, has more reliability, validity, and
credibility than any other performance assessment process.
In this quotation we clearly see both an assessment center-based use of multi-
rater feedback in which bystanders give feedback in order to help the focal
person in his/her competence development, and a ``TQM-based use'' of multi-
rater feedback in which information about ``customer satisfaction'' is organized
back to the ``serving person''. In addition a number of technical claims are made
about reliability and validity which, at present, cannot be substantiated by
research findings (as amply discussed in previous sections). Research into and
reflection on the application, conditions and effects of multi-rater-feedback,
both on an individual and organizational level, are seriously needed. For the
moment there is a lot of good research on the individual level, but little
adequate research on the organizational level.
To conclude, the contribution of this study consists of an overview and
interpretation of research findings in multi-rater feedback. Generally, the
ample research at an individual-psychological level indicates that reliability
of bystanders' ratings is low, and that focal persons react in different ways
to feedback depending on discrepancies between their self-image on the one
hand and bystander feedback on the other hand. Research indicates that in
order to enhance the development of the focal person, it is necessary to
supplement multi-rater feedback with feedback obtained by assessment
center methods.
In addition, it appeared that there are two kinds of uses and corresponding
interpretations of multi-rater feedback, originating from quite different
traditions: appraisal of a focal person's performance in the tradition of the
assessment center, or appraisal of customers' satisfaction in the domain of
quality movement. Especially in the latter kind of use of rater feedback,
concomitant effects on communication patterns in organizations should be
taken into account. It seems that in more popular publications the TQM-kind of
use prevails.
And finally, we repeat that more empirical research is needed in which both
psychological and organizational conditions and effects are studied.
Journal of
Managerial
Psychology
14,6
474
References
Alicke, M.D., Klotz, M.L., Breitenbecker, D.L., Yurak, T.J. and Vredenburg, D.S. (1995), ``Personal
contact, individuation and the better than average effect'', Journal of Personality and Social
Psychology, Vol. 68 No. 5, pp. 804-25.
Antonioni, D. (1996), ``Designing an effective 360-degree appraisal feedback process'',
Organizational Dynamics, Vol. 25, Autumn, pp. 24-38.
Ashford, S.J. (1984), ``Self-assessments in organizations: a literature review and integrative
model'', Research in Organizational Behavior, Vol. 11, pp. 133-74.
Atwater, L.E. and Yammarino, F.J. (1992), ``Does self-other agreement on leadership perceptions
moderate the validity of leadership and performance predictions?'', Personnel Psychology,
Vol. 45 No. 1, Spring, pp. 141-64.
Atwater, L., Roush, P. and Fischtal, A. (1995), ``The influence of upward feedback on self and
follower-ratings of leadership'', Personnel Psychology, Vol. 48 No. 1, Spring, pp. 35-59.
Bartlett, C.A. and Ghoshal, S. (1994), ``Changing the role of top management'', Harvard Business
Review, Vol. 72, November-December.
Bass, B. and Yammarino, F. (1991), ``Congruence of self and others' leadership ratings of naval
officers for understanding successful performance'', Applied Psychology: An International
Review, Vol. 40, pp. 437-54.
Bettenhausen, K.L. and Fedor, D.B. (1997), ``Peer and upward appraisals: a comparison of their
benefits and problems'', Group & Organization Management, Vol. 22 No. 2, June,
pp. 236-63.
Blumer, H. (1969), Symbolic Interactionism: Perspective and Method, Prentice-Hall, Englewood
Cliffs, NJ.
Brabet, J. and Klemm, M. (1994), ``Sharing the vision: company mission statements in Britain and
France'', Long Range Planning, Vol. 27 No. 1, pp. 84-94.
Church, A.H. (1997a), ``Do I see what I see? An exploration of congruence in ratings from multiple
perspectives'', Journal of Applied Social Psychology, Vol. 27 No. 11, June, pp. 983-1020.
Church, A.H. (1997b), ``Managerial self-awareness in high-performing individuals in organ-
izations'', Journal of Applied Psychology, Vol. 82 No. 2, pp. 281-92.
Church, A.H. and Bracken, D.W. (1997), ``Advancing the state of the art of 360-degree feedback:
guest editors' comment on the research and practice of multirater assessment methods'',
Group & Organization Management, Vol. 22 No. 2, June, pp. 149-61.
Cooper, W. (1981), ``Ubiquitous halo'', Psychological Bulletin, Vol. 90, pp. 218-44.
Edwards, M. and Ewen, A.J. (1996), 360
o
Feedback. The Powerful New Model for Employee
Assessment and Performance Improvement, Amacom, New York, NY.
Engelbrecht, A.S. and Fischer, A.H. (1995), ``The managerial performance implications of a
developmental assessment center process'', Human Relations, Vol. 48 No. 4, pp. 387-404.
Farh, J. and Dobbins, G. (1989), ``Effects of comparative information on the accuracy of self-
ratings and agreement between self and superior ratings'', Journal of Applied Psychology,
Vol. 74, pp. 606-10.
Festinger, L. (1954), ``A theory of social comparison processes'', Human Relations, Vol. 7,
pp. 117-40.
Fleenor, J.W., McCauley, C.D. and Brutus, S. (1996), ``Self-other agreement and leader
effectiveness'', Leadership Quarterly, Vol. 7 No. 4, pp. 487-506.
Francis-Smythe, J. and Smith, P.M. (1997), ``The psychological impact of assessment in a
development center'', Human Relations, Vol. 50 No. 2, pp. 149-67.
Multi-rater
feedback
methods
475
Funderburg, S.A. and Levy, P.E. (1997), ``The influence of individual and contextual variables on
360-degree feedback system attitudes'', Group & Organization Management, Vol. 22 No. 2,
June, pp. 210-35.
Ghoshal, S. and Bartlett, C.A. (1995), ``Changing the role of top management: beyond structures to
processes'', Harvard Business Review, Vol. 73, January-February.
Jansen, P.G.W. and de Jongh, F. (1997), Assessment Centers: A Practical Handbook, Wiley, New
York, NY.
Landy, F.J. and Farr, J.L. (1983), The Measurement of Work Performance, Academic Press, New
York, NY.
Larkin, T.J. and Larkin, S. (1994), Communicating Change. Winning Employee Support for New
Business Goals, McGraw-Hill, New York, NY.
Latham, G.P., Wexley, K.N. and Pursell, E.D. (1975), ``Training managers to minimize rating
errors in the observation of behavior'', Journal of Applied Psychology, Vol. 65, pp. 422-7.
Legge, K. (1995), Human Resource Management. Rhetorics and Realities, Macmillan Business,
London.
Lepsinger, R. and Anntoinette, D. (1997), The Art and Science of 360
o
Feedback, Pfeiffer, San
Francisco, CA.
London, M. and Smither, J.W. (1996), ``Can multi-source feedback change perceptions of goal
accomplishment, self-evaluations, and performance-related outcomes? Theory-based
applications and directions for research'', Personnel Psychology, Vol. 48, pp. 803-39.
London, M., Smither, J.W. and Adsit, D.J. (1997), ``Accountability: the Achilles' heel of multisource
feedback'', Group & Organization Management, Vol. 22 No. 2, June, pp. 149-61.
Lorenzo, R.V. (1984), ``Effects of assessorship on managers' proficiency in acquiring, evaluating
and communicating information about people'', Personnel Psychology, Vol. 37, pp. 617-34.
Mabe, P. and West, S. (1982), ``Validity of self-evaluation of ability: a review and meta-analysis'',
Journal of Applied Psychology, Vol. 67, pp. 280-96.
Maciel, A.G., Heckhausen, J. and Baltes, P.B. (1994), ``A life-span perspective on the interface
between personality and intelligence'', in Sternberg, R.J. and Ruzgis, P. (Eds), Personality
and Intelligence, Cambridge University Press, pp. 61-103.
Miles, R.E. and Snow, C.C. (1994), Fit, Failure and the Hall of Fame, Free Press, New York, NY.
Murphy, K.R. and Cleveland, J.N. (1995), Understanding Performance Appraisal. Social,
Organizational, and Goal-Based Perspectives, Sage Publications, Thousand Oaks, CA.
Nonaka, I. and Takeuchi, H. (1995), The Knowledge-Creating Company, Oxford University Press,
New York, NY.
Pasmore, W.A. (1995), ``Social science transformed: the socio-technical perspective'', Human
Relations, Vol. 48 No. 1, pp. 1-21.
Prue, D. and Fairbank, J. (1981), ``Performance feedback in organizational behavior management:
a review'', Journal of Organizational Behavior Management, Spring, pp. 1-16.
Reilly, R.R., Smither, J.W. and Vasilopoulos, N.L. (1996), ``A longitudinal study of upward
feedback'', Personnel Psychology, Vol. 49 No. 3, Fall, pp. 599-612.
Salam, S., Cox, J.F. and Sims, H.P. Jr (1997), ``In the eye of the beholder: how leadership relates to
360-degree performance ratings'', Group & Organization Management, Vol. 22 No. 2, June,
pp. 185-209.
Spreitzer, G.M. (1995), ``Psychological empowerment in the workplace: dimensions,
measurement, and validation'', Academy of Management Journal, Vol. 38 No. 5,
pp. 1442-65.
Journal of
Managerial
Psychology
14,6
476
Waldman, D.A. (1997), ``Predictors of employee preferences for multirater and group-based
performance appraisal'', Group & Organization Management, Vol. 22 No. 2, June, pp. 264-
87.
Ward, P. (1995), ``A 3608-degree, for the better'', People Management, February.
Westerman, J.W. and Rosse, J.G. (1997), ``Reducing the threat of rater nonparticipation in 360-
degree feedback systems: an exploratory examination of antecedents to participation in
upward ratings'', Group & Organization Management, Vol. 22 No. 2, June, pp. 288-309.
Wheeler, L. (1991), ``A brief history of social comparison theory'', in Suls, J. and Wills, T.A. (Eds),
Social Comparison. Contemporary Theory and Research, Lawrence Erlbaum, Hillsdale, NJ,
pp. 3-21.
Wicklund, R. (1979), ``The influence of self on human behavior'', American Scientist, Vol. 67,
pp. 187-93.
Wills, T.A. and Suls, J. (1991), ``Commentary: neo-social comparison theory and beyond'', in Suls,
J. and Wilss, T. (Eds), Social Comparison: Contemporary Theory and Research, Erlbaum,
Hillsdale, NJ, pp. 395-411.
Wrightsman, L.S. (1972), Social Psychology in the Seventies, Wadsworth, Belmont, CA.
