Content uploaded by Allen Huffcutt
Author content
All content in this area was uploaded by Allen Huffcutt on Dec 21, 2018
Content may be subject to copyright.
Psychometric Properties of Multisource Performance Ratings: A meta-Analysis of Subordinate, Supervisor, Peer, and Self-Ratings
Abstract
The purpose of this investigation was to examine the psychometric properties (interrater reliabilities within source and correlations between sources) of subordinate, supervisor, peer, and self-ratings of job performance. Different job types and dimension types were compared. Using meta-analytic methodology, we found that subordinates showed the lowest mean reliability (.30) and supervisors showed the highest (.50), with peers in between (.37). Mean correlations between sources were low for subordinate ratings (.22 with supervisor, .22 with peer, and .14 with self-ratings) and for self-ratings (.22 with supervisor and .19 with peer ratings). The mean supervisor-peer correlation was higher at .34. Both reliabilities and correlations between sources tended to be higher for nonmanagerial and lower complexity jobs. Comparisons of between-source correlations with within-source reliabilities indicated that, with some qualifications, the different sources had somewhat different perspectives on performance. Dimension reliabilities differed somewhat for interpersonal and cognitive dimensions.
... While supervisors are in a privileged position to assess their employees' in-role performance, their measurement is not necessarily exhaustive, and their view may differ from that of other potential raters (e.g., colleagues, subordinates). As Conway and Huffcutt (1997) suggest, there may be different perspectives on an employee's job performance levels and involving more than one rater can improve measurement outcomes. It is thus possible that a more accurate measure for in-role performance would be conducive to more supportive findings for H1a. ...
... These would allow more comprehensive measurements to be obtained, and, possibly, more accurate findings (cf. Conway & Huffcutt, 1997). ...
This study examines employee evaluations of HR practices (HRPs), which are a critical and yet underexplored antecedent of their effectiveness. Drawing on the Job Demands–Resources model, it proposes a moderated mediation model that studies the relationship between employee satisfaction with HRPs and in-role and extra-role performance as mediated by engagement. The results suggest satisfaction with HRPs influences both the performance dimensions considered, although its effects are greater for extra-role performance. Engagement intervenes significantly only in the relationship between this satisfaction and extra-role performance. Furthermore, the study acknowledges the burdens HRP-related resources may place on the workforce and examines the moderating effects of a personal resource such as health on the satisfaction with HRPs-engagement link, finding that it significantly strengthens it. It also finds that the indirect effect of satisfaction with HRPs on extra-role performance is stronger for healthier employees. These findings provide novel insights into the HR causal chain and help practitioners to better manage HRP design, communication, and audits.
... None of these studies empirically tested the quality of feedback managers receive although Nadler et al (2003) found that employees across all levels generally perceived upward feedback to be of lower quality. Conway and Huffcutt (1997) provide empirical evidence that managers receive relatively inconsistent performance appraisals compared to non-managers. Specifically, their study found the inter-rater reliability of different sources of managers' performance appraisal to be lower than the interrater reliability among performance appraisal sources for non-managers. ...
... Given that managers receive conflicting reports about their performance (at least relative to non-managers) managers should be less certain what the performance appraisals are telling them. Conway and Huffcutt (1997) gathered objective evidence about performance appraisal quality. That is, the measure of performance appraisal quality was the inter-source consistency. ...
This paper compares the performance appraisal quality and feedback seeking of managers to that of non-managers. This study answers Ashford et al's (2003) call for more research on the feedback seeking of managers, especially compared to non-managers. This study also extends work by Ashford and Tsui (1991) and Tsui et al's (1995) on negative feedback seeking by examining the phenomenon among managers. Data were collected via survey from 355 employees of two parts of an international retail chain. Results indicate that managers receive lower quality feedback than non-managers, do not feel inhibited by the psychological and impression costs of feedback seeking, relative to non-managers, and seek feedback, more than non-managers do. This study helps us understand factors that may help us improve the feedback managers receive. This study is limited by use of a single question measure of managerial status and generalization will require sampling more broadly.
... Observer rating of personality is based on outward expressions of traits and behavioral cues, which is markedly different from self-ratings based on inward expressions and intrapersonal processes (Colbert et al., 2012). As different rating sources include different perspectives on leader performance, previous literature has shown the discrepancies between self-ratings and observer-ratings (Conway and Huffcutt, 1997 ...
Purpose
This study aims to examine the relationship between leaders' personality traits and their internal communication practices from the leaders' and the followers' perspectives. The effectiveness of leader communication on the followers' perceived relational quality with their organization is also tested.
Design/methodology/approach
Two survey studies in the United States were conducted focusing on leaders' and followers' perspectives on communication practices, respectively.
Findings
Results of Study 1 showed that leaders' self-reported extraversion, conscientiousness, agreeableness, emotional stability and openness to experience are all significantly and positively associated with their symmetrical and transparent communication practices. From the followers' perspective (Study 2), extraversion and agreeableness positively influenced symmetrical leader communication, while agreeableness and conscientiousness were positively related to transparent leader communication. Both symmetrical and transparent leader communication enhanced the followers' perceived relationship quality with their organization.
Originality/value
The current study is among the first attempts to incorporate the trait theory of leadership in understanding organizational leaders' effective and ethical communication practices with their followers.
... When this value was not reported, we used reliability estimates obtained from meta-analyses. Specifically, we corrected subjective performance measures using a reliability estimate of 0.60 (Conway & Huffcutt, 1997) and we corrected objective performance measures using a reliability estimate of 0.61 (Sturman et al., 2005). For measures of performance that were unlikely to be affected by measurement error (e.g., administrative records of training grades or turnover), we assumed a reliability of 1.00. ...
Cognitive ability is one of the best predictors of performance on the job and past research has seemingly converged on the idea that narrow cognitive abilities do not add incremental validity over general mental ability (GMA) for predicting job performance. In the present study, we propose that the reason for the lack of incremental validity in previous research is that the narrow cognitive abilities that have been assessed most frequently are also the abilities that are most highly correlated with GMA. Therefore, we expect that examining a broader range of narrow cognitive abilities that are less highly correlated with GMA will demonstrate incremental validity for narrow abilities. To examine this prediction, we conducted an updated meta-analysis of the relationship between cognitive ability and a multidimensional conceptualization of job performance (task performance, training performance, organizational citizenship behavior, counterproductive work behavior, withdrawal). Using several different methods of analyzing the data, results indicated that the narrow cognitive abilities that are the least highly correlated with GMA added substantial incremental validity for predicting task performance, training performance, and organizational citizenship behavior. These results have important implications for the assessment of cognitive ability and the employee selection process.
... Although the requirement for multisource approaches to assess the impact of leadership on employee outcomes is not new (Conway & Huffcutt, 1997), only a few studies examined the impact of HoL on mental health from supervisor and employee perspectives ( However, a positive relationship between supervisor and employee ratings does not inform about whether or not self-and other ratings differ from each other. Prior research has shown that people tend to overestimate their own abilities, their performance, their chance of success, or their level of control (Meikle et al., 2016). ...
Im Zuge der Globalisierung, Digitalisierung und des gesellschaftlichen Wandels, hat sich die Arbeitswelt in den letzten Jahren stark verändert. Diese Veränderungen stellen erhöhte Anforderungen an die psychische Gesundheit der Beschäftigten. Die Prävention psychischer Störungen am Arbeitsplatz und die Frage, wie Arbeit gesund gestaltet werden kann, gewinnt daher zunehmend an Bedeutung. Eine zentrale Rolle nimmt dabei die Führungskraft ein, da diese maßgeblich die Arbeitsumgebung und die Arbeitsprozesse gestaltet und im direkten Kontakt mit ihren Mitarbeitenden steht. Zudem haben in den letzten Jahren achtsamkeitsbasierte Programme zur Prävention psychischer Störungen enorm an Bedeutung gewonnen, deren Wirksamkeit im Arbeitskontext jedoch oft in Frage gestellt wurde. Ziel dieser Dissertation ist es, die Wirksamkeit und potentielle Wirkmechanismen achtsamkeitsbasierter Programme in der Arbeitswelt zu prüfen und deren Integration im Rahmen der gesunden Führung zu untersuchen. Zu diesem Zweck ist die vorliegende Dissertationsarbeit in insgesamt vier Abschnitte gegliedert. In Abschnitt 1 wird der Effekt psychischer Belastungen von N = 2.287 Studienteilnehmenden im Hinblick auf deren Arbeitsunfähigkeitstage und Krankheitskosten in den folgenden zwei Jahren geprüft, um die langfristigen ökonomischen Folgen psychischer Belastungen zu erörtern und die Relevanz von Präventionsangeboten aus einer sozioökonomischen Perspektive zu beurteilen. Die Ergebnisse zeigen einen signifikanten Zusammenhang zwischen subjektiv erlebter psychischer Belastung und späteren Arbeitsunfähigkeitstagen bzw. Krankheitskosten. So zeigt sich, dass schwer belastete Personen 27mal so viele Arbeitsunfähigkeitstage im ersten Jahr und 10mal so viele Arbeitsunfähigkeitstage im zweiten Jahr aufweisen, verglichen mit Personen ohne psychische Belastungen. Außerdem zeigten schwer belastete Personen 11fach erhöhte Krankheitskosten im ersten Jahr und 6fach erhöhte Krankheitskosten im zweiten Jahr, verglichen mit nicht belasteten Personen. Auch schon bei leichten und mittleren psychischen Belastungen zeigten sich signifikant erhöhte Arbeitsunfähigkeitstage und Krankheitskosten (2fach bis 11fach erhöht). Diese Ergebnisse verdeutlichen die sozioökonomische Relevanz psychischer Belastungen und bilden eine empirische Grundlage für die Annahme, dass durch effektive Präventionsmaßnahmen sowohl individuelles Leid gelindert als auch sozioökonomische Kosten reduziert werden können. In Abschnitt 2 wird in Form einer Übersichtsarbeit der aktuelle Forschungsstand zu achtsamkeitsbasierten Programmen in der Arbeitswelt skizziert und deren Wirksamkeit und potentielle Wirkmechanismen metaanalytisch geprüft. Über k = 56 randomisiert-kontrollierte Interventionsstudien hinweg zeigen sich kleine bis mittlere Effekte (g = 0,32 bis 0,77) auf unterschiedlichen gesundheitsbezogenen Variablen (z.B. Wohlbefinden, Stress, subsyndromale Symptome, Burnout und somatische Beschwerden) sowie arbeitsbezogenen Variablen (z.B. Arbeitsengagement, Arbeitszufriedenheit und Produktivität), die bis zu 12 Wochen nach der Intervention bestehen bleiben. Diese Ergebnisse legen nahe, dass achtsamkeitsbasierte Programme effektiv in verschiedenen Arbeitskontexten eingesetzt werden können und somit eine gute Grundlage zur Prävention psychischer Störungen in der Arbeitswelt bilden. In Abschnitt 3 wird der Zusammenhang zwischen gesunder Führung und psychischer Gesundheit aus Perspektive von Führungskräften und deren Mitarbeitenden in einem querschnittlichen Studiendesign mit Hilfe von Mehrebenenanalysen untersucht, um den komplexen Zusammenhang zwischen Führung und Gesundheit besser zu verstehen. Dabei zeigt sich, dass die Einschätzungen der gesunden Führung zwischen Führungskräften und ihren Mitarbeitenden deutlich abweichen und nur auf konkreten Verhaltensdimensionen signifikante Zusammenhänge aufweisen. Die subjektive Wahrnehmung der gesunden Führung durch die Mitarbeitenden zeigt einen signifikanten Zusammenhang mit deren psychischer Gesundheit, nicht aber die Selbsteinschätzungen der Führungskräfte. Insgesamt weisen diese Ergebnisse darauf hin, dass die subjektive Wahrnehmung gesunder Führung eine wichtige Determinante für die psychische Gesundheit bei der Arbeit darstellt, dass das Thema gesunde Führung jedoch expliziter im Arbeitskontext ausgestaltet werden sollte, um ein gemeinsames Verständnis von gesunder Führung zwischen Führungskräften und Mitarbeitenden zu schaffen. In Abschnitt 4 wird dargestellt, wie die gesunde Führung mit Hilfe einer gezielten Intervention gefördert werden kann und welche Rolle das Konzept der Achtsamkeit dabei einnimmt. Die achtsamkeitsbasierte Intervention umfasst drei Seminartage (à 8 Stunden) zu i) gesunder Selbstführung, ii) gesunder Mitarbeiterführung und iii) Umgang mit psychisch belasteten Mitarbeitenden sowie zwei Nachhaltigkeitstermine (à 3 Stunden). Anschließend werden die Wirksamkeit sowie potentielle Wirkmechanismen der Intervention in einem quasiexperimentellen Studiendesign sowohl auf Ebene der Führungskräfte als auch auf Ebene der Mitarbeitenden empirisch geprüft. In 12 Unternehmen nahmen insgesamt 117 Führungskräfte an der Intervention teil. Die Führungskräfte und deren 744 Mitarbeitende machten Angaben zu ihrer psychischen Belastung sowie zur gesunden Führung zu drei Messzeitpunkten (Prä, Post, 3 Monate Follow-Up). Diese Angaben wurden mit einer passiven Kontrollgruppe, basierend auf Propensity Score Matching, verglichen. Hierarchische lineare Modelle ergaben, dass die Führungskräfte, die an der Intervention teilgenommen haben, eine signifikant stärkere Abnahme der psychischen Belastung und eine Zunahme der gesunden Selbst- und Mitarbeiterführung im Zeitverlauf aufzeigen als die gematchten Kontrollpersonen (g = 0,27 bis 0,55). Der signifikante Interventionseffekt auf die psychische Belastung der Führungskräfte wird durch die Häufigkeit der selbstständig durchgeführten Achtsamkeitsübungen moderiert und durch eine Zunahme der gesunden Selbstführung vermittelt. Auf Mitarbeiterebene ergaben sich keine signifikanten Effekte zwischen den Gruppen im Zeitverlauf. Es zeigte sich jedoch ein signifikanter Zusammenhang zwischen der subjektiv erlebten gesunden Führung und der späteren psychischen Belastung. Dies deutet darauf hin, dass die subjektive Wahrnehmung der gesunden Führung eine wichtige Determinante der psychischen Gesundheit von Mitarbeitenden darstellt. Insgesamt tragen diese Ergebnisse zu unserem Verständnis bei, wie eine gesunde Führung effektiv trainiert werden kann, um die gesunde Selbst- und Mitarbeiterführung der teilnehmenden Führungskräfte zu erhöhen und deren psychische Belastung zu reduzieren. Die Ergebnisse weisen jedoch gleichermaßen auf die Herausforderung hin, Interventionsprogramme weiter zu verbessern, um deren indirekte Effektivität auf Mitarbeiterebene zu erhöhen. Diese Dissertation trägt insgesamt dazu bei, die Bedeutung der Prävention psychischer Störungen in der Arbeitswelt zu verdeutlichen und zeigt Möglichkeiten auf, wie eine effektive Prävention in Unternehmen ausgestaltet werden kann.
... These analyses were based on received ratings (from officers and political colleagues), which were averaged to create a single score for each politician. The correlations between self-and received ratings on the performance factors are significant and, although small, are comparable with other studies and meta-analyses that indicate self-other agreement is typically low in multisource feedback (Conway & Huffcutt, 1997;Fleenor, Smither, Atwater, Braddy & Sturm, 2010). ...
... Bu etkinin, başka bir kaynağın (yönetici değerlendirmesi veya nesnel ölçümün) değerlendirmesi ile çalışanın kendini değerlendirmesi arasındaki farklılığı açıkladığını yorumlayan çalışmalar mevcuttur (Jaramillo, Carrillat, ve Locander, 2003;Jaramillo vd., 2005;Plouffe vd., 2009). Ek olarak çeşitli araştırmalarda (Conway ve Huffcutt, 1997;Salgado ve Moscoso, 2019;Viswesvaran, Ones, ve Schmidt, 1996) yöneticilerin gerçekleştirdiği değerlendirmelerin güvenilirliğinin daha yüksek olduğu doğrulanmıştır. Yukarıda ifade edilen yönetici/çalışan değerlendirmeleri arasındaki farklılığın "hata" olarak değerlendirilebilmesi için "yönetici değerlendirmesinin" esas/doğru olarak ele alınması da aslında başka bir tartışma alanıdır. ...
Bu çalışmanın amacı, liderlik davranışlarının ve yetenek geliştirme programlarının satış temsilcilerinin algılanan (öznel) ve nesnel performansına etkisini incelemektir. Aynı zamanda performansın görev ve görev dışı boyutu ile nesnel performans arasındaki ilişki de incelenmektedir. Veri toplama süreci aralarında üç ay olan iki farklı dönemde gerçekleştirilmiştir. Türkiye’de faaliyet gösteren bir moda perakendesi firmasında çalışan 269 çalışan, ilgili yetenek geliştirme eğitiminde başarılı olmuş ve araştırmanın sonraki aşamalarında değerlendirmeye katılmıştır. Araştırmada; çalışanların kendini değerlendirdiği, yöneticilerin çalışanları değerlendirdiği ve çalışanların yöneticileri değerlendirdiği çeşitli anketler bilgisayar destekli sistemler aracılığı ile uygulanmıştır. Araştırma sonuçlarına göre, hem görev ve ilişki odaklı liderlik davranışları hem de eğitim faaliyetleri performansı olumlu yönde etkilemektedir. Söz konusu nesnel performans olduğunda ise bu etki sadece eğitim faaliyetleri için geçerlidir. Ayrıca nesnel performans ile görev performansı arasında orta, nesnel performans ile bağlamsal performans arasında zayıf bir korelasyon tespit edilmiştir. Uygulayıcılar çalışanların nesnel performansını olumlu etkilemek için eğitim ve gelişim faaliyetlerine güvenebilirler. Çalışanların örgütsel vatandaşlık davranışlarında olumlu gelişmeler, ilişki odaklı liderlik davranışları ile sağlanabilir. Çalışma, liderlik davranışlarının ve eğitim değerlendirme seviyelerinin tamamının, farklı performans türleriyle ilişkisini tek bir araştırma içinde değerlendirerek sık rastlanmayan bir katkı ortaya koymayı hedeflemektedir.
... The literature on individual performance appraisal is abundant and has emphasized a consistent difference between supervisory and self-ratings of individual performance such that self-ratings are more lenient (Heidemeier and Moser, 2009). Supervisory ratings, on the other hand, seem to be considered more reliable (Conway and Huffcutt, 1997), are better linked to performance as measured by external criteria (promotions and salary, etc.) (Beehr et al., 2001;Atkins and Wood, 2002) and, as such, seem to be favored in both organizational practice and research. ...
Our study tests in a sample of 87 organizational groups (297 employees and 87 supervisors) the mediating role of leader-member exchange (LMX) and collective narcissism in the relationship between supervisors’ dark triad (SDT) personality traits and ratings of team outcomes made by supervisors and team members. We show that LMX mediates the association between SDT and team performance and innovation as rated by team members, while collective narcissism mediates the association between SDT and supervisory ratings of team innovation and team performance. Moreover, collective narcissism also mediates the association between SDT and team innovation as rated by team members. Results show that team-level performance appraisal is influenced by supervisory attributes and that the quality of relational exchanges and collective narcissism are plausible mechanisms explaining this association. The use of supervisory ratings of team outcomes in empirical research should also account for the supervisory attributes.
... which represented a moderate level of agreement among raters and comparable to those reported in earlier studies where peer ratings are used (e.g. Berry et al., 2013;Conway & Huffcutt, 1997). McDonald's ω for interpersonal competency indicated by the aggregated peer-reported items was .78. ...
The honesty‐humility factor from the HEXACO model of personality has been found to offer incremental validity in predicting several work‐related criteria over the remaining factors, yet its interplay with other personality factors is rarely examined. In this study, we examined how honesty‐humility (the tendency to be sincere, fair, non‐materialistic, and modest) can moderate the relation between agreeableness and interpersonal competency. Specifically, drawing on the theory of self‐concept, we proposed that agreeableness will have a stronger association with interpersonal competency amongst individuals who are higher on honesty‐humility, and relatively less so amongst individuals who are lower on honesty‐humility. Across three samples of people in managerial roles from two different cultures (Australia and Kenya), we found that honesty‐humility indeed moderated the agreeableness – interpersonal competency relation, both when the criterion was measured by self‐report (Sample 1, N = 167; Sample 2, N = 320; Sample 3, N = 296) and other‐report (Sample 3, N = 195). In all three samples, the positive relation of agreeableness with interpersonal competency was strongest among those who were also higher on honesty‐humility. Such an interaction effect was robust after controlling for the remaining HEXACO personality factors.
Performance appraisal of municipal police officers has presented problems similar to those encountered in industrial performance evaluation. The present study was concerned with the development of peer and supervisory rating scales for patrol officers. A total of 58 police departments cooperated in constructing and field testing supervisory and peer rating scales. A total of 8 supervisory and 9 peer scales were developed. The psychometric characteristics of the scales indicated that they were suitable for use in evaluating the performance of patrol officers. Principal components analyses indicated that 3 linear components described the variation in ratings for both supervisory and peer scales. While these components seemed to represent similar constructs, they were represented by performance dimensions specific to the level of observation. The generalizability of behaviorally anchored rating scales to settings other than those in which they were constructed is discussed.
Leadership has long been considered a crucial element of a manager's role and thus an important determinant of organizational effectiveness. Much of the importance attributed to leadership stems from the presumed effect of a leader's behavior on his subordinates' performance and satisfaction. Empirical evidence for this argument has come largely from the numerous field studies which have investigated the relationships among leadership styles (e.g., initiating structure and consideration) and subordinate performance and satisfaction. The problem which arises from this research and, for that matter, with virtually all of the field research which has examined these relationships is one of interpretation. The research design and analytical method employed in these studies, static correlational analysis, provides little basis for inferring causality. For example, a highly significant static correlation between leader initiating structure and subordinate performance indicates only that the two variables are related. The leader's initiating structure may have caused variance in subordinate performance or, conversely, changes in subordinate performance may have caused variance in the leader's initiation of structure. Third, there may have been no causal relationship between the two variables; the correlation may have been spurious or a third or additional variables may have caused the two variables in question to covary. Static correlational analysis, unfortunately, provides too little information to evaluate any of these alternative explanations of the significant, correlation.
• Leniency errors, halo effects, and differential dimensionality were explored in an analysis of superior, self-, and peer performance ratings of 107 managerial and 76 professional employees in a medium-sized manufacturing location, representing 95% of the managerial and professional staff. Self-ratings showed greater leniency effects than superior or peer ratings. A multitrait–multimethod (MTMM) analysis supported the presence of strong halo effect and significant convergent validity but not discriminant validity. The dimensional analysis supported the presence of strong halo effect. A statistical control procedure for the halo effect was developed that involved calculating residuals of the performance items, controlling for the "overall effectiveness" variance component in each item. The procedure did not reduce the significant halo effect, nor did it improve the nonsignificant discriminant validity in the MTMM analysis. It did, however, clarify the dimensional structure of ratings by superiors. Data from 4 previously published studies were also reanalyzed using the statistical control procedure. (19 ref) (PsycINFO Database Record (c) 2012 APA, all rights reserved)
• Leniency errors, halo effects, and differential dimensionality were explored in an analysis of superior, self-, and peer performance ratings of 107 managerial and 76 professional employees in a medium-sized manufacturing location, representing 95% of the managerial and professional staff. Self-ratings showed greater leniency effects than superior or peer ratings. A multitrait–multimethod (MTMM) analysis supported the presence of strong halo effect and significant convergent validity but not discriminant validity. The dimensional analysis supported the presence of strong halo effect. A statistical control procedure for the halo effect was developed that involved calculating residuals of the performance items, controlling for the "overall effectiveness" variance component in each item. The procedure did not reduce the significant halo effect, nor did it improve the nonsignificant discriminant validity in the MTMM analysis. It did, however, clarify the dimensional structure of ratings by superiors. Data from 4 previously published studies were also reanalyzed using the statistical control procedure. (19 ref) (PsycINFO Database Record (c) 2012 APA, all rights reserved)
The interrater reliabilities of ratings of 9,975 ratees from 79 organizations were examined as a function of length of exposure to the ratee. It was found that there was a strong, nonlinear relationship between months of exposure and interrater reliability. The correlation between a logarithmic transformation of months of experience and reliability was .73 for one type of ratings and .65 for another type. The relationship was strongest during the first 12 months on the job. Changes in reliability were accounted for mostly by changes in criterion variance. Asymptotic levels of reliability were only about .60, even with 10-20 years of experience. Implications for estimating reliabilities in individual and meta-analytic studies and for performance appraisal were presented, and possible explanations of the reliability-variance relationship were advanced.