The Impact of Job Complexity and Performance Measurement on the Temporal Consistency, Stability, and Test–Retest Reliability of Employee Job Performance Ratings

School of Hotel Administration, Cornell University, Ithaca, NY 14853, USA.
Journal of Applied Psychology (Impact Factor: 4.31). 04/2005; 90(2):269-83. DOI: 10.1037/0021-9010.90.2.269
Source: PubMed


Although research has shown that individual job performance changes over time, the extent of such changes is unknown. In this article, the authors define and distinguish between the concepts of temporal consistency, stability, and test-retest reliability when considering individual job performance ratings over time. Furthermore, the authors examine measurement type (i.e., subjective and objective measures) and job complexity in relation to temporal consistency, stability, and test-retest reliability. On the basis of meta-analytic results, the authors found that the test-retest reliability of these ratings ranged from .83 for subjective measures in low-complexity jobs to .50 for objective measures in high-complexity jobs. The stability of these ratings over a 1-year time lag ranged from .85 to .67. The analyses also reveal that correlations between performance measures decreased as the time interval between performance measurements increased, but the estimates approached values greater than zero.

1 Follower
27 Reads
  • Source
    • "Thus the independent variables temporally preceded the ratings. The correlation between a person's rating in year t and in t – 1 was above .60 in both firms, which is consistent with prior work on performance ratings in high-complexity jobs (Sturman, Cheramie, and Cashen, 2005). Thus, consistent with prior studies, the ratings were relatively stable over time but also showed some non-trivial temporal variation, making it possible to estimate fixed-effects models. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Using a longitudinal study of professionals in two information technology services firms, as well as interview data, this paper illuminates how organizational fortunes influence individual performance over time, examining how the economic situation of an organization leaves a lasting imprint on new employees and how that imprint affects subsequent job performance. The core hypothesis, supported by the results, is that the more similar the initially experienced level of organizational munificence is to the level of munificence in a subsequent period, the higher an individual’s job performance. This relationship between what I call “imprint–environment fit” and performance is contingent on the individual’s career stage when entering the organization and the influence of secondhand imprinting resulting from the social transmission of others’ imprints. A possible implication of the core hypothesis may be a “curse of extremes,” whereby both very high and very low levels of initial munificence are associated with lower average performance during a person’s subsequent tenure. One mechanism underlying these patterns is that employees socialized in different resource environments develop distinct approaches to problem solving and client interactions, which then lead to varying levels of imprint–environment fit in subsequent resource environments.
    Full-text · Article · Nov 2014 · Administrative Science Quarterly
    • "Subjective measures often correct for determinants of performance outside the control of the individual (Campbell et al., 1993) and can encompass a wide variety of behaviors and outputs relevant to the job (Medoff and Abraham, 1981). Meta-analyses have shown high test-retest reliability in performance evaluations (Sturman, Cheramie, and Cashen, 2005), and supervisory ratings are among the most common dependent variables in studies of performance (Sturman, 2003). Compensation. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Individuals often enter similar jobs via two different routes: internal mobility and external hiring. I examine how the differences between these routes affect subsequent outcomes in those jobs. Drawing on theories of specific skills and incomplete information, I propose that external hires will initially perform worse than workers entering the job from inside the firm and have higher exit rates, yet they will be paid more and have stronger observable indicators of ability as measured by experience and education. I use the same theories to argue that the exact nature of internal mobility (promotions, lateral transfers, or combined promotions and transfers) will also affect workers’ outcomes. Analyses of personnel data from the U.S. investment banking arm of a financial services company from 2003 to 2009 confirm strong effects on pay, performance, and mobility of how workers enter jobs. I find that workers promoted into jobs have significantly better performance for the first two years than workers hired into similar jobs and lower rates of voluntary and involuntary exit. Nonetheless, the external hires are initially paid around 18 percent more than the promoted workers and have higher levels of experience and education. The hires are also promoted faster. I further find that workers who are promoted and transferred at the same time have worse performance than other internal movers.
    No preview · Article · Feb 2011 · Administrative Science Quarterly
  • Source
    • "Similarly, interview methodologies tend to give lower validities for more complex jobs, despite the fact that interview results typically have moderate to high correlations with general mental ability (Berry, Sackett, & Landers, 2007; Huffcutt, Conway, Roth, & Klehe, 2004). These findings raise doubts regarding the moderating effect of Data, People, & Things job complexity on general mental ability validity coefficients (e.g., Bertua et al., 2005; Wilk & Sackett, 1996; Sturman, Cheramie, & Cashen, 2005). One of the most problematic issues is that complexity is an aggregated estimate of a number of job characteristics. "
    [Show abstract] [Hide abstract]
    ABSTRACT: This paper develops synthetic validity estimates based on a meta-analytic-weighted least squares (WLS) approach to job component validity (JCV), using position analysis questionnaire (PAQ) estimates of job characteristics, and the Data, People, & Things ratings from the Dictionary of Occupational Titles as indices of job complexity. For the general aptitude test battery database of 40,487 employees, nine validity coefficients were estimated for 192 positions. The predicted validities from the WLS approach had lower estimated variability than would be obtained from either the classic JCV approach or local criterion-related validity studies. Data, People, & Things summary ratings did not consistently moderate validity coefficients, whereas the PAQ data did moderate validity coefficients. In sum, these results suggest that synthetic validity procedures should incorporate a WLS regression approach. Moreover, researchers should consider a comprehensive set of job characteristics when considering job complexity rather than a single aggregated index.
    Full-text · Article · Aug 2009 · Personnel Psychology
Show more