The Impact of Job Complexity and Performance Measurement on the Temporal Consistency, Stability, and Test-Retest Reliability of Employee Job Performance Ratings

Article · April 2005with55 Reads
DOI: 10.1037/0021-9010.90.2.269 · Source: PubMed
Abstract
Although research has shown that individual job performance changes over time, the extent of such changes is unknown. In this article, the authors define and distinguish between the concepts of temporal consistency, stability, and test-retest reliability when considering individual job performance ratings over time. Furthermore, the authors examine measurement type (i.e., subjective and objective measures) and job complexity in relation to temporal consistency, stability, and test-retest reliability. On the basis of meta-analytic results, the authors found that the test-retest reliability of these ratings ranged from .83 for subjective measures in low-complexity jobs to .50 for objective measures in high-complexity jobs. The stability of these ratings over a 1-year time lag ranged from .85 to .67. The analyses also reveal that correlations between performance measures decreased as the time interval between performance measurements increased, but the estimates approached values greater than zero.
    • Unstandardized coefficients are useful in terms of understanding practical importance because researchers can use unstandardized coefficients to generate meta-analytic effect size estimates under varying levels of the boundary condition. For example,Sturman et al. (2005)examined time as a boundary condition, with the expectation that the temporal consistency and stability of job performance ratings would decline as time between measurement episodes decreased. Therefore, they chose practically relevant time gaps and entered these values into their metaregression equation to provide actual estimates and accompanying confidence intervals of the temporal consistency and stability of job performance ratings at different time lags (see alsoSturman, 2003).
    [Show abstract] [Hide abstract] ABSTRACT: Understanding boundary conditions, or situations when relations between variables change depending on values of other variables, is critical for theory advancement and for providing guidance for practice. Metaregression is ideally suited to investigate boundary conditions because it provides information on the presence and strength of such conditions. In spite of its potential, results of our review of 63 metaregression articles published in the Journal of Management, Journal of Applied Psychology, Personnel Psychology, Journal of Management, Academy of Management Journal, and Strategic Management Journal uncovered a surprising lack of transparency, frequently implemented erroneous practices, and a lack of attention to important methodological choices. Results also suggest that many substantive conclusions are ambiguous at best and, unbeknownst to authors and readers, potentially misleading. Drawing from our review of the substantive literature as well as the latest statistical and methodological research, we offer evidence-based best-practice recommendations on how to conduct and report the results of a metaregression study. We offer recommendations on calculating statistical power and heterogeneity, choosing an appropriate model, testing boundary condition hypotheses, adjusting R² for known variance, explaining methodological choices, and reporting and interpreting model coefficients and other results. Also, we conducted two illustrative metaregression studies that incorporate all of our recommendations with accompanying syntax and data. Our recommendations can be used by authors, readers, journal editors, and reviewers wishing to conduct and evaluate metaregression studies, as well as practitioners interested in understanding conditions under which organizational practices are more or less likely to be effective.
    Full-text · Article · Jun 2017
    • To correct for unreliability in subjective ratings of performance, we used the 0.60 reliability estimate provided by Conway and Huffcutt (1997). For objective measures of performance, we used a reliability estimate of 0.61 (Sturman, Cheramie, & Cashen, 2005 ). For indicators of performance taken from administrative records (e.g., training grades, actual turnover), reliability was assumed to be 1.00.
    [Show abstract] [Hide abstract] ABSTRACT: Recently, there has been a growing interest in the study and use of vocational interests for predicting workplace behavior. The renewed attention to this topic is at least partially due to two recent meta-analyses (Nye, Su, Rounds, & Drasgow, 2012; Van Iddekinge, Roth, Putka, & Lanivich, 2011a) demonstrating the validity of interests for predicting job performance. Both studies came to the conclusion that interests predict performance but their results differed with respect to the validity of interest congruence. Although the congruence (or match) between an individual's interests and his or her work is particularly important for theories of vocational interest, there is some debate in the literature about the validity and utility of interest congruence for predicting work outcomes and the varying results reported in the two recent meta-analyses do little to resolve this issue. Therefore, the goal of the present study was to address these differences and the broader debate about interest congruence by conducting a more comprehensive meta-analysis of the validity of interest congruence for predicting job performance. An analysis of 92 studies and 1858 correlations suggested that interest congruence is a stronger predictor of performance outcomes than interest scores alone, with baseline correlations of 0.32 and 0.16, respectively. These results are discussed in the context of the broader person-environment fit literature and the implications for the interest literature and personnel selection research are discussed.
    Full-text · Article · Nov 2016
    • For repeated-measurement (withinsubject or crossover) designs, we considered the published lagged correlation of measures or, if not reported, estimated this with a conservative value of r = .50 (see meta-analysis of Sturman et al. 2005). To ensure independence of effect sizes, we used average effect size estimates for studies that reported multiple performance measures.
    [Show abstract] [Hide abstract] ABSTRACT: Within-shift rest breaks are important to prevent an accumulation of impairing short-term effects of strain over working time. In this meta-analysis (k = 11, N = 705), we investigated how supplementary, frequent short rest breaks affect task performance and strain. We found positive effects on quality (g = 0.23) and quantity (g = 0.12) measures of task performance. The mean reduction of working time due to rest breaks was 9.3%. Performance improvements occurred not at costs of higher strain. Thus, our study shows that both employees' performance and well-being benefits from scheduled within-shift breaks. We found no further effects of potentially moderating variables. Future research should examine the boundary conditions and underlying mechanisms of these effects.
    Full-text · Article · Nov 2016 · Pedagogika
    • drop-out za??naj?c?ch u?itel?: @BULLET USA 39?50 % (Darling-Hammond, 2000; Ingersoll, 2001; Ingersoll & Smith, 2003; Johnson & Birkeland, 2004; Smethem, 2007) @BULLET Velk? Brit?nie 35 % (Sturman, Cheramie & Cashen, 2005) @BULLET Austr?lie 20?50 % (Manuel, 2003; Buchanan et al., 2013) O jak? " pohyb " na pracovn?m trhu se jedn?, nazna?uj? data ze studie Johnsonov? et al. (2005, s. 5): v obdob?
    [Show abstract] [Hide abstract] ABSTRACT: Abstrakt: Jedním z faktorů ovlivňujících kvalitu vzdělávání je nedostatek učitelů, způso-bený mj. vzrůstající mírou ztráty kvalifi kovaných učitelů způsobené jejich odchodem z profese či přechodem na jinou školu, tzv. drop-outem. Nejohroženější profesní skupinou jsou přitom učitelé–novicové. Text nabízí literární přehled studií drop-outu začínajících učitelů v různých kulturních rámcích. Analýza studií byla zaměřena (1) na míru drop-outu začínajících učitelů a (2) na faktory, které drop-out způsobují, a to zejména na kontextuální proměnné – tzv. objektivní determinanty profesní dráhy v učitelství. Výsledky mohou být informativní pro českou odbornou veřejnost, pedagogický výzkum i vzdělávací politiku, které zatím drop-outu příliš pozornosti ne-věnují. Cílem článku je proto iniciovat odbornou diskusi a výzkum k problematice drop-outu začínajících učitelů u nás. Klíčová slova: literární přehled, učitelé a drop-out, začínající učitel, míra drop-outu, objek-tivní determinanty drop-outu.
    Full-text · Article · Oct 2016
    • Such an assumption of static influences of individual characteristics has also been shared in other areas, such as job performance (Sturman, 2007). This is probably one important reason why researchers examined or called for more research on dynamic job performance (e.g., Dalal, Bhave, & Fiset, 2014; Lievens, Ones, & Dilchert, 2009; Sturman, Cheramie, & Cashen, 2005; Thoresen, Bradley, Bliese, & Thoresen, 2004; Zyphur, Chaturvedi, & Arvey, 2008). Such a focus on change over time has also been emerging in other areas of applied psychology research, such as justice (e.g., Hausknecht, Sturman, & Roberson, 2011), extrinsic career success (e.g., Judge, Klinger, & Simon, 2010), and job satisfaction (e.g., Chen, Ployhart, Thomas, Anderson, & Bliese, 2011; Liu, Mitchell, Lee, Holtom, & Hinkin, 2012).
    File · Data · Aug 2016 · Pedagogika
    • Such an assumption of static influences of individual characteristics has also been shared in other areas, such as job performance (Sturman, 2007). This is probably one important reason why researchers examined or called for more research on dynamic job performance (e.g., Dalal, Bhave, & Fiset, 2014; Lievens, Ones, & Dilchert, 2009; Sturman, Cheramie, & Cashen, 2005; Thoresen, Bradley, Bliese, & Thoresen, 2004; Zyphur, Chaturvedi, & Arvey, 2008). Such a focus on change over time has also been emerging in other areas of applied psychology research, such as justice (e.g., Hausknecht, Sturman, & Roberson, 2011), extrinsic career success (e.g., Judge, Klinger, & Simon, 2010), and job satisfaction (e.g., Chen, Ployhart, Thomas, Anderson, & Bliese, 2011; Liu, Mitchell, Lee, Holtom, & Hinkin, 2012).
    File · Data · Aug 2016 · Pedagogika
Show more
Chapter
September 2015
    This chapter suggests that decision-makers should focus on the correlation between a single year's performance and a teacher's career performance. Despite the fact that value-added measures are unreliable by conventional standards and unstable over time, they are strong predictors of an individual teacher's career performance that can be used to improve decision making. The chapter studies the... [Show full abstract]
    Article
    January 2012
      The goal of personnel selection is to select employees who will succeed in a job, not just over the short-term, but over months and years. Yet, most research and theory in personnel selection does not explicitly consider the effect of time on personnel selection and training. The research that has been done has raised concerns about the utility of predictors used in selection related to... [Show full abstract]
      Article
      May 2010 · International Journal of Automation and Computing
        The oil and gas (O&G) industry on the Norwegian continental shelf (NCS) leads the world in terms of the number of subsea O&G installations. Ensuring the dependability of these assets is critical. Non-intrusive inspection, maintenance and repair (IMR) services are therefore needed to reduce risks. These services are planned and executed using a mono-hull offshore vessel complete with remotely... [Show full abstract]
        Article
        January 2010 · Journal of Applied Psychology · Impact Factor: 4.31
          The present two studies integrate and extend the literatures on dynamic performance, performance attributions, and rating purpose, making several important contributions. First, examining attributions of dynamic performance, Study 1 predicted that performance mean and trend would affect judged ratee ability and effort and that performance variation would affect locus of causality; both... [Show full abstract]
          Discover more