No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Further considerations of evaluation competencies in Taiwan
Abstract
A list of evaluator competencies (Stevahn, King, Ghere, & Minnema, 2005) was adapted to fit the Taiwanese context by Lee, Altschuld, & Lee (2012). It was studied as to how it generalized to a large sample in Taiwan. Likert and Fuzzy surveys with needs assessment formats (importance and competence) were mailed via random assignment to two groups of participants. The questions for the study were: do the modified competencies relate country-wide to Taiwan, did the investigation uncover training needs for evaluators, and were there convergent rating patterns across the two forms of the instrument? The results supported a fit of the modified competencies to the context and convergent validity was observed but strong competency needs were not apparent. Reasons for the findings and implications for future research are discussed.
... Due to where the workers were in their careers and job understandings, the obtained item distributions were restricted in range which in turn affected the marginal expectations in PRE calculations. Analogously, in a needs assessment we did in 2007 (Lee et al., 2007) and in 2013 (Lee et al., 2013), similar results were found. This may not happen in all needs assessments and would be a function of the specific group of respondents, but it is thought that limited range is in many and affects the meaning of the data. ...
Surveys are frequently employed in needs assessment to collect information about gaps (the needs) in what is and what should be conditions. Double-scale Likert-type instruments are routinely used for this purpose. Although in accord with the discrepancy definition of need, the quality of such measures is being questioned to the point of suggesting that the results are not to be trusted. Eight factors supporting that proposition are described with explanations of how they operate. Literature-based examples are provided for improving surveys with double scales especially as they relate to attenuating the effects of the factors. Lastly, lessons learned are offered with a call for more research into this issue in assessing needs.
In the last two decades, evaluator competency frameworks have become ubiquitous. Many have been developed by reviewing the literature and engaging in various research processes to ask evaluation practitioners what they do, in order to determine what evaluators should know and be able to do. Due to this practice focused approach, the underlying philosophical and theoretical assumptions of competencies are rarely questioned. In this article, the authors explore this territory by categorising competencies using two taxonomies. Mertens and Wilson’s 2018 Program evaluation theory and practice provides a philosophical framework, and Schwandt & Gates’ 2021 Evaluating and valuing in social research provides a theoretical framework. The authors apply these frameworks to the three evaluation specific domains of the Australian Evaluation Society (AES) Evaluators’ Professional Learning Competency Framework as refined for the Learn Evaluation Assessment Platform in 2020. Findings of this exploratory study suggest that the AES ’ conception of evaluator competencies aligns primarily with Mertens and Wilson’s pragmatist/neopragmatist paradigm and Schwandt and Gates ’ expanded frame. The authors discuss implications of the results and propose ideas for further improvement and testing of this research approach.
تأثير الأداء السياقي كمتغير وسيط على العلاقة بين تمكين العاملين
والجدارات الاستراتيجية: دراسة مقارنة
هدف البحث إلى تناول متغيرات ثلاثة بالبحث من حيث المفهوم، و الأبعاد وهى متغيرات التمكين كمتغير مستقل و الأداء السياقي كمتغير وسيط و الجدارات الاستراتيجية كمتغير تابع. ثم التوصل إلى علاقات الارتباط والتأثير المتبادل بين أبعاد المتغيرات الثلاث.
تمثل المتغير المستقل في أبعاد التمكين ، و الذي تم دراسته في ضوء بُعدى الإستقلالية، و نشر و المشاركة في المعلومات.
أما المتغير الوسيط فتمثل في الأداء السياقي ، و الذي يتكون من بعدي الإلتزام التنظيمي والمواطنة التنظيمية. و يتكون بُعد الإلتزام التنظيمي من ثلاثة عناصر هى الإلتزام الإستمراري والإلتزام العاطفي و الإلتزام المعياري، أما بُعد المواطنة التنظيمية فيتكون من خمسة عناصر عبارة عن الإيثار، و الكياسة، و الروح الرياضية ، و الطاعة العامة ، و السلوك الحضارى.
وتمثل الجدارات الإستراتيجية المتغير التابع ، وتتضح من خلال توافر الرؤية المشتركة، والتعاون، والتمكين، و الإبداع.
توصل البحث إلى توافر متغيرات البحث بدرجة مرتفعة في منظمات القطاع الخاص مقارنة بمنظمات قطاع الأعمال العام. كذلك توجد علاقة ارتباط ذات دلالة احصائية بين أبعاد التمكين و أبعاد الأداء السياقي، وكذلك بين التمكين و أبعاد الجدارات الإستراتيجية من ناحية، و بين أبعاد الأداء السياقي و الجدارات الإستراتيجية من ناحية أخرى، بالإ ضافة إلى علاقة ارتباط ذات دلالة احصائية بين المتغيرات الثلاث .
و قدمت الباحثة مجموعة من التوصيات تتعلق بتقديم برامج تدريبية للعاملين بالمنظمات بصفة عامة ومنظمات قطاع الأعمال العام بصفة خاصة تكون موجهة بالتمكين و الأداء السياقي لتحقيق الجدارات الإستراتيجية بالمنظمات على إختلاف أنواعها.
Influence of Contextual Performance as a Mediator Variable between Employees ' empowerment and Strategic Competencies: a Comparative Study
Abstract
Purpose –Organizations in public and private sector have various behaviors that support initiating strategic competencies, empowerment and contextual performance. Thus empowerment has significant effect on contextual performance, and also has a significant effect on strategic competencies creation processes. Then empowerment dimensions (dependent and information share and publicity) create contextual performance in the public and private organizations. Organizational commitment and organizational citizenship comprise contextual performance. Each empowerment and contextual performance affects creating strategic competencies dimensions. Vision, corporate, empowerment, and creative are the dimensions of strategic competencies. The purpose of this study is to investigate the Influence of Contextual Performance as a Mediator Variable between Employees ' empowerment and Strategic Competencies through a comparative study between organizations in public and private sector.
Methodology ــUsing survey approach, the data collected from 270 respondents selected from various organizations, these organizations divided into public and private sectors ; that is to make comparisons between these sectors to define the variance between variables affect strategic competencies, and the validation of empowerment, contextual performance, and strategic competencies in different sectors on the other hand. Empowerment is a independent variable, contextual performance is a mediator variable , and strategic competencies is a dependent variable.
Findings – This study found that empowerment dimensions affect contextual performance dimensions, which play a critical role in creating strategic competencies in public and private organizations. There is a strong positive correlation between research variables.
Originality– This study not only contributes to empowerment, contextual performance, and strategic competencies by identifying both their concept and dimensions also is of importance to the three variables literature by demonstrating the proper effect of empowerment and contextual performance on the strategic competencies dimensions between organizations in various sectors which divided to public and private sector.
Recommendations-Researcher not only recommended organizing training program oriented with the core of empowerment and contextual performance to enhance being of strategic competencies, but also proposed future studies related to the research subjective.
Keywords Empowerment, contextual performance, strategic competencies, organizational citizenship, organizational commitment, continuous commitment, affective commitment, and normative commitment.
The current evaluation literature tends to conceptualize evaluator roles as a single, overarching orientation toward an evaluation, an orientation largely driven by evaluation methods, models, or stakeholder orientations. Roles identified range from a social transformer or a neutral social scientist to that of an educator or even a power merchant. We argue that these single, broadly construed role orientations do not reflect the multiple roles evaluators actually assume as they complete the activities encompassing an external evaluation. In contrast to the current literature, this article suggests that typical evaluation activities create functional demands on evaluators, and that evaluators respond to these demands through a limited number of specified evaluator roles. This depiction of a set of specific multiple evaluator roles, generated in response to particular evaluation activities and their associated demands, has implications regarding how evaluation is conceptualized, practiced, and studied. This article concludes with a discussion of these implications.
"This paper advocates a validational process utilizing a matrix of intercorrelations among tests representing at least two traits, each measured by at least two methods. Measures of the same trait should correlate higher with each other than they do with measures of different traits involving separate methods. Ideally, these validity values should also be higher than the correlations among different traits measure by the same method." Examples from the literature are described as well as problems in the application of the technique. 36 refs. (PsycINFO Database Record (c) 2010 APA, all rights reserved)
In this article we revisit the comprehensive taxonomy of Essential Competencies for Program Evaluators and explore its utility in university-based evaluation training programs. We begin by briefly summarizing the development of the taxonomy, then elaborate on how it can be used to enhance evaluation training through six decision points in graduate degree or certificate programs. We then discuss the challenges of credentialing/licensure and accreditation within the field of program evaluation and end with a proposal for the development of standards for program evaluation training programs.
This article explores various evaluator roles and strategies that have the potential to increase the likelihood that the evaluation process will have an influence on an organization and its members. These roles are (1) educator, (2) consultant, (3) facilitator, and (4) counselor. A brief case study presents the discussion dynamics of an evaluation workgroup. This workgroup resulted in programmatic changes based primarily on the discussion of developing indicators for measuring student outcomes. Practical implications for evaluators seeking to enhance process influence may involve the evaluator using a "client-centered" approach. Facilitating collaborative, open environments and managing discussion dynamics are fundamental in establishing a community of organizational learners. These dynamics may take the form of both task-focused and task-related discussions.
This article examines evaluation field building in South Asia and the role of international collaboration in this field building. The article explores aspects of the supply and demand of development evaluation and the political, historical, and systemic factors that bridge or block evaluation use. The article calls for and suggests elements to include in constructing a framework for evaluation field building, including the role of civil society, evaluation leadership, the formation and reformation of evaluation communities of practice and professional associations, formal evaluation curriculum and training, iterative applied approaches to capacity building, and fostering a culture of evaluation quality and use. Although the focus of the article is on development evaluation in South Asia, the questions explored are pertinent to other evaluation sectors and regions. The article flags the absence of literature on the state of evaluation in South Asia, and, given the limited evidence, develops hypothesis and questions for further probing and debate.
This article discusses an exploratory study designed to determine the extent to which evaluation professionals, representing diverse backgrounds and approaches, could reach agreement on a proposed taxonomy of essential evaluator competencies. Participants were 31 diverse individuals in the field of program evaluation in the greater Minneapolis-St. Paul, Minnesota area who systematically engaged in a Multi-Attribute Consensus Reaching process. Both quantitative and qualitative results predominantly indicated consensus on more than three-fourths of the proposed competencies. Areas of disagreement reflected the role- and context-specific nature of evaluation practice.
No self-report measure of cultural competence currently exists in program evaluation. Adapting items from cultural competence measures in fields such as counseling and nursing, the researchers developed the Cultural Competence of Program Evaluators (CCPE) self-report scale. The goals of this study were to validate the CCPE and to assess differences in level of cultural competence among program evaluators based on various demographic variables. The sample consisted of 174 evaluators. Principal components analyses revealed three factors of the CCPE: cultural knowledge, cultural skills, and cultural awareness. The overall alpha of the CCPE was .88, and convergent validity was established via significant positive correlations between the CCPE and the Multicultural Counseling Inventory (MCI). Additionally, individuals who had received cultural competence training scored significantly higher on the CCPE, and receipt of cultural competence training was a significant predictor of scores on the CCPE.
This paper shows how the socialization process (viewed by role theorists as a “role-person merger”) can be represented and analyzed as a fuzzy (dynamical) system. Given a self-concept (a fuzzy set of roles) and social norms (logical implications from roles to actions), an individual infers actions (through approximate reasoning). Given these actions, alters make biased attributions (about the roles in the individual's self-concept) that are gradually internalized by the individual. This feedback loop creates a fuzzy ystem (a role-space vectorfield) that generates a set of stable long-run selves (role-space attractors). I illustrate this general “endogenous-self” framework with a model based loosely on Tally's Corner; the analysis examines how the individual's long-run self-concept is influenced by a constraint on employment opportunity.
This article explores various evaluator roles and strategies that have the potential to increase the likelihood that the evaluation process will have an influence on an organization and its members. These roles are (1) educator, (2) consultant, (3) facilitator, and (4) counselor. A brief case study presents the discussion dynamics of an evaluation workgroup. This workgroup resulted in pro-grammatic changes based primarily on the discussion of developing indicators for measuring student outcomes. Practical implications for evaluators seeking to enhance process influence may involve the evaluator using a "client-centered" approach. Facilitating collaborative, open environments and managing discussion dynamics are fundamental in establishing a commu-nity of organizational learners. These dynamics may take the form of both task-focused and task-related discussions.
This article presents a comprehensive taxonomy of essential competencies for program evaluators. First, the authors provide a rationale for developing evaluator competencies, along with a brief history of the initial development and validation of the taxonomy of essential evaluator competencies in King, Stevahn, Ghere, and Minnema (2001). Second, they present a revised version of that taxonomy and describe the revision process. Third, a crosswalk accompanying the taxonomy indicates which competencies address standards, principles, and skills endorsed by major evaluation associations in North America. Finally, the authors identify future needs related to the taxonomy, including the need for validation research, a shared understanding of terms, and the construction of descriptive rubrics for assessing competence.
What are six distinct aspects of construct validation? How do these aspects apply to performance assessment? Are the consequences of performance assessment on teaching and learning relevant to construct validation?
In our dynamic social world, a premium is placed on the individual's ability to innovate and to change (Giddens 1984; Sewell 1992; Simmel 1955). Yet traditional role theory has difficulty accounting for innovation, leaving unanswered the question of how individual level negotiations affect social-structural processes (see Callero 1994). This study addresses this tension by linking role theory with social cognition. By positioning behavior and cognition as two interrelated continuums, I stretch the meaning of role enactment to include 4 role typologies. I utilize these typologies as a heuristic to chart the processes through which individuals adapt to and affect a role performance over time. I conclude by outlining how sociocognitive role typologies aid social researchers in accounting for individual efficacy in response to social-structural situations.
Training is critical for the growth of the evaluation profession as more qualified and competent individuals will be required to meet the increased demand for evaluation. This paper deals with the training needs of professionals and is based on a part of a study on the status and challenges of educational program evaluation in the Asia-Pacific region. A web-based Delphi was used with a needs assessment framework for examining the current nature of training, what should be done, and discrepancies (gaps) between the two. The results should be of value for academic training programs and those in non-academic settings.
Essential evaluator competencies as identified by Stevahn, King, Ghere, and Minnema (2005) were studied in regard to how well they generalize to an Asian (Taiwan) context. A fuzzy Delphi survey with two iterations was used to collect data from 12 experts. While most competencies fit Taiwan, there were a number of unique ones. A complete set of results is provided along with the implications of the findings and what they might mean for evaluation in Taiwan particularly in relationship to the professionalization of evaluation.
Three median alpha-levels of a fuzzy number A are introduced. Two of them are based on the vertical and horizontal dimensions of A, one on the cardinality of A. Approximation of fuzzy numbers by alpha-cuts associated with the median alpha-levels is discussed. Percentile characterizations of alpha-cuts with respect to the selected properties of fuzzy numbers are proposed.
While educational program evaluation has become more important in recent years because of increasing governmental demands for accountability, little is known about the development of and issues in regard to this topic in the Asia-Pacific region. The findings from a Delphi study conducted in a number of relevant countries are provided in this article. Thirty panelists participated in three rounds of iterations to identify current status, challenges, and concerns in the near future. The results should be of value to educational evaluators, researchers, and decision makers.
A conversation with Michael Quinn Patton. The Evaluation Exchange Designing and constructing instruments for social research and evaluation Development and validation of the Cultural Competence of Program Evaluators (CCPE) self-report scale
- J Coffman
- Spring
- D Colton
- R W Covert
Coffman, J. (2002). A conversation with Michael Quinn Patton. The Evaluation Exchange, 8, 10–11, Spring. Colton, D., & Covert, R. W. (2007). Designing and constructing instruments for social research and evaluation. San Francisco: John Wiley & Sons, Inc. Dunaway, K., Morrow, J., & Porter, B. (2012). Development and validation of the Cultural Competence of Program Evaluators (CCPE) self-report scale. American Journal of Evaluation, 33(4), 496–514.
Utilization-focused evaluation in Africa. Lectures delivered to the inaugural conference of the African Evaluation Association
- M Q Patton
Patton, M. Q. (1999). Utilization-focused evaluation in Africa. Lectures delivered to the
inaugural conference of the African Evaluation Association September.
Utilization-focused evaluation Thousand Oaks, CA: Sage Evaluator competencies: Standards for the practice of evaluation in organizations
- M Q Patton
- D F Russ-Eft
- M J Bober
- I Teja
- M Foxon
- T A Koszalka
Patton, M. Q. (2008). Utilization-focused evaluation (4th ed.). Thousand Oaks, CA: Sage. Russ-Eft, D. F., Bober, M. J., de la Teja, I., Foxon, M., & Koszalka, T. A. (2008). Evaluator competencies: Standards for the practice of evaluation in organizations. San Francisco: Jossey-Bass.
Standards of validity and the validity of standards in performance assessment. Educational Measurement: Issues and Practice
- S Messick
Messick, S. (1995). Standards of validity and the validity of standards in performance assessment. Educational Measurement: Issues and Practice, 14(4), 5–8.
Toward better understanding of alternative evaluator roles
- M M Mark
Mark, M. M. (2002). Toward better understanding of alternative evaluator roles. In K. E.
Ryan & T. A. Schwandt (Eds.), Exploring evaluator role and identity (pp. 17-36).
Greenwich, CT: Information Age Publishing.
An initial study to develop instruments and validate the essential competencies for program evaluators (ECPE)
- Y Wilcox
Wilcox, Y. (2012). An initial study to develop instruments and validate the essential
competencies for program evaluators (ECPE). Minneapolis, MN: The University of
Minnesota (Unpublished doctoral dissertation).
A conversation with Michael Quinn Patton
- Coffman
Competencies for Canadian evaluation practice Accessed 03
Canadian Evaluation Society. (2009). Competencies for Canadian evaluation practice
Accessed 03.12.10 http://www.evaluationcanada.ca/txt/20090531_competen-cies_companion.pdf.
A conversation with Michael Quinn Patton. The Evaluation Exchange
- J Coffman
Coffman, J. (2002). A conversation with Michael Quinn Patton. The Evaluation Exchange,
8, 10-11, Spring.
Evaluator competencies: Standards for the practice of evaluation in organizations
- M Q Patton
- D F Russ-Eft
- M J Bober
- I De La Teja
- M Foxon
- T A Koszalka
Patton, M. Q. (2008). Utilization-focused evaluation (4th ed.). Thousand Oaks, CA: Sage.
Russ-Eft, D. F., Bober, M. J., de la Teja, I., Foxon, M., & Koszalka, T. A. (2008). Evaluator
competencies: Standards for the practice of evaluation in organizations. San Francisco:
Jossey-Bass.
is an Associate Professor at National Taiwan Normal University in Taiwan. She has published and presented in the area of needs assessment, program evaluation, and evaluation in STEM education
- Yi-Fang Lee
Yi-Fang Lee, Ph.D., is an Associate Professor at National Taiwan Normal University in
Taiwan. She has published and presented in the area of needs assessment, program
evaluation, and evaluation in STEM education.
is a Professor Emeritus at The Ohio State University. He has presented and published extensively on evaluation topics, especially on needs assessment and the evaluation of science and technology education
- James W Altschuld
James W. Altschuld, Ph.D., is a Professor Emeritus at The Ohio State University. He has
presented and published extensively on evaluation topics, especially on needs assessment and the evaluation of science and technology education.
is a Professor and the President of National United University, Taiwan. He has participated in a variety of institutional and program evaluation for more than 30 years
- Sheng Steven Lung
- Ph D Lee
Lung-Sheng Steven Lee, Ph.D., is a Professor and the President of National United
University, Taiwan. He has participated in a variety of institutional and program
evaluation for more than 30 years.