Melvin M Mark

Pennsylvania State University | Penn State · Department of Psychology

Evaluation policy involves the dictates that guide the planning, conduct, and use of evaluation in any organization. It is – or at least should be – a central concern to those involved with evaluation. Evaluation policy shapes what evaluation practice looks like, while enabling or constraining what it can accomplish. This chapter offers a brief and...

We highlight some key issues regarding evaluation policy, including themes that emerged across chapters of this volume. These topics include what an evaluation policy is, the kind of content that evaluation policies can have, learning agendas (which are an increasingly common component of evaluation policies, especially at the U.S. federal level),...

Background Stakeholders are often involved in evaluation, such as in the selection of specific research questions and the interpretation of results. Except for the topic of whether stakeholder involvement increases use, a paucity of research exists to guide practice regarding stakeholders. Objectives We address two questions: (1) If a third-party...

Premised on the idea that evaluators should be familiar with a range of approaches to program modifications, I review several existing approaches and then describe another, less well-recognized option. In this newer option, evaluators work with others to identify potentially needed adaptations for select program aspects in advance. In describing th...

Frameworks for classifying adaptations and modifications to an intervention have been developed and continue to evolve. These frameworks capture a range of attributes of adaptations and modifications. However, the size and the scope of a change, relative to the intervention’s overall size and structure, are not currently included. This omission can...

Melvin M. Mark considers possible connections between his youthful informal evaluations and his professional interests in theory-driven evaluation and contingent selection of evaluation methods. See the interview document here: http://onlinelibrary.wiley.com/wol1/doi/10.1002/ev.20288/suppinfo. Read only. This should not be used in any form without...

George Grob presented the fifth and final Eleanor Chelimsky Forum address at the 2017 annual meeting of the Eastern Evaluation Research Society. In this commentary, I respond to several points that George raises in the American Journal of Evaluation paper based on that address. An overarching theme of my comments involves the potential for strength...

Inspired by a set of evaluators building evaluation capacity in China, India, and Chile, this paper applies a conceptualization of evaluation influence to evaluations and evaluation capacity building related to inequities. © 2017 Wiley Periodicals, Inc., and the American Evaluation Association

Must investigations of treatment effects adopt a logic of causal investigation that defaults to the design and application of randomized control trials in education? Does the language of scientific research impel this too for intervention studies? In this brief response to Scriven’s chapter, we explore the pragmatics of research planning to identif...

Evaluation theories can be tested in various ways. One approach, the experimental analogue study, is described and illustrated in this article. The approach is presented as a method worthy to use in the pursuit of what Alkin and others have called descriptive evaluation theory. Drawing on analogue studies conducted by the first author, we illustrat...

Time-series data can be applied to an array of different research purposes and analyzed with a variety of statistical techniques. In the present chapter, we focus primarily on the interrupted time-series design and on autoregressive integrated moving average (ARIMA) statistical analyses in a social science context. Advantages of interrupted time-se...

We comment on a set of papers in this journal issue that have examined three evaluation theories by constructing a logic model for each and conducting a set of comparative content analyses. We consider the set of papers as an instance of research on evaluation. For instance, we question the extent to which linear logic models convey the contingent...

Attention to evaluation quality is commonplace, even if sometimes implicit. Drawing on her 2010 Presidential Address to the American Evaluation Association, Leslie Cooksy suggests that evaluation quality depends, at least in part, on the intersection of three factors: (a) evaluator competency, (b) aspects of the evaluation environment or context, a...

This chapter discusses the concept of validity as it applies to outcome evaluation. We address the historical adoption and contributions of the Campbellian typology to evaluation. We also discuss related criticisms and controversies and address future directions. © Wiley Periodicals, Inc., and the American Evaluation Association.

The author addresses the issue of generalizability as it arises in outcome evaluations. Methods for enhancing knowledge about generalizability in the Campbellian tradition are briefly reviewed. Recommendations are made for future practice and promising areas of future advancement regarding understanding generalizability in outcome evaluation. © Wil...

Evaluation is typically carried out with the intention of making a difference in the understandings and actions of stakeholders and decision makers. The author provides a general review of the concepts of evaluation “use,” evaluation “influence,” and “influence pathways,” with connections to multisite evaluations. The study of evaluation influence...

The following values have no corresponding Zotero field: PB - Wiley Subscription Services, Inc., A Wiley Company

This compelling work brings together leading social psychologists and evaluators to explore the intersection of these two fields and how their theory, practices, and research findings can enhance each other. An ideal professional reference or student text, the book examines how social psychological knowledge can serve as the basis for theory-driven...

Happy people often fail to elaborate on persuasive arguments, while people in sad moods tend to scrutinise messages in greater detail. According to some motivational accounts, however, happy people will elaborate a message if they believe it might maintain their positive mood. The present research extends this reasoning by demonstrating that happy...

Evaluation policy is of considerable importance, especially in relation to the limited amount of attention it receives as a general topic in the mainstream evaluation literature. Evaluation policies matter for several reasons, among them that they can profoundly affect evaluation practice, they underlie many recent and current controversies about e...

Three issues for evaluation policy and practice are described: evaluation policy dimensions, evaluation policy instruments, and the political and economic environment for evaluation policy. Selected future directions are outlined, including the need to describe the evaluation policy landscape, further articulate an evaluation policy taxonomy, and d...

At the twentieth birthday of the American Evaluation Association, its twentieth-year president looks back at the growth of the association and the field of evaluation, and identifies where AEA and evaluation might go in the future.

Borrowing from the negotiation literature, we tested 2 factors that might improve stakeholder dialogue in program and policy evaluation. Undergraduate stakeholders (61 pairs) engaged in dialogue about their universities' alcohol policies. Pairs were randomly assigned to levels of accountability audience and dialogue structure. The audience for the...

The following is excerpted from the introduction to Volume 25, Issue 4 of the American Journal of Evaluation, by former AJE editor Dr. Melvin M. Mark. It is reprinted here with permission from Dr. Mark; AJE’s current editor, Dr. Robin Miller; and the American Evaluation Association (AEA).The American Journal of Evaluation is the official journal of...

Some of the essential points of applying emergent realist theory to practice are sketched in this chapter. First a running conversation is offered to explain emergent realist evaluation; then an example pre-proposal highlights important aspects of practice.

This chapter focuses on ethical issues that can arise in the collection and analysis of data in evaluations. We move beyond common ethical issues in data collection and analysis, such as informed consent and coercion, to address four issues: the application of cost-benefit thinking to judgments about research ethics, the quality of research design...

Social programming is an evolved mechanism for addressing social needs (among other things). Evaluation focuses largely on social programs. We review selected aspects of the complex dynamic, political, and contextual factors that influence social programming, and from these we draw conclusions about the conduct of emergent realist evaluation.

Validity typologies can enhance our understanding of quasi-experiments, but they can also serve as blinders. We can go beyond the primary concerns of validity typologies by studying causal process and by being explicit about the reasoning that underlies our inferences.

The benefits of program theory will not fully accrue unless program theory is tested; alternative approaches to such testing are discussed.

Different models can be followed in using multiple methods, and researchers need to be aware of the benefits associated with different approaches.

The use of multiple methods can result in an inferential challenge, though there are strategies to strengthen our inferences.

Past literature has identified several putative precursors of use, as well as alternative forms of use. However, important shortcomings still exist in previous work on use. In particular, inadequate attention has been given to the underlying processes that may mediate the effects of evaluation on attitude and action. In essence, a key part of the t...

Although use is a core construct in the field of evaluation, neither the change processes through which evaluation affects attitudes, beliefs, and actions, nor the interim outcomes that lie between the evaluation and its ultimate goal—social betterment—have been sufficiently developed. We draw a number of these change mechanisms, such as justificat...

We examined a phenomenon related to hindsight bias, specifically, retrospective judgements about the foreseeability of an outcome. We predicted that negative, self-relevant outcomes would be judged as less foreseeable by the recipient of the outcome than by others, unlike either positive outcomes or outcomes that are not self-relevant. In the conte...

Previous research demonstrates that people use their mood as information when making a variety of judgments. The present research examines the extent to which people use their current mood as information when making attributions to discrimination. Women were given a positive or negative mood induction and either provided with an external attributio...

Using Christie's research as an example, the authors describe a variety of forms that a more evidence-based approach to evaluation theory could take and offer some suggestions to help increase the amount and impact of evidence in evaluation theory.

This study tested the effectiveness of a multicomponent after-school substance abuse prevention program for high-risk second- and third-grade children implemented collaboratively by Boys & Girls Clubs and their local schools. The 2-year program was designed to reduce risk factors predictive of later substance abuse, and to enhance protective factor...

This study tested the effectiveness of a multicomponent after‐school substance abuse prevention program for high‐risk second‐ and third‐grade children implemented collaboratively by Boys & Girls Clubs and their local schools. The 2‐year program was designed to reduce risk factors predictive of later substance abuse, and to enhance protective factor...

Part of the controversy surrounding this year's presidential election in the United States concerns the potential for systematic bias in the ballot-card format — could the butterfly ballot used in Palm Beach County, Florida, have led to confusion and caused people who had intended to vote for Al Gore to vote for Pat Buchanan by mistake? Here we sho...

Articles should deal with topics applicable to the broad field of program evaluation. Articles may deal with evaluation practice or theory, but if the latter, implications for practicing evaluators should be clearly identified. Examples of contributions include, but are not limited to, reviews of new developments in evaluation and descriptions of a...

Evaluation has been beset with serious divisions, including the paradigm wars and the seeming segmentation of evaluation practice into distinct evaluation theories and approaches. In this paper, we describe key aspects of an integrative framework that may help evaluators move beyond such divisions. We offer a new scheme for categorizing evaluation...

Recent Supreme Court decisions suggest that judges should conduct a gatekeeping inquiry before admitting testimony that is based on psychology or other social sciences. Perspectives from other areas of applied social research may be of use to psychologists and others as they consider these decisions by the Court. A view of research as "assisted sen...

Recent Supreme Court decisions suggest that judges should conduct a gatekeeping inquiry before admitting testimony that is based on psychology or other social sciences. Perspectives from other areas of applied social research may be of use to psychologists and others as they consider these decisions by the Court. A view of research as "assisted sen...

Realism, as a foundational philosophy and as an applied approach to inquiry, is increasingly visible in social science. Ray Pawson and Nick Tilley have provided an important service to the field of evaluation by applying some of the basic concepts of realism to frame an approach that they refer to as ‘realistic evaluation’. As they note, their cont...

In the regression-discontinuity (RD) design, scores on a pretreatment measure (assignment variable) determine assignment to groups such that respondents above a certain score are assigned to one group, whereas respondents below the score are assigned to another group (e.g., candidates for promotion above a cutoff point on a criterion dimension are...

Realism offers an alternative to the traditional positivist and constructivist paradigms. The principles of this approach are described here, as are its implications for knowledge construction.

There is a need for rigorous definitional and empirical specification of what is meant by firm learning and the growing number of variants on this important theme. Based on a review of the literature on firm learning, the authors present a series of propositions about the state of knowledge regarding organizations' learning behaviors and capabiliti...

We argue that external cues provide affective information that influences processing strategy and, therefore, examination performance. Participants completed 2 supposedly different forms of a midterm examination; in fact, randomly assigned participants completed identical midterm examinations printed on either red or blue paper. Blue paper led to s...

The five articles of this sourcebook, organized around the five-component framework for evaluation described by W. Shadish, T. Cook, and L. Leviton (1991), present a new theory of realist evaluation that captures the sensemaking contributions of postpositivism and the sensitivity to values from the constructivist traditions. (SLD)

A review of qualitative methods used in a predominantly quantitative evaluation indicated a variety of roles for such a mixing of methods, including framing initial questions and method choices, revising evaluation questions during the inquiry, assessing the validity of measures, assessing adaptations in program implementation, and gauging the prop...

We argue that environmental cues provide affective information that directly influences processing strategy, with positively valenced (i.e., happy) cues leading to nonsystematic processing and negatively valenced (i.e., sad) cues leading to systematic processing. Two studies addressed this issue. In Study 1, participants were exposed to a set of pr...

The study evaluated the incremental effect of a 3-year drug prevention program for high-risk early adolescents, combined with monthly youth activities and parent involvement (Family Advocacy Network [FAN] Club group) relative to: (a) the 3-year drug prevention program with monthly youth activities but without parent involvement (Prevention Plus You...

J. L. Morgan, K. M. Bonamo, and L. L. Travis (see record 1995-24603-001) applied synchronous and time-series regression techniques to observational data to detect effects of recasted error correction on children's emerging grammar. Results showed that recasts did not facilitate learning but actually impeded it. In this study, a formal modeling pro...

J. L. Morgan, K. M. Bonamo, and L. L. Travis (see record 1995-24603-001 ) applied synchronous and time-series regression techniques to observational data to detect effects of recasted error correction on children's emerging grammar. Results showed that recasts did not facilitate learning but actually impeded it. In this study, a formal modeling pro...

Research indicates that effortful performances are reduced when participants cannot be evaluated, relative to when they can be evaluated. It was hypothesized that mood would interact with goals to attenuate such reductions in performance. As predicted, when participants' goal was to do as much as they could, those in negative moods put forth more e...

The establishment of a national system of federal-state manufacturing modernization centers to serve small- and medium-sized manufacturing systems is a major experiment in the US. The system's rapid expansion has made it impossible to begin with an experimental design that would systematically permit tests of the variables and relationships likely...

Three studies investigated the influence of social- and self-evaluative motives on self-handicapping and performance. In each study, efficacy expectancies were manipulated by varying the difficulty of a preliminary task, and social- and self-evaluation were manipulated orthogonally. In Study 1, participants who self-handicapped performed better tha...

Under commonly observed conditions, happy subjects appear to process information in a relatively passive or nonsystematic, less detailed manner and rely on peripheral cues and heuristics in judgement, whereas sad subjects appear to process in a more active or systematic, more detailed manner. Happy subjects should therefore display less accuracy on...

Meta-analytic methods were used to synthesize the results of published randomized, controlled-outcome studies of psychosocial interventions with adult cancer patients. Forty-five studies reporting 62 treatment-control comparisons were identified. Samples were predominantly White, female, and from the United States. Beneficial effect size ds were .2...

Meta-analytic methods were used to synthesize the results of published randomized, controlled-outcome studies of psychosocial interventions with adult cancer patients. Forty-five studies reporting 62 treatment–control comparisons were identified. Samples were predominantly White, female, and from the United States. Beneficial effect size d s were ....

This study evaluated an abstinence-only sexual activity prevention program, part of a multifocus program, Stay SMART, also aimed at preventing drug use among Boys & Girls Club youths. No effects were observed for virgins. Desired effects for sexual behavior and attitudes toward sexual activity were found for nonvirgins who participated in only Stay...