Applying mixed methods under the framework of theory‐driven evaluations
ABSTRACT The application of mixed methods under the framework of theory-driven evaluations can minimize the potential tension and conflict of mixing qualitative and quantitative methods, as well as compensate for each method's weaknesses. Mixed methods should not be applied indiscriminantly, however, but rather contingently under particular conditions as described in this chapter.
- SourceAvailable from: msera.org
[Show abstract] [Hide abstract]
- "Due to the comprehensiveness of this type of evaluation, it usually requires application of both qualitative and quantitative approaches to evaluate the action model and the change model as shown in Figure 1. For example, the evaluation of a garbage reduction program in Taiwan (Chen et al., 1997) was an integrative process/outcome evaluation. Garbage was collected by government sanitation workers on a daily basis in Taiwan. "
ABSTRACT: The purpose of this article is to discuss the conceptual framework and strategies used in theory-driven evaluations in relation to mixed methods research and to explore the opportunities and challenges emerging from theory–driven applications. Theory-driven evaluations have frequently applied mixed methods in the past, and these experiences provide some insightful information for future development of mixed methods. In theory-driven evaluations, the application of mixed methods is justified and applied under a conceptual framework called program theory. The conceptual framework of program theory provides a plan and agenda for mixed methods to work collaboratively and de-emphasizes their differences and incompatibilities. Based upon the conceptual framework of program theory, this article provides several strategies for combining qualitative and quantitative methods in theory-driven evaluations. Procedures in applying these strategies are systematically illustrated. Finally, this article discusses challenging issues related to the future development of mixed methods, such as implications of the use of pure versus modified forms of mixed methods and the advocacy of mixed methods research as a "method" paradigm versus a "method use" paradigm. Mixed methods research is the systematic combination of qualitative and quantitative methods in research or evaluation. There has been a growing interest in this topic (Johnson & Onwuegbuzie, 2004). Advocates have argued that mixed methods can overcome weaknesses of a single (qualitative or quantitative) method (Greene & Caracelli, 1997; Howe, 1988; Johnson & Onwuegbuzie, 2004; Sechrest & Sidana, 1995). Greene and Caracelli (1997) provided the following major justifications for mixed methods: (a) triangulation: combining qualitative and quantitative methods to study the same phenomenon in order to gain convergence and increase validity (Denzin, 1970), (b) compensatory: using strengths of each method to overcome the weaknesses of the other to enrich the study of a phenomenon, and (c) expansion: using each method to obtain a fuller picture of a phenomenon. Quantitative and qualitative purists, however, view these two approaches as being based upon incompatible premises and techniques, and argue that mixing methods is neither meaningful nor valuable to pursue (Guba, 1990). Johnson and Onwuegbuzie (2004) have argued that there are some commonalities between quantitative and qualitative methods, and mixed methods research can narrow the divide between quantitative and qualitative researchers, enhancing the quality of a study. So far, many discussions or debates about mixed methods have been concentrated on philosophical or methodological issues. The discussion or development of mixed methods also can benefit from experiences based on the application of mixed methods in the field. Practical feedback can provide insightful information about strategies used in combining different methods, and the opportunities and challenges faced in such applications. This type of information could energize the future development of mixed methods. Theory-driven evaluations have frequently applied mixed methods in the past (Chen, 1990, 1997, 2005). The purpose of this article is to discuss some practical experiences of using mixed methods in theory-driven evaluations. More specifically, in this article, I will discuss the conceptual framework and strategies used in theory-driven evaluation that apply mixed methods and the opportunities and challenges emerging from such applications.
[Show abstract] [Hide abstract]
- "Finally, data collection efforts were based on the premise that no single data source was bias-free or a completely accurate representation of reality. Evaluation plans were designed to specifically encourage each grantee to use multiple methodological strategies with different strengths and weaknesses to answer evaluation questions (Chen, 1997; Cook, 1985; Donaldson, 2003a; Shadish, 1993). A special effort was made to understand cross-cultural and language concerns so that the methodologies employed were sensitive enough to detect program effects when they existed. "
ABSTRACT: Despite the burgeoning literature explicating the benefits of theory-driven program development and evaluation, there remains a strong need for practical advice, written insights and experiences, and examples from evaluation practice illustrating how to implement this approach. The purpose of this paper is to move the field closer to a concrete understanding of the strengths, limitations, and challenges of implementing theory-driven program development and evaluation in modern human service organizations. This is accomplished by describing the evaluation process, resulting program theories, and lessons learned from the evaluation a five year, $20 million statewide Work and Health Initiative.Evaluation and Program Planning 11/2003; 26(4-26):355-366. DOI:10.1016/S0149-7189(03)00052-1 · 0.90 Impact Factor
- "triangulation), all argue that using multiple methods offers many advantages and few costs 1 (Rossman and Wilson 1985; Chen 1997; Tashakkori and Teddlie 1998). Newman and Benz, 1998). "