Article

Evaluation Research: An Overview

Authors:
To read the full-text of this research, you can request a copy directly from the author.

Abstract

Evaluation research can be defined as a type of study that uses standard social research methods for evaluative purposes, as a specific research methodology, and as an assessment process that employs special techniques unique to the evaluation of social programs. After the reasons for conducting evaluation research are discussed, the general principles and types are reviewed. Several evaluation methods are then presented, including input measurement, output/ performance measurement, impact/outcomes assessment, service quality assessment, process evaluation, benchmarking, standards, quantitative methods, qualitative methods, cost analysis, organizational effectiveness, program evaluation methods, and LIS-centered methods. Other aspects of evaluation research considered are the steps of planning and conducting an evaluation study and the measurement process, including the gathering of statistics and the use of data collection techniques. The process of data analysis and the evaluation report are also given attention. It is concluded that evaluation research should be a rigorous, systematic process that involves collecting data about organizations, processes, programs, services, and/or resources. Evaluation research should enhance knowledge and decision making and lead to practical applications. published or submitted for publication

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the author.

... According to Powell (2006), an evaluative study uses examinations and judgments towards the efficacy of achievement. The further argument claims that to develop an idea on the value of something is referred to as valuation. ...
... The further argument claims that to develop an idea on the value of something is referred to as valuation. Evaluative research incorporates a systematic assessment of the value or worth of efforts and resources in achieving intended outcomes (Powell, 2006). ...
... This statement aligned to assess the efforts of environmental impact assessment specialists in using various environmental monitoring and enforcement tools to achieve sustainability in the water infrastructure project development of South Africa. Evaluation is qualitative in nature, being a systematic and empirical process involving a thoughtful analysis (Creswell, 2003 andPowell, 2006). ...
Thesis
Water infrastructural projects are major parts of the development that simplify human life in meeting their needs. They are holistic solutions to the increasing demands of increasing population rates. The world's population is exponentially growing, giving rise to increased demands for environmental resources. Water as the most crucial resource for sustaining daily life forms, is placed under stress through pollution and depletion. Water supply infrastructure projects of Southern Africa are subjected to summons due to failure in terms of wastewater treatment facilities discharging unclear effluent and projects that do not deliver sustainability commitments. For sustainable water use and resource protection efforts are required to regulate the abstraction and pollution mainly through enforcements of regulations, as well the assurance of effective regular follow-up. Environmental Impact Assessment (EIA) is realized as a regulatory requirement that accounts for impacts in all stages of the project implementation. It serves as a guide to facilitate proper project planning through outlining feasible approaches and impact mitigation strategies. Together with the legislative requirements and enforcement licenses they help regulate developmental actions on the environment. This study was intended to critically evaluate the role that EIA specialists play in the water sector of South Africa, with focus on the task of EIA follow-up during the construction, implementation, and operational stages. Secondary data was gathered through literature reviews as a guide to develop a criterion for providing judgments to the primary data collected through self-administered surveys. The evaluation of the outcomes followed a statistical analysis that involved interpretation in the Likert scale evaluation matrix. This study revealed that EIA specialists are successfully engaged in water infrastructure projects, mainly in the planning and execution phases where they add value significantly. In the EIA post decision stage, they add value to an extent as they are not involved in full project implementation nor guaranteeing facility operation and maintenance. The Post EIA follow-up is carried out limitedly, thus the weakness is prone. The weakness in EIA follow-up has been reported as an ongoing challenge both nationally and internationally. Commitment to EIA follow-up should be ensured in all stages of project implementation to achieve sustainability. Keywords: Monitoring; compliance; Enforcement; Role; Environmental Impact Assessment; Specialist; Sustainable development; water infrastructure project, and Challenges.
... An evaluation research was chosen to answer these questions. The purpose of this type of research is to evaluate the impact of an (educational) intervention (Powell, 2006). The results of such research are then used to modify or adapt a programme to enhance its success (Powell, 2006). ...
... The purpose of this type of research is to evaluate the impact of an (educational) intervention (Powell, 2006). The results of such research are then used to modify or adapt a programme to enhance its success (Powell, 2006). ...
... Evaluation is an ongoing process, which starts with the development of an initiative and continues throughout the life of a project and beyond. Utilized as a tool to support and enhance, evaluation should provide a means for continual monitoring, diagnosis, and change (Powell, 2006). A rigorous, systematic evaluation framework should be part of the overall planning process of an organization or program. ...
... A rigorous, systematic evaluation framework should be part of the overall planning process of an organization or program. Powell (2006) further notes that solid evaluation monitors progress towards goals and objectives, measures effectiveness, and is integral to the assessment of services and resources. Evaluation guides strategies for change and is critical for program sustainability and replication. ...
Article
This article discusses and demonstrates the discrepancies between ideal, theoretical program evaluation processes and real world evaluation practice, which is constrained by numerous and varying factors. The article describes the real world experience of Mount Royal University’s Centre for Criminology and Justice Research researchers in conducting an evaluation of the Police and Crisis Team (PACT) in Grande Prairie, Alberta, including a Social Return on Investment (SROI) analysis. PACT, which partners an Royal Canadian Mounted Police (RCMP) officer with a mental health professional, represents a blend of secondary and tertiary crime prevention and attempts to diminish crime in the community by addressing the risk factors of individuals with mental health concerns (creating trust with individuals, increasing awareness of resources, and decreasing stigmatization in the community). PACT also specifically targets those individuals with mental health issues who are in contact with the law to try to decrease recidivism and increase community safety. Challenges were present in the evaluation due to the time frame, staff turnover, program start-up issues, and confidentiality and sensitivity of the program focus. Despite the challenges, the CCJR team completed an evaluation including a forecast SROI, identifying several successes, challenges, and recommendations for change.
... Este es un estudio de caso evaluativo (Stake & Schwandt, 2006;Powell, 2006;Guba, 1987;Stake, 2006) y cualitativo, pues la investigación cualitativa está cargada de valores y busca la profundización en las implicaciones de las acciones (Stake, 2006); a su vez la investigación se ubica en el paradigma interpretativo-constructivista, que no busca la generalización de los resultados, sino la profundización en las creencias, sentimientos y conceptos sobre el mundo, cómo deben entenderse, estudiarse e interpretarse sus significados (Denzin & Lincoln, 2018;Cisneros-Cohernor, 2011;Schwandt, 1998), recordando que el estudio de caso se encuentra dentro de la perspectiva constructivista social (Stark y Torrance, 2005). ...
Article
Full-text available
Esta investigación presenta las experiencias y los retos de jóvenes de comunidades rurales en la zona maya de México que se forman en competencias para la investigación científica y tecnológica. El proyecto siguió un diseño por estudio de caso cualitativo, enmarcándose en el paradigma interpretativo, por lo que no se pretende que los resultados sean generalizables y aplicables a otros contextos o participantes, sino que se buscó profundizar en las vivencias que experimentan los jóvenes rurales de la región maya que se forman en ámbitos como la ciencia y la tecnología. Los resultados indican que los participantes enfrentan cuestiones críticas relacionadas con la calidad de la educación rural, pobreza, colonialidad, discriminación, estereotipos de género en la ciencia y la tecnología, migración académica, así como el acceso a la formación científica y tecnológica.
... Firstly, it is evaluative; since it appraises the implications of sensational reporting in Kwayedza and uMthunywa to the development of indigenous languages. Evaluation research is defined by Powell (2006) as a study that uses standard qualitative research methods for evaluative purposes. Secondly, the study is comparative in nature. ...
Article
Full-text available
Indigenous language media are active spaces for the development, maintenance and renewal of indigenous languages in Africa. Juxtaposing Kwayedza and uMthunywa tabloids, which publish in Shona and Ndebele languages, alongside mainstream English language press, The Herald and The Chronicle, this study interrogates the implications of sensationalism on the structural and functional development of indigenous languages and the indigenous language press in Zimbabwe. This article is framed within the concepts of language development, diglossia and decolonial theory. This is an evaluative and comparative qualitative inquiry of headlines in 30 issues of uMthunywa and Kwayedza. Using interviews, data was gathered from a conveniently and purposively selected sample comprising of journalists, language and media academics, as well as readers and non-readers of Kwayedza and uMthunywa. The study exposes nuances of the marginalisation, folklorisation and corruption of indigenous languages in the press; and the inadvertent validation of English as the language of the media. However, this study ultimately shows that tabloidisation has ensured the survival of indigenous languages and indigenous language press in the context of the hegemony of English and competition from the English language press in Zimbabwe.
... MATERIALS AND METHODS This study utilized evaluation research and a survey questionnaire adopted from NEUST Graduate School as the main instrument of this study. According to [6] as cited in [7] "evaluation research focuses on assessing a particular practice or intervention at a given site." The researchers conducted a series of observation and site visitation as well as focus group discussion to get the necessary information with the used of the questionnaire regarding the concerns and profiles of Tanawan Mushroom Grower's Association. ...
... The need for constantly evaluating library's effectiveness, according to Blagden (1975) is to prove to both the funders and users that the services rendered by the library meets the purpose of creation and that the library is an internal control mechanism. But Powell (2006) itemized ten reasons that subsume all the earlier reasons for evaluating effectiveness. The researcher stated that it is necessary because libraries need to account for the way they made use of their limited resources, explain their actions, enhance their visibility, describe their effects, elude errors, support planning activities, express concern for their public, sustain decision making, and reinforce their political position. ...
Article
The goal of the library is to provide quality service and lifelong learning facilities to its users. However, there are indications of low level of effectiveness in many polytechnic libraries. Studies revealed that this could be attributed to leadership style. The study, therefore, investigated the extent to which leadership style influenced institutional effectiveness of polytechnic libraries in South-West, Nigeria. The research adopted survey research design. The population consisted of 429 library personnel from the 32 polytechnic libraries in South-West, Nigeria. Simple random sampling technique was used to select 24 (75%) polytechnic libraries. Total enumeration was used for the 368 personnel of the selected polytechnic libraries. A validated questionnaire was used as the instrument for data collection. The Cronbach’s alpha reliability coefficients ranged from .81 to .95. The return rate was 86.96%. Data were analyzed using descriptive and inferential (simple and multiple regression) statistics. Findings revealed that leadership style had a positive and significant influence on institutional effectiveness (β = 0.0134, t(317) = 0.1326, R2= 0.13037 p < .05). Furthermore, the level of institutional effectiveness was low (M =2.20 on a scale of five); leadership style predominately practiced was transactional (M =2.87 on a scale of five) closely followed by transformational leadership style (M =2.66 on a scale of five). In conclusion, leadership style is a major factor contributing to low level of institutional effectiveness. Therefore, the study recommended that heads of libraries should practise more of transformational leadership style.
... The qualitative research design of Evaluation Research was employed which reviewed the literature and conducted a survey of selected simulation centres around the world. According to Powell (2006) evaluation research can be defined as 'a type of study that uses standard social research methods for evaluative purposes, as a specific research methodology, and as an assessment process that employs special techniques unique to the evaluation of social programs'(p.102). Silver and Pratt (2006) indicate that evaluation is a systematic approach to acquiring and assessing information so as to generate appropriate feedback about an object, which is the focus and intent of this research. ...
Article
This study used a social constructivist approach to explore the literature in relation to the acknowledgment and application of models and frameworks in simulation based learning and teaching activities in health care education. This paper addresses the data collection phases in the research design in developing a Conceptual Framework for Simulation in Healthcare Education. Background: Two literature review activities occurred in the study. The first literature review explored the challenges in the literature for further research in relation to the educational factors that have an impact upon simulation as a learning and teaching modality. This part of the research process was to identify the publications that address simulation as a learning and teaching modality and uncover evidence to justify the study. The literature revealed that there were limited publications in relation to simulation as a learning and teaching modality that was underpinned by a conceptual framework. Design: The study research design employed Evaluation Research to conduct a further intensive review of the literature as part of phase one of data collection. The second part of phase one of data collection was the surveying of randomly selected simulation education centres around the world to ascertain their employment of a conceptual framework in the application of simulation as a learning and teaching modality. Phase two was the genesis of the conceptual framework and Phase three was the evaluation of the draft framework using a modified Delphi Technique by randomly selected simulation experts. Results: The intensive literature review highlighted that authors acknowledged education-based theories but this did not extend to the application of a conceptual framework to simulation as a learning and teaching modality. The survey provided the evidence to support the development of a Conceptual Framework for Simulation in Healthcare Education for the conceptualisation, design, development and delivery of simulation teaching and learning activities in health care education. The simulation experts provided valuable data and commentary in the final formation of the conceptual framework. Conclusion: Based on the outcomes of the literature review, questionnaire and modified Delphi Technique the need for the development, and subsequent structure and utility of the Conceptual Framework for Simulation in Healthcare Education was verified.
... Evaluative research involving systematic collection of data about the organisation (Powell, 2006) of an in-service training session is the research methodology of this study. This section describes the methodology for data collection and analysis of this study. ...
Article
Full-text available
Problem-based Learning (PBL) is practised in Malaysian universities, but it is rarely carried out in Malaysian secondary schools. Hence SEAMEO RECSAM conducts PBL courses on a regular basis, to introduce teachers to PBL. The 32 respondents of this study were selected from a large group of Malaysian teachers who had attended the 2017 STEM Workshops at SEAMEO RECSAM. The literature showed that the benefits of PBL come at a price, in that PBL requires more ICT resources and facilitation by educators who have reported the task as difficult and frustrating. The author facilitated this class of PBL, and can thus share the first-hand experiences. Six research questions were answered in this study. The author found that although it was challenging to see to the needs of 32 individuals, it was not frustrating as there were positive findings. The respondents agreed that the PBL lesson was appropriately conducted, and was aware that the roles of the teacher and students have shifted in PBL. All the respondents (except for one) were willing and confident to practise PBL in some of their lessons with their students. This implied that certain educators adapted more easily to be a facilitator of PBL than other who was more difficult to be convinced. It is hoped that more teachers will be empowered through training of PBL facilitation skills in the subsequent year with the knowledge and experience gained through this study.
... According to [4] as cited in [5], "descriptive survey can systematically describe a situation, problem, phenomenon, service or programs, or provide information or describe the attitude towards an issue". On the other hand, "evaluation research aimed to enhance knowledge and decision making and lead to practical applications" [6] as mentioned by [7]. The respondents of the study who were chosen purposively were 10 barangay officials, 3 local government unit (LGU) representatives and 30 household representatives. ...
... The system supposed to capture the statistics about the developer in the predefined categories. For this proof of concept purpose, the system used a rule-based approach to derive those statistics [21]. Then those statistics were compared against the information collected by an interview process and a code review process. ...
Conference Paper
Full-text available
Sharing and preserving coding best practices among the developers are becoming an important objective of the software development lifecycle. There have been numerous researches done in order to mitigate the issues related to bad coding practices. One of the most challenging tasks towards mitigating this is to identify the skill level of the developers, coding patterns and likelihood for bad coding practices. The widely used methods for this is conducting one on one interviews with the developers and review developers work, which does not cost and time efficient and depends on various variable factors. This particular research tried to contribute to the field of software architecture by analyzing the feasibility of using machine data to identify the developer coding patterns and related data and provide a mechanism to enhance the skills of a developer. Empirical and theoretical evaluations have been carried out to prove that the analysed machine log data results can match the interview results in an 80% success rate.
... Lack of approved courses for research process before submission of synopsis, places the newly admitted post graduate students in great confusion. Insufficient time is also another contributing factor 3 .In many cases ultimate research work is mere repetition of previous works 4 . This fact leads to few noteworthy scientific contributions 5 . ...
Article
Full-text available
Back ground: Thesis is an obligatory part of Medical post graduate or post doctoral courses under West Bengal University of Health Sciences. The purpose of this thesis work is to make the doctors familiar with research methodology. There is no procedure or scale to evaluate the quality of thesis works in our state. Aim: Formation of a scale for evaluation of thesis work & using this scale evaluation of ongoing thesis works of this medical university. Procedure: At first we develop a scale containing five factors Format, Study design; Knowledge of the post graduate trainee, new ideas, Literature reviewed and validation of that scale was done. With the help of this validated scale we have evaluated 135 thesis works of outgoing post graduate medical students. Result: After evaluation it was found that most of the thesis work falls in the excellent category. Very few are exceptional as well as good or acceptable. The scale also indicated that the weak areas of the thesis work. Conclusion: Our scale will indicate the clear picture of quality of a research work including strong and weak areas. This fact will help us in future to plan a better research work. Key Words: Thesis evaluation, Validated Scale, research methodology
... Evaluation is described as the act of measuring or exploring properties of a system (in planning, development, implementation, or operation), the result of which informs a decision to be made concerning that system in a specific context (Brender (Brender, Talmon, de Keizer et al., 2013;McNair, 2006). Evaluation research can be viewed as a type of study that applies various assessment processes (Powell, 2006). ...
Conference Paper
Full-text available
Sarietjie Musgrave UFS Ideas Lab MusgraveS@ufs.ac.za Glynis Jaeschke UFS Ideas Lab JaeschkeGJ@ufs.ac.za ABSTRACT-The aim of this paper is to describe the adaption and scaled application of a Teacher Professional Development (TPD) course, towards, not only changing classroom practice, but empowering teachers to become lifelong learners. The ICT for Rural Education Development (ICT4RED) TPD Curriculum was designed and validated as a standalone intervention in a single district, for rural teachers, but did it carry Higher Education Institution accreditation. This implementation gap is addressed in its adaption to a Short Learning Programme which will be part of a module towards the Advanced Certificate in Teaching at the University of the Free State (UFS) as part of the DRDLR ICT4E project. This adaption necessitated some reflection on methodology and methods implemented and presented opportunities to incorporate experiences gained at the IDEA Lab at the UFS. The research initiative applied Evaluation research as a research methodology, through considering the Technological Pedagogical Content Knowledge (TPACK) model, working hypotheses were developed to capture the various dimensions of the rural mobile learning ICT4RED TPD Curriculum. These dimensions were evaluated and prioritised by reflecting on the requirements that the accreditation and scaling would impose. The results of the evaluation are presented and the outcomes and the subsequent design decisions are outlined and discussed.
... It is concluded that evaluation research is a systematic process that involves collecting data about organizations, processes, programs, services and/or resources. Evaluation research should enhance knowledge and Journal of Applied Mathematics and Physics decision making and lead to practical applications [7]. ...
Article
Full-text available
This research paper evaluated and investigated the effectiveness of the comprehensible technique (CT) introduced by the researcher as an alternative solution to Algebra’s linear equations in solving consecutive number problems. The CT is a proposed easy and efficient way of solving number problems which aimed to help the learners reduce their difficulties in using variables since the technique only uses basic arithmetic and mental Mathematics. Evaluation research was utilized in this study where 20 people, from various ages and professions, were chosen purposively as respondents of the study. The study found out that CT was very effective for the respondents especially in unlocking difficulties in solving consecutive number problems. Moreover, according to them, the CT is easy to use, it is engaging and enjoyable, direct and simple and can be utilized as shortcut way of solving consecutive number problems when competing in Math quiz bees or taking board examinations where time is of the essence. KEYWORDS Algebra, Comprehensible Technique, Consecutive Number Problems, Linear Equation
... Het onderzoek is vormgegeven als een kwalitatief evaluatieonderzoek (Powell, 2006). Het doel hiervan is om de impact van een interventie te evalueren, zoals een onderwijsproject of nieuw ontwikkeld curriculum. ...
Article
Full-text available
Voor aankomende leraren is het belangrijk om vakoverstijgend onderwijs te kunnen verzorgen waarin leerlingen, naast vakinhouden, ook vakoverstijgende vaardigheden opdoen zoals creatief-, kritisch-en probleemoplossend vermogen. Daartoe dienen student-leraren ook zelf ervaring op te doen met vakkenintegratie. In deze bijdrage worden het proces en de leeropbrengsten van een zogenaamde designathon beschreven. Hierin worden aspecten van kunst en bètavakken geïntegreerd rond een centrale ontwerpwerpvraag, waarmee gemengde teams van student-leraren met verschillende achtergronden (kunst, pabo en bèta) aan de slag zijn gegaan. In een kleinschalig onder-zoek is door middel van observaties en het inzetten van learner reports gekeken naar het proces van de designathon, de rol van de begeleiders en de leeropbrengsten. Het blijkt dat vooral het aspect van de 'pressurecooker' (in korte tijd een volledig ontwerpproces doorlopen) goed werkte: het daagde uit tot uitwisseling van ideeën en geëngageerde discussies. Uit observaties blijkt dat een combinatie van procesmatige en inhoudelijke begeleiding het meest effectief was. De meeste gerapporteerde leerervaringen waren gerelateerd aan het opdoen van ervaring met het samenwerken met studenten uit andere disciplines en met de inhoud en didactiek van de designathon.
... Evaluation is described as the act of measuring or exploring properties of a system (in planning, development, implementation, or operation), the result of which informs a decision to be made concerning that system in a specific context (Brender (Brender, Talmon, de Keizer et al., 2013;McNair, 2006). Evaluation research can be viewed as a type of study that applies various assessment processes (Powell, 2006). ...
Conference Paper
Full-text available
The aim of this paper is to describe the adaption and scaled application of a Teacher Professional Development (TPD) course, towards, not only changing classroom practice, but empowering teachers to become lifelong learners. The ICT for Rural Education Development (ICT4RED) TPD Curriculum was designed and validated as a standalone intervention in a single district, for rural teachers, but did it carry Higher Education Institution accreditation. This implementation gap is addressed in its adaption to a Short Learning Programme which will be part of a module towards the Advanced Certificate in Teaching at the University of the Free State (UFS) as part of the DRDLR ICT4E project. This adaption necessitated some reflection on methodology and methods implemented and presented opportunities to incorporate experiences gained at the IDEA Lab at the UFS. The research initiative applied Evaluation research as a research methodology, through considering the Technological Pedagogical Content Knowledge (TPACK) model, working hypotheses were developed to capture the various dimensions of the rural mobile learning ICT4RED TPD Curriculum. These dimensions were evaluated and prioritised by reflecting on the requirements that the accreditation and scaling would impose. The results of the evaluation are presented and the outcomes and the subsequent design decisions are outlined and discussed.
... The research methodology followed is that of evaluation research [31,32] with elements of action research [33] embedded within. The student enrichment programme (see Figure 1) went through a planning and implementation phase and was followed by an evaluation phase, the two phases typical of evaluation research. ...
Article
Full-text available
This paper presents an enrichment case study to showcase a possible avenue for attending to the needs of academically strong mathematics students. We report on a group of university students who were presented with the opportunity of exploring a specific first year mathematics topic deeper, using an inquiry-based learning approach as part of an enrichment programme. Following the intervention, students completed a questionnaire and a few were interviewed to establish their experiences of the enrichment programme. We discuss the successes and pitfalls of the intervention and report on the impact it had on the participants.
... 246-247). In an overview of evaluation research, Powell (2006) adds service quality assessment, process evaluation, benchmarking, standards, quantitative methods, qualitative methods, cost analysis, organizational effectiveness, and program evaluation methods to input measures, performance measures (or outputs), and impact assessment (or outcomes). ...
Article
Library data in the form of holdings, expenditures, and transactions continues to be collected. Recently emphasis is being placed on connecting library usage data to student success indicators for purposes of indicating impact. This article describes how one library involved in a large-scale, student success assessment project analyzed usage data linked to unique student IDs prior to connecting data to student academic information. Results revealed a trove of useful information that led to a more robust understanding of how students interacted with library services over the course of a year and are discussed in relation to process improvement.
... The categorization provided a structured overview. Most papers fell into the category of 'evaluation research' [22] and used standard social science data collection methods including self-evaluation surveys [23][24][25][26][27][28][29][30][31], questionnaires [24,[32][33][34][35], interviews [36][37][38] or literature reviews [39][40][41][42]. In a few cases, Delphi techniques were used to determine the importance of evaluation criteria [43], or the quality of a tool [44]. ...
Article
The evaluation of emergency, disaster and crisis management exercises supports both individual and organizational learning, facilitates the development of response capabilities, and helps to determine whether the current level of preparedness is good enough. Nevertheless, despite its importance in the field of disaster risk management, there is a lack of a comprehensive overview of research in the area. The aim of the paper is to provide such an overview.
... According to Chelimsky (1997) and Patton (2008), the purpose of an evaluation determines the use that it will be put into. Evaluation is not conducted for any one purpose (Patton, 2008;Powell, 2006;Chelimsky, 1997). According to Chelimsky (1997), evaluation purposes are classified into three, namely: evaluation for accountability, evaluation for programme improvement and evaluation for knowledge generation. ...
... According to Chelimsky (1997) and Patton (2008), the purpose of an evaluation determines the use that it will be put into. Evaluation is not conducted for any one purpose (Patton, 2008;Powell, 2006;Chelimsky, 1997). According to Chelimsky (1997), evaluation purposes are classified into three, namely: evaluation for accountability, evaluation for programme improvement and evaluation for knowledge generation. ...
Article
Full-text available
An evaluation inquiry has been noted to have a significant influence on whether the evaluation outcomes are used or not. However, only limited studies have examined evaluation inquiry and its implication on the use of evaluation in Ghana. This study therefore seeks to contribute to knowledge on evaluation inquiry and the implications it has on evaluation use in Northern Ghana. Using a multistage sampling process as part of a broader study on evaluation inquiry and use in Northern Ghana, the study examined key elements of evaluation inquiry in 27 programmes implemented between the period 2000 and 2010 in Northern Ghana. Two main levels of data collection were carried out. The first level collected data on the programmes while the second level collected data from programme staff of three programmes as the units of enquiry. It emerged that evaluation was mostly externally funded, premium was placed on project completion evaluation; and evaluation was more an external accountability driven activity. The potential of mid-term evaluation to contribute to evaluation use was undermined by the tag associated with it as basis for the termination of programmes. The study therefore recommends building the evaluation capacity of staff and professionalization of evaluation practice in the country as a way of enhancing evaluation inquiry.
... Assessing the research methodology is essential in social science studies (Powell 2006) to create and maintain their trustworthiness (Eriksson & Kovalainen 2008). The assessment also determines the robustness of studies that use qualitative methods (Horsburgh 2003;Long & Geoffrey 2004). ...
Book
Full-text available
Commercialization of high technologies refers to a process that transforms radical and disruptive innovations into consumable products, services and solution packages. This process generates returns on research and development investments, employs highly educated people, empowers semi-skilled persons and makes business enterprises prosperous and sustainable. Hence, this study investigated the commercialization process purposely to provide in-depth knowledge of it. The study aimed to investigate how high technologies are commercialized, especially in ICT, Cleantech and the Life Sciences industries to provide a better understanding of the commercialization process. This understanding is essential due to the increase in new high technologies and rapid changes in these industries. The study employed a case study research method to achieve its goal. It used interviews, an online survey, observations and documentary for data collection, and content and thematic methods for data analysis. It used primary and secondary data that were collected between 2012 and 2017. Its participants represented stakeholders of the above-mentioned industries. The participants mostly came from Finland, but some came from other European countries. The study found that the commercialization process models are rooted in the Technology Acceptance Model (TAM), Unified Theory of Acceptance and Use of Technology (UTAUT), Stage-gate, Effectuation, Bricolage and Lean Start-up theoretical frameworks. Similarly, it found that the process makes use of a mixture of linear and non-linear logic. Additionally, it found that the commercialization process starts during the pre-innovation phase and continues after the innovation’s marketing phase, and that commercialization activities must be performed in parallel (simultaneously). Most interestingly, the study revealed that digitalization has changes on the commercialization process. Based on the findings, the study proposed a new model for commercializing high technologies in the above-mentioned industries; a framework termed the “Integrative Commercialization Process” (ICP). The model was built on theories, theoretical frameworks and models of the innovation process, New Product Development (NPD), Entrepreneurship, Spinoff, Transfer of Technology (TOT) and Marketing. Similarly, the model was built on the context of ICT, Cleantech and the Life Sciences. Hence, the study offered insights into the theoretical frameworks of the innovation, the NPD, Spinoff and marketing fields. Furthermore, the study provided practical insight for the stakeholders of the aforementioned industries, especially potential entrepreneurs (e.g., scientists and engineers), technology entrepreneurs, entrepreneurship educators, commercialization practitioners, universities and research institutes and companies. Therefore, this study makes contributions to the theory and practice of commercialization.
... Evaluation is described as the act of measuring or exploring properties of a system (in planning, development, implementation, or operation), the result of which informs a decision to be made concerning that system in a specific context [33,34]. Evaluation research can be viewed as a type of study that applies various assessment processes [35]. For the purpose of this paper the criteria that was found from literature reviews (on what criteria should be applied to evaluate an Innovative educational ecosystem), will be used to evaluate the results found from the ICT4E project to determine if this project can be regarded as an instantiation of an innovative educational ecosystem in South Africa. ...
Conference Paper
Full-text available
The purpose of this paper is to evaluate a specific mobile technology rural schools project known as ICT4E in South Africa to determine if it complies to be regarded as an innovative educational ecosystem. The criteria that were used were sourced from the literature. The project was explained and why it is regarded as a rural school mobile tablet project in education. Some high-level results and lessons learnt from this project are also provided that was used to evaluate whether it can be regarded as an instantiation of an innovative educational ecosystem. The evaluation methodology was applied to do the evaluation. The main results were that based on the criteria and the findings from the ICT4E project it could be regarded as an instantiation of an innovative educational ecosystem as it complied to most of the criteria except for two where it only partially complied. The project was also seen as a good example of how a country's system of innovation can be supported from a socio-cultural context-specific perspective.
... Through qualitative methods e.g. interviews and observations, researchers explore how the intervention was perceived by different groups and individuals (Calidoni-Lundberg, 2006;Powell, 2006). The results of the research are used to modify or adapt a programme to enhance the success of its design (Calidoni-Lundberg, 2006). ...
Article
Full-text available
This article describes the qualitative study of the redesigning of a course based on the Canadian "4CO-teaching method" for student teachers. This method consists of four different phases of co-teaching: co-design, co-execution, co-debriefing, and co-reflection. Through this way of co-teaching, student teachers in both primary education and in arts education (theatre, dance, and visual arts) were taught how to design arts lessons for primary education together, how to carry them out as a couple, and how to jointly reflect on their lessons. The course, called "Teamplayers", aimed to teach these student teachers how to complement their knowledge and skills during the designing and teaching of arts lessons and, thus, enhance the quality of arts education. This research study evaluated the design of Teamplayers and the students' experience with the method of 4CO-teaching with the aim of improving the course.
... At a more practical level, the purpose of evaluation is to establish the effectiveness of a service or programme and take action based on that information. As Powell (2006) notes, evaluation must have a purpose; without the potential for some action there is no need to evaluate. ...
Chapter
This chapter explores the role that the evaluation of public library services plays in governance. The link between good governance and evaluation lies in the contribution that the latter can make to improved organisational performance. Evaluation supports the achievement of the goals of the organisation by focusing on accountability, transparency, effectiveness and efficiency. Organisations must have in place sound processes and systems to support and enhance performance, accountability and service development. The results of a study of evaluation activities undertaken by public libraries in New Zealand are discussed to demonstrate the use and value of evaluation in the public library service context.
... Arvioinnilla tarkoitetaan jonkin kohteen tai asian arvon, merkityksen tai erityisyyslaadun määrittämistä suhteessa valittuihin arviointikri teereihin (Macnamara 2014). Arviointi on syste maattinen prosessi, joka sisältää datan keräämi sen, analysoinnin, raportoinnin ja johtopäätös ten tekemisen (Askim 2007;Powell 2006). Vies tin nän tutkimuksessa arviointi nähdään vaiheena, jota edeltää mittaaminen ja seuranta ja jota seu raavat suunnittelu ja toiminta. ...
Article
Full-text available
Communication measurement and evaluation – Goal-orientation, agility or fumbling in the dark? The interest of organizations in assessing stakeholder relationships and expectations, intangible assets, and social impact has grown strongly in the 21st century. This change is linked to new views on the strategic role of communication and strategic management. We examine the development and current state of the measurement and evaluation (M&E) of communication through research literature and empirical material. We suggest that the models of M&E used in organizations can also tell us about their views of strategic communication and the connection to organizational strategy. Present practices of M&E are dominated by logical models as well as summative evaluation, which are connected with a functionalist view of strategic communication. The data also provide indications of agile evaluation. Many organizations also engage in practices that we call blind measurement: metrics are not directly linked to the goals of communication.
... Check and Schutt (2021) state that any educational programme is likely to have multiple outcomes, some intended and some unintended, some positive and others that are viewed as negative. Powell (2006) states that evaluation research is usually employed for decision making, dealing with specific questions about a programme. In this regard, an evaluation research design offers a systematic approach to feedback, providing researchers with answers to their questions informing subsequent decisions and actions. ...
Thesis
Taking the form of a portfolio of papers written across my six-year research journey, this professional doctoral thesis presents the findings of a discipline-independent curriculum intervention designed to improve students’ self-regulated learning skills. The study also examines the relationship between students’ self-regulated learning skills and their academic achievement. Research suggests that discipline-independent training interventions improve students’ self-regulated learning skills, also having a major impact on students’ academic achievement across childhood and adolescence. Founded on Zimmerman's (2000) cyclic model of self-regulated learning, the 10-week discipline-independent intervention underpinning this research was designed and implemented to support the development of Year 9 (13-14 year old) students’ self-regulated learning skills. Following a pre-test post-test non-equivalent group design, participants were divided into two groups by way of purposeful sampling, with each group receiving the training intervention consecutively during the academic year 2017/18. Data was collected at three timepoints (before the start of Phase 1, at the end of Phase 1, and at the end of Phase 2) using the Motivated Strategies for Learning Questionnaire (MSLQ) and an original and tailored instrument created specifically for this research; the Self-Regulated Learning Experimental Design Survey (SRLEDS). Forming a significant contribution to the field, this instrument was validated using the MSLQ. Results show that although students’ self-regulated learning skills and academic achievement improved across timepoint, there were not any significant differences between group and nor did students’ self-regulated learning skills predict their level of academic achievement. In light of this, the discussion focuses on providing context to these results, exploring local changes within the research setting that account for the findings before outlining the implications of this study for both research and practice. In addition to the development and validation of the SRLEDS, this study also contributes to the fields of research and practice by offering a critical reflection on the challenges of using control groups within a live school research setting. In response to this, an alternative research design is proposed as well as a range of future research directions, with implications for practice highlighted and discussed.
... This is an evaluative study with research questions derived from the project education materials, 14 with a rigorous approach to analysis undertaken to allow generation of knowledge that can lead to practical applications. 15 Quantitative data Data were analysed using IBM SPSS Statistics version 25. Categorical data were summarised using frequency and percentages. ...
Article
Objective: To evaluate the End-of-Life Essentials education module 'Emergency Department End-of-Life Care' and explore learners' views on what constitutes a compassionate response in the ED. Methods: The present study used a multi-methods approach. Learners comprised a mix of nurses, doctors and allied health professionals. A quantitative pre-post evaluation analysis of learners' (n = 959) knowledge, skills, attitude and confidence was conducted, along with a qualitative thematic content analysis on learner responses (n = 538) to the post-evaluation question, 'What is a compassionate response for you in the emergency department?' Data were extracted for a 12-month period, 6 May 2019 to 6 May 2020. Results: Learners' post-evaluation ranks of knowledge, skill, attitude and confidence were significantly higher than the pre-evaluation ranks (P < 0.001). Emerging themes from the qualitative data were organised into three overarching categories: communication skills (e.g. listening and use of names), care discussion and provision (e.g. provide information and discuss care plans) and humanising healthcare (e.g. emotional support and empathy, taking the time, and offering kindness and comfort). Conclusion: The 'Emergency Department End-of-Life Care' module had a significant positive impact on learners in relation to perceived knowledge, skill, attitude and confidence. This evaluation suggests that the End-of-Life Essentials ED module could be a useful online learning resource for health professionals.
... Therefore, it helps determine what works well and what a programme or initiative could improve. Evaluation research also helps to determine if an approach would be appropriate to replicate in other locations with similar needs (Patton, 1990;Powell, 2006;RHIhub, 2020). ...
... The first phase of the evaluation involved an analysis of the NYR activities in the two case study authorities of Calderdale and North Lincolnshire. Powell (2006), in providing an overview of evaluation research says that it should enhance knowledge and decision making and lead to practical applications. One way of exploring the challenging question of how libraries contribute to the cohesion and development of their communities is to use the concept of social capital. ...
... These can be compared with our value proposition's focus on people, space, and resources. • Outcome-based planning and evaluation: We must be clear about the outcomes we want to achieve, design programs and services to achieve those outcomes, evaluate whether these outcomes (as well as outputs) are met, and use our results to demonstrate our impact (Gross, Mediavilla, and Walter 2016;Powell 2006). • Welcoming spaces: We must provide library spaces that are accessible and welcoming and inviting and meet the diverse needs of the community (Brewster 2014). ...
Article
The purpose of this paper is to show how Californian libraries deliver value, by taking a preliminary value framework previously developed from a study of academic literature, and applying it to the public library landscape in California. A “bricolage” approach was used to develop an evidence-based framework of value and subsequently to gather data about Californians’ use of and attitudes about public libraries. The framework was developed through an extended literature review and applied across data gathered through a survey of residents, interviews with library stakeholders, a review of statistics on library usage, and a review of the outcomes of grant-funded programs in California public libraries. The most significant outcome of our project is a unique value proposition for California’s public libraries. These libraries deliver value for individuals, families, groups, communities, and society as a whole. They provide opportunities and support for learning and knowledge development, health and wellbeing, community development, and economic development. Value is delivered through a unique combination of resources, people, and space. California’s public libraries are largely offering services that are valued by the public; however, public awareness of the range and nature of services offered remains lower than desired. Practitioners can use the evidence-based framework and the unique value proposition to better understand the aspects of their library services that offer most value to their communities, and to make more persuasive cases for library support. This paper is the first to use an evidence-based framework of value in assessing the impact of California’s public libraries.
... Anthony D. Harris and his coauthors have performed quasi-experimental research in infectious diseases in the area of interventions aimed at decreasing the spread of antibiotic-resistant bacteria [Harris et al., 2004]. Ronald R. Powell discusses several evaluation methods, such as input measurement, output/performance measurement, impact/outcomes assessment, service quality assessment, process evaluation, benchmarking, standards, quantitative methods, qualitative methods, cost analysis, organizational effectiveness, program evaluation methods, and library and information (LIS) centered methods [Powell, 2006]. Monika Mueller and her coauthors study a systematic scoping review of published methodological recommendations on how to systematically review and meta-analyze observational studies. ...
Article
Full-text available
Research is the framework used for the planning, implementation, and analysis of a study. The proper choice of a suitable research methodology can provide an effective and successful original research. A researcher can reach his/her expected goal by following any kind of research methodology. Quantitative research methodology is preferred by many researchers. This article presents and analyzes the design of quantitative research. It also discusses the proper use and the components of quantitative research methodology. It is used to quantify attitudes, opinions, behaviors, and other defined variables and generalize results from a larger sample population by the way of generating numerical data. The purpose of this study is to provide some important fundamental concepts of quantitative research to the common readers for the development of their future projects, articles and/or theses. An attempt has been taken here to study the aspects of the quantitative research methodology in some detail.
... It is in this explanatory mode that evaluation overlaps most directly with mainstream social science. Elliot Stern [9], explained, evaluation research is a type of study that uses standard social research methods for evaluative purposes, as a specific research methodology, and as an assessment process that employs special techniques unique to the evaluation of social programs [10]. ...
Article
Full-text available
This study aims to evaluate the implementation of the entrepreneurship learning process from the aspects of planning, implementation and evaluation in five study programs in four higher education institutions. It employs the evaluation method by using input, output/performance and impact/outcome assessment. Data is collected through assessment by experts to assess the quality of the syllabus content, assessment by participants-students and lecturers-by using online questionnaires (Google form), and interviews with lecturers. Data is analysed by comparing the existing syllabus components with the syllabus preparation criteria of the Directorate of Islamic Higher Education, calculating the average student response to the learning performance, and interpreting the open questionnaire of lecturers, as well as examining the patterns of the interview results. The results suggest that the syllabus is not in line with the learning achievement criteria. The implementation of all components in entrepreneurship learning is perceived positively by students. The evaluation result reveals that the cognitive domain assessment instrument is dominantly used, while the attitude and skill domain is not maximally implemented. Entrepreneurship education curriculum design needs to be standardized, so that the quality of the entrepreneurship learning process can be well-measured.
Article
Full-text available
The Kapatid Mentor ME (KMME) Program of the Philippine Government was designed with the goal to help Micro, Small and Medium Enterprises (MSMEs) scale up their business by weekly coaching and mentoring. It is claimed that it is a successful program because of the success stories of the graduates. However, there is still no published articles or researches that show the impact of the program to the graduates in Digos City. Thus, this study was conducted. The main objective of the study was to assess the impact of the KMME in the overall performance on the registered business(s) of the 2017 graduates in Digos City. A qualitative evaluation research design was employed and a series of key informant interviews were conducted with the five participants chosen for the study. Data were analyzed using thematic analysis and revealed that the program has a positive impact on both the personal improvement of the participants, and business performance has scaled up.
Article
Full-text available
Background Cardiovascular disease (CVD) is the first cause of death globally, with huge costs worldwide. Most cases of CVD could be prevented by addressing behavioural risk factors. Among these factors, there is physical and amateur sports activity (PASA), which has a linear negative correlation with the risk of CVD. Nevertheless, attempts to encourage PASA, as exercise prescription programmes, achieved little impact at the community-wide level. A new frontier to promote PASA is represented by mobile health tools, such as exergaming, mobile device apps, health wearables, GPS/GIS and virtual reality. Nevertheless, there has not yet been any evident turnabout in patient active involvement towards CVD prevention, and inactivity rates are even increasing. This study aims at framing the state of the art of the literature about the use of m-health in supporting PASA, as a user-centric innovation strategy, to promote co-production health policies aiming at CVD prevention. Methods A mixed-method systematic literature review was conducted in the fields of health and healthcare management to highlight the intersections between PASA promotion and m-health tools in fostering co-produced services focused on CVD prevention. The literature has been extracted by the PRISMA logic application. The resulting sample has been first statistically described by a bibliometric approach and then further investigated with a conceptual analysis of the most relevant contributions, which have been qualitatively analysed. Results We identified 2,295 studies, on which we ran the bibliometric analysis. After narrowing the research around the co-production field, we found 10 papers relevant for the concept analysis of contents. The interest about the theme has increased in the last two decades, with a high prevalence of contributions from higher income countries and those with higher CVD incidence. The field of research is highly multi-disciplinary; most of documents belong to the medical field, with only a few interconnections with the technology and health policy spheres. Although the involvement of patients is recognized as fundamental for CVD prevention through PASA, co-design schemes are still lacking at the public management level. Conclusions While the link between the subjects of motor activity, medicine and technology is clear, the involvement of citizens in the service delivery process is still underinvestigated, especially the issue concerning how “value co-creation” could effectively be applied by public agencies. In synthesis, the analysis of the role of co-production as a system coordination method, which is so important in designing and implementing preventive care, is still lacking.
Article
This paper examines the intellectual structure of evaluation by means of citation analysis. By using various article attributes and citation counts in Google Scholar and (Social) Science Citation Index Web of Science, we analyze all articles published in Evaluation and Program Planning from 2000 until 2012. We identify and discuss the characteristics and development of the field as reflected in the history of those citations.
Article
Non-governmental organisations (NGOs) have the potential to combine social objectives with trading activities, while social reporting represents a tool for disclosing NGOs' peculiarities. Although the number of NGOs adopting social reporting is increasing, research is still lacking on the relationships between firms and NGOs and their impacts at the government level, at which policy makers must both favour social/environmental commitments and ensure that NGOs pursue their core objectives, also when carrying out trading activities. This paper investigates the challenges of policy making in this field by analysing NGOs in Italy, the first country to make social reporting compulsory. We randomly selected 20 social reports from the Italian NGO register to analyse eight components regarding the disclosure of their impact. The results show that a compulsory requirement leads to a paradox, given the trade-off between stakeholder engagement principles on the one hand and disciplinary power and accounting as action at a distance principles on the other, leading NGOs to account for their output rather than for their impact. We thus contribute to the literature about NGOs and corporate social responsibility (CSR), delineating the relevance of a paradox perspective. To the best of our knowledge, this is the only study to apply this perspective to the analysis of CSR and sustainable development trajectories in NGOs.
Article
Full-text available
In India, the service sector has assumed greater economic importance over the past decade and enjoys the largest share in GDP. Banking and financial services, being an important part of service sector, are facing critical challenges to compete with the international players while satisfying customers by offering quality services. As delivering quality service to customers is a must for success, it needs to be continuously evaluated. Extant literature on the subject supports the contention that SERVQUAL and SERVPERF are the two most prominent scales forming the genesis for service quality assessment in different service sectors. Present paper attempts to present a review of the above mentioned two scales in an elucidative, concise and thoroughly documented manner. It also tries to posit the best approach of evaluating service quality in a more efficient and valid method for marketing managers/researchers in Indian context.
Article
The general practice of civil servant training providers in Hungary is to evaluate their products only through surveying the reaction of participants. The obvious weakness of this practice is that the variance in the level of satisfaction does not necessarily coincide with the positive professional impact that the trainings are aimed to produce. This paper presents the results of an effectiveness assessment survey of a large public ethics and integrity training programme that was delivered to civil servants in Hungary. The trainings examined were delivered by the same methodology but conducted by 26 different trainers for 7362 participants. The assessment was not part either of the original project, or of the training design; it was run as an additional activity. The aim of the survey was to show that even with a simple method, applicable even with weak organizational capacities, information can be produced both for validation and curriculum development. The results prove that the trainings made a moderate but statistically significant impact on participants’ knowledge and attitudes, and most changes happened in the targeted direction. Beyond the validation of the training methodology, the survey also produced information on trainers’ performance and a relatively differentiated picture on participants’ learning that can contribute to the further development of the methodology. Thus, the results prove that even with simple quantitative survey methods, evidence for both validation and learning can be produced. The results also raised some questions for further research.
Article
The new Design Science department at the Journal of Operations Management invites submissions using a design science research strategy for operations management (OM) issues. The objective of this strategy is to develop knowledge that can be used in a direct and specific way to design and implement actions, processes or systems aimed at achieving desired outcomes. This knowledge is developed by engaging with real-life OM problems or opportunities. Manuscripts submitted to this department will be evaluated on pragmatic validity and practical relevance. Because design science research (DSR) differs in some important aspects from other OM research strategies, this essay examines in some depth its challenges and possible solutions.
Article
This study aimed to adapt the Get Set2 quiz and design a diagnostic quiz for implementation under assumed knowledge admissions into university science degrees. The GetReady Quiz that was designed, comprised five modules benchmarked against the high school curriculum in Mathematics, Chemistry and Biology. The significance of this study stems from the development of an alternate approach to support students, with deficiencies in mathematical skills, who enter university STEM programmes with recommended foundational knowledge in mathematics and science. A mixed-mode research methodology was employed in which quiz design was coupled with implementation and evaluation with a pilot study. The GetReady Quiz design featured five discipline modules in Mathematics, Chemistry and Biology, with progression of question difficulty. Knowledge gaps and apparent difficulties for some students with conceptual understanding were evident. Students displayed a realization of their lack of knowledge by reporting that completion of the GetReady Quiz helped them identify areas for revision. The findings indicated that the GetReady quiz was appropriate for assumed knowledge self-diagnosis for commencing students. Approaches to diagnostic testing, such as the GetReady Quiz, should be recommended, if not compulsory, for university students who commence study with STEM assumed knowledge admission.
Book
Full-text available
In our third volume of the Education Theory Made Practical series, we continue to try to close the gap betwen the theoretical world of health professions education and the real world of clinical and classroom teaching. Originally written as a joint venture between the International Clinician Educator blog (https://icenetblog.royalcollege.ca/) and the Academic Life in Emergency Medicine Faculty Incubator program (https://www.aliem.com/faculty-incubator/), this book is a great primer for those seeking to apply theory to their day-to-day teaching. The series was originally posted as a blog series and was subject to open peer review. This book is an edited compendium of these peer reviewed works, which have been reformatted and laid out in a textbook format.
Thesis
Full-text available
Future disasters or crises are difficult to predict and therefore hard to prepare for. However, while a specific event might not have happened, it can be simulated in an exercise. The evaluation of performance during such an exercise can provide important information regarding the current state of preparedness, and used to improve the response to future events. For this to happen, evaluation products must be perceived as useful by the end user. Unfortunately, it appears that this is not the case. Both evaluations and their products are rarely used to their full extent or, in extreme cases, are regarded as paper-pushing exercises. The first part of this research characterises current evaluation practice, both in the scientific literature and in Dutch practice, based on a scoping study, document and content analyses, and expert judgements. The findings highlight that despite a recent increase in research attention, few studies focus on disaster management exercise evaluation. It is unclear whether current evaluations achieve their purpose, or how they contribute to disaster preparedness. Both theory and practice tend to view, and present evaluations in isolation. This limited focus creates a fragmented field that lacks coherence and depth. Furthermore, most evaluation documentation fails to justify or discuss the rational underlying the selected methods, and their link to the overall purpose or context of the exercise. The process of collecting and analysing contextual, evidence-based data, and using it to reach conclusions and make recommendations lacks methodological transparency and rigour. Consequently, professionals lack reliable guidance when designing evaluations. Therefore, the second part of this research aimed to gain an insights into what make evaluations useful, and suggest improvements. In particular, it highlights the values associated with the methodology used to record and present evaluation outcomes to end users. The notion of an ‘evaluation description’ is introduced to support the identification of four components that are assumed to influence the usefulness of an evaluation: its purpose, object description, analysis and conclusion. Survey experiments identified that how these elements – notably, the analysis and/ or conclusions – are documented significantly influences the usefulness of the product. Furthermore, different components are more useful depending on the purpose of the report (for learning or accountability). Crisis management professionals expect the analysis to go beyond the object of the evaluation, and focus on the broader context. They expect a rigorous evaluation to provide them with evidence-based judgements that deliver actionable conclusions and support future learning. Overall, this research shows that the design and execution of evaluations should provide systematic, rigorous, evidence-based and actionable outcomes. It suggests some ways to manage both the process and the products of an evaluation to improve its usefulness. Finally, it underlines that it is not the evaluation itself that leads to improvement, but its use. Evaluation should, therefore, be seen as a means to an end.
Article
Full-text available
This study was evaluating the Ministry of Religion Affairs’ policy intending to provide input for improving the program. The evaluation took the case in Madrasah Ibtidaiyah (MI) Ma'arif in Ambal, Kebumen, Central Java. This MI uses the concept of inclusive education that sees every child entitled to educational services regardless of their physical differences and who they are. At the level of implementation, there is still a gap between a theory that tends to be ideal with practices that emphasize real needs. To meet the educational services of children with disabilities in their environment, MI Ma'arif Sidomulyo chose to go directly and practice. To overcome obstacles in the field, MI initiated the building of networks with local governments, schools, and madrassas that provide inclusive education. The network was also built with institutions and observers of inclusive education as an effort to strengthen inclusive education in MI Sidomulyo, because guidance from the Ministry of Religion is still very weak.AbstrakKajian ini mengevaluasi kebijakan Kementerian Agama yang hendak memberikan masukan untuk perbaikan program. Evaluasi mengambil kasus di Madrasah Ibtidaiyah (MI) Ma'arif di Ambal, Kebumen, Jawa Tengah. MI ini menggunakan konsep pendidikan inklusif yang memandang setiap anak berhak atas layanan pendidikan tanpa memandang perbedaan fisik dan siapa mereka. Pada tataran implementasi, masih terdapat gap antara teori yang cenderung ideal dengan praktik yang menekankan pada kebutuhan nyata. Untuk memenuhi layanan pendidikan anak difabel di lingkungannya, MI Ma'arif Sidomulyo memilih terjun langsung dan praktek. Untuk mengatasi kendala di lapangan, MI memprakarsai pembangunan jejaring dengan pemerintah daerah, sekolah, dan madrasah penyelenggara pendidikan inklusif. Jejaring juga dibangun dengan lembaga dan pemerhati pendidikan inklusif sebagai upaya penguatan pendidikan inklusif di MI Sidomulyo, karena pembinaan dari Kementerian Agama masih sangat lemah.
Thesis
Full-text available
The increasing adherence to self-regulatory behaviour of Multinational Enterprises under Private Transnational Governance Regimes has been viewed as a step in the right direction in the improvement of Corporate Sustainability Reporting. Hence, I investigate the output effectiveness of Global Reporting Initiative (GRI), environmental corporate sustainability reporting practices in accordance with the GRIS with a set of mixed methods and Benchmarking Scoring System. The case study looks at the Taiwanese computer sector with Asus and Acer as focus points. The results show that GRI has been overall successful in terms of output effectiveness by increasing the quality and quantity of the disclosed information, and an increase of the corporate capacity to incorporate GRIS. However, the output effectiveness has also been proven to result in unregular report contents indicating that corporations manage to comply with GRIS-Core Option of reporting, but with great discretion in terms of the quality and amount of specific content disclosures found within the GRIS. As GRI still lacks the absolute enforcement power of its sustainability reporting standards due to its voluntary regulatory regime nature, it remains incapable of addressing the 'governance gap' on its own.
Chapter
This chapter is aimed to discuss the ethical issues of qualitative research. Because of nature and forms of qualitative research, ethical issues, validity and reliability of qualitative research has been discussed in many research investigations. By nature ethical issues in qualitative research are vague and inherent. These ethical issues of qualitative research have been discussed by researchers and experts from different areas. This chapter discusses the focal points of ethical issues. The interactions and the interest of concerned parties of qualitative research form the ethical issues. Moreover, different views (Positivistic and relativistic) of ethical issues of qualitative research raise some complexity. This chapter discusses the various issue of qualitative research from various perspectives. Researchers' opinions on the ethical issues of qualitative research have been discussed, and after that the chapter has been arranged with the basic ethical issues of qualitative research as the area is vary vast.
Technical Report
Full-text available
Randall, G. E., Mulvale, G. M., Wakefield, P. A., Embrett, M. G., Barr, N. G., Miatello, A. M., Longo, C. J., Nguyen, T. D., Di Rezze, B. M., Vania, D. K., & McRae, S. A. (2016). Mapping the policy to practice landscape for youth mental health in Ontario: A report from the “youth to adult transitions in health care – the case of mental health services in Ontario” research team.
Article
Full-text available
THIS ARTICLE IDENTIFIES A NUMBER OF RESEARCH TOPICS related broadly to outcomes assessment in a networked environment and discusses issues affecting these research topics. It also proposes a framework to relate traditional evaluation components and terminology to the networked environment and identifies a number of factors in the networked environment that affect outcomes and other assessment methods. The article suggests that outcomes assessment has the potential to complement other assessment techniques to better assist libraries and related information organizations enhance their decisions in the provision of information services and resources. Given the increased rate at which libraries are using the networked environment to provide services and resources, however, much work remains before most libraries can implement outcomes assessment efforts successfully.
Article
Full-text available
This article examines the theoretical models applied to date in library assessment activities. A brief review of the history of library assessment practices and the evolution of their respective approaches is presented. A discussion of the theoretical concepts applied to these assessment activities in library and information science (LIS) as introduced from other fields, such as marketing and management information systems (MIS), fol- lows. The conceptual issues and practical concerns in library assessment are then discussed. Focus is placed on the review of research concepts of service quality, customer/user satisfaction, and their applications in library assessment activities.
Article
Full-text available
This report is derived from a large study sponsored by the Council on Library Resources. Two of the objectives of the study were to develop a taxonomy of value-in-use of library and information services based on users assessments and to propose methods and instruments for similar studies of library and information services in general. The corresponding results are reported in two parts. In this, the first part, we discuss underlying concepts related to value that must be clarified in order to proceed with any pragmatic study of value, and we establish a theory of use-oriented value of information and information services. We examine the notion of “value” in philosophy and economics and in relation to library and information services as well as the connection between value and relevance. We develop two models: One related to use of information and the other to use of library and information services. They are a theoretical framework for pragmatic study of value and a guide for the development of a Derived Taxonomy of Value in Using Library and Information Services. In the second part of this report, we present the methodology employed in development of the Taxonomy, the Taxonomy itself, and the results of testing the Taxonomy. We believe that the Taxonomy covers most dimensions of value related to use of library and information services. In the second part we also present suggestions for pragmatic applications of the Taxonomy. © 1997 John Wiley & Sons, Inc.
Article
Full-text available
Library managers must respond to institutional demands for the justification of maintaining or supplementing resources and services. The difficulty of assigning costs and values to library and information services and resources is magnified by the advent of a broad variety of computerized and electronic modes of library service. This article provides a general discussion of the issues relating to costs of library and information services in the library in transition, reviews the functional cost analysis method, and presents selected data gathered from a larger study on the costs and value of various library services. The results of applying functional cost analysis to reference service show great variation in unit costs, within and across libraries, indicating that there are substantial challenges remaining in determining the sources of these variations, and in understanding the cost basis of library services. © 1996 John Wiley & Sons, Inc.
Article
This is the seventh edition of a much- referred-to classic text on programme evaluation. As the authors point out, when the book was first published in the l970s, evaluation of social programmes was not well-established - a situation which is very different now with evaluation research embedded throughout all types of public and non-public organisations and on a global basis. This makes the new edition a little less revolutionary, although a lot more useful.
Article
In an electronic era, the evaluation of reference and related information services should still be based on the same principles used to evaluate traditional face-to-face reference services and printed reference tools. Traditional research methods - which are surveys and questionnaires, observation, individual and focus group interviews, and case studies - can be utilized very effectively in an electronic environment. However, electronic technologies offer interesting research opportunities not present in the traditional reference environment.
Article
This paper explores library evaluation in Australia and proposes a return to research fundamentals in which evaluators are asked to consider the centrality of philosophical issues and the role of different research methods. A critique of current evaluation examples demonstrates a system-centred, quantitative, input/output focus which fails to engage with users’ experiences or to address service objectives, and provides insufficient information upon which to make decisions about service improvements. This reliance on traditional evaluation is juxtaposed with the emphasis library and information studies places on alternative methodologies. The paper asks why a perspective that has become mainstream in academic research remains peripheral to practice-based evaluation.
Article
The article considers methodological issues which arose in undertaking a two-year British Library funded research project – Assessment Tools for Quality Management in Public Libraries – jointly managed by the Department of Information Science, Loughborough University and the Department of Information Studies, Sheffield University and discusses the principal findings. The aim of the research was to assess the potential of and then to develop and evaluate self-assessment techniques for the public library and information sector. The authors argue that by using an action research methodology they were best able to produce a self-assessment toolkit which matched the needs of public library services. Using evidence drawn from three case study library authorities and the literature, the authors go on to suggest that whilst self-assessment can have a role to play in the evaluation of library and information services, there is a real need to tailor and adapt tools to reflect specific service cultures and experience.
Article
Outcome-based evaluation is increasingly used for digital library projects. The point of outcome-based evaluation is to establish a project’s effectiveness with its target audience. Although many small-scale human-oriented project outcomes do not readily lend themselves to survey or statistical analysis, other research methods can be used to evaluate the amount and quality of a project’s effect. Anthropology offers not only a suite of observational methods, but standards for persuasiveness that apply equally to academics and to legislators.
Article
This manual is a follow-up to AGARD Lecture Series 'The Evaluation of Information Centres and Services,' LS 160, published in September 1988 (AD-A202 498). It suggests an approach to evaluation devised by the authors, which they have refined over the last 20 years during their involvement in nearly 300 projects involving the planning, evaluation and design of information centres and services. The manual emphasizes the use of evaluation measures, models and methods and includes data and results from actual studies. It also includes a comprehensive bibliography.
Article
New ways to evaluate libraries are needed. The SERVQUAL is a diagnostic tool to measure service quality, defined as the difference between customer perceptions and expectations of service. This article discusses implications for library management and future explorations of a tool applicable to academic libraries.
Article
The authors trace recent changes in the idea of evaluating performance in UK education and libraries and examine some of the differences in impact assessment between the two fields; concluding that little attention has so far been given by librarians to service impact assessment. A process model for generating outcome/impact indicators and targets was developed in the course of research on the effectiveness of further education college libraries. This was then adapted into a development planning model to enable education library managers to work through the processes required to arrive at appropriate impact indicators. The model is described, various issues in working with the model are explored and its further adaptation for public library service managers through another national research project is reported.
Article
Contenido: Parte I.Cuestiones conceptuales en la investigación cualitativa: Naturaleza de la investigación cualitativa; Temas estratégicos en la investigación cualitativa; Diversidad en la investigación cualitativa: orientaciones teóricas; Aplicaciones cualitativas particulares. Parte II. Diseños cualitativos y recolección de datos: Estudios de diseños cualitativos; Estrategias de trabajo de campo y métodos de observación; Entrevistas cualitativas. Parte III. Análisis, interpretación e informe: Análisis cualitativo e interpretación; Incrementar la calidad y la credibilidad del análisis cualitativo.
Article
THIS ARTICLE SUMMARIZES THE ATTEMPTS to define and measure quality and effectiveness in academic libraries, from traditional evaluative studies to Total Quality Management (TQM) and new research on user-defined criteria. Focusing on the organizational analysis of the library as a whole and the contribution it makes to the university or college, the article outlines a number of fundamental concepts and tools common to models of evaluation. Particular attention is then given to assessment in higher education as a whole and ways in which determinants of library quality must be linked to educational outcomes. The concluding sections suggest several areas for future research and for collaboration among library managers, educational administrators, scholars, and measurement theorists. ,.
Article
THE KEY CHARACTERISTICS OF evaluative research (ER) are outlined. Raizen and Rossi's fine-grained model of ER in education is applied to the library information field. Using published and unpublished examples of ER in library and information settings, the field's strengths and weaknesses in the various types of ER are explored. It is concluded that the overall volume of ER is reasonably good in the library and information field, but that it is fragmented and noncumulative, lacking sufficient basic research and research on the impact of libraries and information services and products.
Article
The empirical literature on cost-benefit analysis in health care concentrates on savings in direct expenditures and averted losses in earnings; valuation of intangible benefits has received little attention. A shift toward cost-effectiveness analysis in health care, although more limited, makes estimates more useful in decision-making within the public sector. But if well executed, it can also lay a firm foundation for future cost-benefit analysis.
The tell it! manual: The complete program for evaluating library performance
  • D Zweizig
Zweizig, D. (Ed.). (1994). The tell it! manual: The complete program for evaluating library performance. Chicago: American Library Association.
Evaluating the school library media center: Analysis techniques and research practices
  • N Everhart
Everhart, N. (1998). Evaluating the school library media center: Analysis techniques and research practices. Englewood, CO: Libraries Unlimited.
If you want to evaluate your library
  • F W Lancaster
Lancaster, F. W. (1993). If you want to evaluate your library... (2nd ed.). Champaign, IL: University of Illinois, Graduate School of Library and Information Science.
Evaluation of library service [Special Issues]
  • S R Reed
Reed, S. R. (Ed.). (1974). Evaluation of library service [Special Issues]. Library Trends, 22(3).
Evaluation of library collections, access and electronic resources: A literature guide and annotated bibliography
  • T E Nisonger
Nisonger, T. E. (2003). Evaluation of library collections, access and electronic resources: A literature guide and annotated bibliography. Westport, CN: Libraries Unlimited.
Benchmarking: A process for improvement Impact assessment of university libraries
  • T M R R Peischl
Peischl, T. M. (1995). Benchmarking: A process for improvement. Proceedings of the First International Conference on TQM and Academic Libraries (pp. 119-122). Washington, DC: Association of Research Libraries Powell, R. R. (1995). Impact assessment of university libraries. In Encyclopedia of library and information science (Vol. 55, pp. 151-164). New York: Marcel Dekker.
Evaluation and library decision making
  • P Hernon
  • C R Mcclure
Hernon, P., & McClure, C. R. (1990). Evaluation and library decision making. Norwood, NJ: Ablex.
Encyclopedia of evaluation
  • S Mathison
Mathison, S. (Ed.). (2004). Encyclopedia of evaluation. Thousand Oaks, CA: Sage Publications.
Building evaluation capacity: 72 activities for teaching and training
  • H Preskill
  • D Russ-Eft
Preskill, H., & Russ-Eft, D. (2005). Building evaluation capacity: 72 activities for teaching and training. Thousand Oaks, CA: Sage Publications.
Output measures for public libraries Chicago: American Library Association Measuring academic library performance: A practical approach
  • Van House
  • N A Lynch
  • M J Mcclure
  • C R Zweizig
  • D L Rodger
  • E J N Weil
  • B Mcclure
Van House, N. A., Lynch, M. J., McClure, C. R., Zweizig, D. L., & Rodger, E. J. (1987). Output measures for public libraries (2nd ed.). Chicago: American Library Association. Van House, N., Weil, B, & McClure, C. (1990). Measuring academic library performance: A practical approach. Chicago: American Library Association.
Library evaluation: A casebook and can-do guide
  • D P Wallace
  • C Van Fleet
Wallace, D. P., & Van Fleet, C. (2001). Library evaluation: A casebook and can-do guide. Englewood, CO: Libraries Unlimited.
What's good? Describing your public library's effectiveness
  • T A Childers
  • N A Van House
Childers, T. A., & Van House, N. A. (1993). What's good? Describing your public library's effectiveness. Chicago: American Library Association.
Collecting and using public library statistics: A how-to-do-it manual for librarians
  • M L Smith
Smith, M. L. (1996). Collecting and using public library statistics: A how-to-do-it manual for librarians. New York: Neal-Schuman.
ARL new measures retreat; Transcript
  • Research Association
  • Libraries
Association of Research Libraries. (2004). ARL new measures retreat; Transcript. Washington, DC: Association of Research Libraries.