No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
A Program Evaluation Study for Measurement and Evaluation Course in Distance Education
Abstract
This study aims to evaluate the online measurement and evaluation course in teacher training programs during the COVID-19 process. In the study, we sought answers to two primary research questions: What are the opinions of teachers and school administrators regarding their measurement and evaluation competencies? Does the online "measurement and evaluation" course have the qualities of an effective program in the "antecedents, transactions, and outcomes" dimension? We structured the research into two phases within a multistage evaluation design framework. The findings show that there were problems and positive aspects in all dimensions of the program. For example, adapting teacher training programs developed before COVID-19 to distance education processes was challenging. In distance education, some practices contradict the modern teaching and assessment approach. Such problems were reflected in teachers' acquisition of measurement and evaluation competencies. The achievement test we applied to the observed groups also confirmed these findings. For this reason, responsible organizations should not ignore the fact that we cannot renounce distance education. During program development, they should reconsider how the teachers will acquire measurement and evaluation competencies and how we will measure and evaluate in distance education.
ResearchGate has not been able to resolve any citations for this publication.
Originally published as Paper #5, Occasional Paper Series, November 1975. * Paper presented at a conference on “New Trends in Evaluation”, Goteborg, Sweden, October, 1973.
Well-designed online courses enhance learning experiences and allow effective development of learners' skills and knowledge. A critical factor contributing to the design of online courses in the higher education settings are well-defined learning objectives that align with course assessments and learning activities. While there are several introspective instruments to evaluate course designs, with the broader adoption of educational technologies and digital tools, there is a wealth of data that offers insights on the alignment of learning objectives to assessments. Such data has paved the way for evidence-based methods of investigating course effectiveness within higher education. This study outlines a methodology for designing and evaluating the alignment between course learning objectives and assessment activities at scale, utilising a combination of learning analytics and measurement theory approaches, more specificially exploratory multi-dimensional item response theory (MIRT) models. We demonstrate the proposed methodology within a professional development MOOC on leadership skills development, where we evaluate the alignemnt between course objectives and reflective writing assessments activities. Our results suggested that the alignment of the existing course objectives to assessment activities can be improved, showing the practical value of the proposed approach. The theoretical and practical implications of this research are further illustrated.
Given that existing research on teacher assessment literacy has focused on teachers’ ability to evaluate students’ academic achievements, this study aims to develop a conceptualization of teachers’ assessment literacy in holistic competencies. Guided by four dimensions of teacher assessment literacy (knowledge, attitude, practice, socio-emotional management) identified from the existing literature, data collected from 18 individual interviews and one focus group interview with academics from four Hong Kong universities were analysed using the constant comparison method. The findings show that although the participants’ conceptions concurred with the four dimensions in general, some aspects were particularly crucial in the context of holistic competency assessment: for example, positive teacher-student relationships and subjectivity in assessment. Findings from the study allowed us to revise the four-dimensional model in the context of holistic competency assessment with more elaborated forms of knowledge, attitude and capabilities that a holistic competency assessment literate teacher might be expected to possess and enact.
Electronic exam's' non-reliability is one of the most critical gaps in this type of exam, whereas any electronic exam is considered entirely unreliable. In addition to that, it is not protected and free from cheat. The continuous validation that the examinee is the one who solves the questions of the electronic exam permanently and continuously is the most crucial goal of this research. Many methods and procedures will be discussed, which will protect the validity of this type of exam. For example, one of these methods matches the student's fingerprint and takes random snapshots while taking the electronic exam in more than one format. In this research, it's proposed to use student handwriting during the electronic exams to built the value of validity. This way is considered one of the rare methods in the electronic exam. The research also tackles the process of reality for electronic exams through a fingerprint during the examination period. It takes random snapshots for the student without paying attention to him or appearing those pictures to the observer or correcting the exam. The difficulty of offering a safe and high-credibility exam and reducing the phenomenon of cheating requires excellent capabilities in some countries. It also needs prior arrangements and good preparation. It also needs to develop technological capabilities continuously in a high-cost material that may reduce the cheating rates in such a kind of exam.
The purpose of this study is to determine measurement and evaluation literacy levels of the prospective science teachers who are studying in the 4th grade level of science teaching department in different universities of Higher Education Institution and to examine measurement and evaluation literacy levels in terms of various variables. In addition, it is aimed to get the opinions of the prospective science teachers about measurement and evaluation literacy and the factors affecting this literacy. Within this context, the research group of the research was determined according to the maximum variation sampling method, which is one of the purposive sampling methods. Participants consisted of 290 prospective teachers in the fourth year of the science teaching department in seven different universities. Quantitative and qualitative research methods were used together in the research, Assessment Literacy Inventory (ALI) was administered to determine measurement and evaluation literacy levels of the participants. After the collected data were processed and required statistical procedures were performed, Semi-Structured Interview Form was applied to pre-service teachers' opinions about assessment and evaluation literacy and the factors affecting measurement and assessment literacy. In this form, the participants' opinions about measurement and evaluation literacy and the potential factors affecting this literacy were taken. The data were analysed by using SPSS 23 package program. Relational and descriptive statistical analysis techniques and one-way analysis of variance (ANOVA) were used to interpret the data. As a result of the analysis of the data, measurement and evaluation literacy levels of the 4th grade prospective science teachers studying at different universities were low. In addition, the participants stated that they had acquired the basic information about measurement and evaluation in their measurement and evaluation course during undergraduate education and that was insufficient. Besides, the fact that some subjects and concept in the inventory were seen by the participants by the first time shows that this course is not sufficient for measurement and evaluation which is one of the crucial courses of the teaching profession knowledge. When the findings obtained from the inventory and semi-structured interviews were evaluated in general, it can be said that the prospective science teachers need more knowledge, skills and practice to be educated as measurement and evaluation literate individuals.
Bu araştırmanın amacı, Covid-19 pandemisi döneminde Türkiye’deki üniversitelerin acil uzaktan eğitime geçişte yaptıkları çalışmaları incelemektir. Araştırmada tarama modeli kullanılmıştır. Araştırma evrenini Türkiye’deki tüm üniversiteler (208 üniversite) oluşturmaktadır. Her bir üniversite ile ilgili veriler üniversitelerin acil uzaktan eğitim sürecinde görevli kişilerden (UZEM’de ve Bilgi İşlem Dairesi’ndeki görevliler vb.) edinilmiştir. Araştırmada tüm evrene erişilmeye çalışılmış ancak 33 katılımcıya (üniversiteye) ulaşılabilmiştir. Araştırmanın verileri bir çevrimiçi anket formu (Google Form) ile toplanmıştır. Araştırmadan elde edilen bulgulara göre en çok kullanılan öğrenme yönetim sistemleri Moodle ve ALMS’dir. Üniversiteler tarafından en çok kullanılan canlı ders yazılımlarının Big Blue Button ve Perculus olduğu görülmüştür. YÖK’ün derslerin senkron işlenmesini tavsiye etmesine rağmen tüm derslerini senkron olarak yürütebilen üniversite sayısı sadece 6’dır. Üniversitelerin çoğu daha önce kurulu olan öğrenme yönetim sistemi (f=29) ve canlı ders yazılımı (f=24) üzerinden süreçleri yönetmeye çalışmışlardır. Üniversitelerin yaklaşık yarısı öğrencilerin ders devam takibini yapmıştır. Katılımcılar, uzaktan eğitime hazırlık sürecinde öğretim elemanlarının eğitimini en çok zorlandıkları durum olarak belirtmişlerdir.
Üniversite öğrencilerinin derslere devamlarının akademik başarıya etkisi birçok akademik çalışma ile irdelenmiş ve derslere devam eden öğrencilerin başarı oranlarının daha yüksek olduğu tespit edilmiştir. Devam durumları birçok yükseköğretim kurumunda klasik yöntemlerle takip edilmektedir. Gelişen teknoloji ile öğrencilerin ders devam takibini bilgi sistemleri vasıtası ile yapmak mümkün olmuştur. Bu çalışmada, temassız akıllı kartlar kullanılarak öğrenci devam takip sistemi tasarlanmış ve geliştirilmiştir. Sistemin yönetimi ve devam durumlarının takibi için web uygulama ara yüzleri geliştirilmiştir. Sistem ile ders devam takibi daha hızlı ve güvenilir hale getirilmiştir. Bu sisteme bütünleşmiş veri madenciliği modülü ile toplanan veriler analiz edilmekte, böylece ileriye dönük önlemler alınabilmekte ve geliştirmeler yapılabilmektedir.
The impact to the academic achievement of the university students' attendance to the courses was examined by many academic studies and it has been found that the success rates of the students attended to the courses are higher. Attendance statuses of students are tracked by classical methods in many higher education institutions. It is possible to track student attendance utilizing current information systems. In this paper, an attendance tracking system has been designed and developed by using contactless smart cards. Web applications for management of the system and tracking the student attendance have been developed. Tracking student attendance has become faster and more reliable with the system. The collected data is analyzed with the data-mining module integrated into this system, thus proactive measures can be taken and improvements can be made.
Ölçme-değerlendirme konusu, öğretmenlik yeterlik alanlarının en temel öğelerinden bir tanesidir. Gelişmiş seviyede ölçme değerlendirme okuryazarlığına sahip olarak yetiştirilen
öğretmen adaylarının, okullarda verilen eğitimin kalitesinin artmasına yardımcı olacağı bir gerçektir. Bu araştırmada, farklı bölümlerde öğrenim gören dördüncü sınıf öğretmen adaylarının ölçme-değerlendirme okuryazarlık düzeyleri, ölçme-değerlendirmeye ilişkin düşünce ve tutumları incelenmiştir. Bu amaçla üç farklı ölçek, Çanakkale Onsekiz Mart Üniversitesi dördüncü sınıfta okuyan 289 öğretmen adayına uygulanmıştır. Betimsel analiz sonuçlarında, öğretmen adaylarının ölçme-değerlendirme okuryazarlık düzeylerinin
düşük olduğu ve geliştirilmesi gerektiği ortaya çıkmıştır. Öğretmen adaylarının ölçmedeğerlendirmeye yönelik düşünce ve tutumlarının ise daha çok yapılandırmacı yaklaşımda olduğu tespit edilmiştir. Neticede bu çalışma, öğretmen adaylarının ölçme-değerlendirme
alanına yönelik bilgi, beceri ve tutumlarının anlaşılmasına katkıda bulunmuştur.
Anahtar kelimeler: ölçme-değerlendirme okuryazarlığı, ölçme-değerlendirmeye ilişkin düşünceler, ölçme-değerlendirmeye ilişkin tutumlar
Uzaktan eğitimin en önemli bileşenleri arasında etkileşim yer almaktadır. Alan yazında öğrenen-içerik, öğrenen-öğrenen, öğrenen-öğretim elemanı ve öğrenen-arayüz etkileşim türleri bulunmaktadır. Bu çalışmanın amacı, yapılmakta olan uzaktan eğitim uygulamalarında alan yazında var olan etkileşim türlerinin gerçekleşme durumlarını incelemektir. Çalışma nitel veri toplama teknikleriyle gerçekleştirilmiştir. Çalışmada durum çalışması deseni kullanılmış ve katılımcılar amaçlı örneklem türlerinden ölçüt örnekleme göre belirlenmiştir. Katılımcılarla yapılan yüz-yüze görüşmelerde görüşme formları aracılığıyla veriler toplanmıştır. Çalışmaya altı öğrenen ve bir öğretim elemanı dahil edilmiştir. Elde edilen veriler içerik analizi aracılığıyla yorumlanmıştır. Çalışmadan elde edilen bazı bulgular, öğrenenlerin kendi aralarındaki veya dersin öğretim elemanıyla olan etkileşimlerinin çok zayıf olduğu şeklindedir. Bununla beraber öğrenenlerin ders içerikleriyle olan etkileşimlerinde, derslerin videolarının olumlu etkileri görülebilmektedir. Sonuç olarak, alanyazında belirtilen tüm etkileşim türlerinin zayıf düzeyde gerçekleştiği anlaşılmaktadır. Çalışmanın önerileri arasında, öğrenenlerin etkileşimli bir arayüz sayesinde uzaktan eğitim sistemini daha etkin bir şekilde kullanabilmesi yer almaktadır. Ayrıca etkileşimli ders içerikleri sayesinde öğrenenlerin derslere yönelik dikkatleri daha iyiye çevrilebilir.
Interaction is located in one of the most important components of distance education. There are four types of interaction as learner-content, learner-learner, learner-instructor and learner-interface in the literature. The purpose of this study was to investigate the interaction types in distance education. The study was conducted with qualitative data collection methods. Case study design was used in this study and participants were selected through criterion sampling as a purposeful sampling method. Six learners and one instructors were included in the study. The data were collected through structured interview forms throughout the study. The obtained data were interpreted through thematic analysis. Some of the findings obtained from the study, it is understood that there are very weak interaction levels between learner-learner and learner-instructor. However, positive effects of the course videos can be seen when learners watching instructors online or on recorded course videos on learner’s interaction with course content and instructors. Consequently, all types of interactions specified in the literature are weak. Learners can use distance education system more effectively by interactive interface. Furthermore, learners interest in the course can be increased by interactive course contents.
This study aims at investigating the needs related to the attainments, content, processes of teaching-learning and measurement-evaluation aspect of Measurement and Evaluation course. This research was designed as a case study which is one of the qualitative research designs. The most common problem in Measurement and Evaluation courses was found to be the statistics and the source of this problem was students’ insufficient mathematical knowledge and skills. It was revealed that there was a need to include more practice, examples and activities in the course. In addition, it was understood that measurement instruments which are peculiar to each discipline need to be included in the scope of the course. Besides, as well as theoretical bases of the course, concrete practices and examples encompassing knowledge and skills that can be used in teaching profession might be included in the course.
Romania is an active player in various international higher education areas, while undergoing a series of higher education reforms within its national framework. The Higher Education Evidence Based Policy Making: a necessary premise for progress in Romania project was implemented by the Executive Agency for Higher Education, Research, Development and Innovation Funding (UEFISCDI) in the timeframe February 2012 - February 2014, being co-financed by the European Social Fund through the Operational Programme "Administrative Capacity Development". The project aimed to increase the capacity of public administration for evidence-based policy making in the field of higher education, while focusing on good practices at international level and impact assessment. With the contribution of the national and international experts, the project has generated a number of analysis and studies on the existing higher education public policies (quality assurance, internationalisation, equity, data collection, the Bologna Process, financing of higher education). Based on the results of the project, the book will reunite a number of policy research articles which would tap into the innovative aspects of the project's activities and provide a concise overview of what good practices can be drawn from the empirical research conducted in this project. The book will therefore aim to improve the information on Romanian higher education reforms, as well as on the concrete evidence-based policy proposals which could be transformed into future policy solutions in the Romanian higher education system. © 2015, Springer International Publishing. All rights resereved.
Assessment literacy is a core professional requirement across educational systems. Hence, measuring and supporting teachers’ assessment literacy have been a primary focus over the past two decades. At present, there are a multitude of assessment standards across the world and numerous assessment literacy measures that represent different conceptions of assessment literacy. The purpose of this research is to (a) analyze assessment literacy standards from five English-speaking countries (i.e., Australia, Canada, New Zealand, UK, and USA) plus mainland Europe to understand shifts in the assessment landscape over time and across regions and (b) analyze prominent assessment literacy measures developed after 1990. Through a thematic analysis of 15 assessment standards and an examination of eight assessment literacy measures, results indicate noticeable shifts in standards over time yet the majority of measures continue to be based on early conceptions of assessment literacy. Results also serve to define the multiple dimensions of assessment literacy and yield important recommendations for measuring teacher assessment literacy.
This paper provides a description of 30 years of research conducted on curriculum-based measurement. In this time span, several subject matter areas have been studied—reading, writing, mathematics, and secondary content (subject) areas—in developing technically adequate measures of student performance and progress. This research has been conducted by scores of scholars across the United States using a variety of methodologies with widely differing populations. Nevertheless, little of this research has moved from a “measurement paradigm” to one focused on “training in data use and decision making paradigm.” The paper concludes with a program of research that is needed over the next 30 years.
Optimal outcomes of the educational assessment of students require that teachers should have adequate knowledge of, strong skills in, and favourable attitudes toward educational measurement. The present study investigated differences between preservice and inservice teachers' knowledge of, perceived skills in, and attitudes toward educational measurement. Participants were 279 preservice teachers and 233 inservice teachers from Oman. Results indicated that inservice teachers had a lower level of knowledge, a higher level of perceived skilfulness, and a more favourable attitude toward educational measurement than preservice teachers. In addition, the results not only testified to the value of preservice measurement training, but also showed the merit of teaching practicum and teaching experience when preparing teachers in educational measurement. Implications for professional preparation in educational measurement as well as recommendations for future research are discussed.
This paper presents a research and its results in the domain of higher education's pedagogical patterns for remote assessments - precisely in the computer science, software engineering and informatics-related courses. This research was motivated by the COVID-19 crisis, which separated teachers, teaching assistants, and students physically. During this period, remote knowledge assessment was one of the most challenging among all educational activities. The lack of available resources and advice on remote knowledge assessment revealed a need for a specialized assessment pattern catalog.
The main result of the research is the assessment pattern catalog that started to grow organically at the Institute of Informatics, where we teach IT-related courses. We started with the initial set of patterns, identified by analyzing recurring practices, applied by teaching staff for remote assessments in the period from March 2020 till December 2020. The patterns were aggregated and gradually refined using a systematic approach. In addition to guided workshops, a systematic literature review was employed, followed by catalog refinements, and, finally, an extensive survey was carried out among teachers and teaching assistants. The latter was used as a validation of the correctness of the novel assessment pattern catalog, as well as the presented patterns’ suitability and popularity among users.
The resulting assessment pattern catalog presented in this paper boasts 47 patterns, classified into four main categories, that support the whole process of (remote) assessment. It is organized and documented systematically. It also boasts several indicators per each pattern to demonstrate its suitability for distant assessments, popularity rankings among teachers, teaching assistants, and top picks in every category per teachers and teaching assistants.
The survey that we performed revealed a subset of patterns that are important for a successful remote assessment, validated in the IT-related courses. Based on the results, the presented assessment pattern catalog showed itself to be useful not only for the remote assessment but also for judging knowledge in the classroom successfully.
Aim
The number of online graduate nursing programs across the United States has increased to address a critical shortage of nurse educators. Web-based learning appeals to nurses returning to school as a means of gaining an education at their convenience. More schools are offering compressed courses to meet this demand. Although students have a preference toward shorter intensive online courses, it is unclear how that affects the quality of the learning experience such as student engagement. The study explored the effect of course length on the student learning experience in a graduate online nurse educator course.
Design
Using the community of inquiry framework, this study examined the effect of course duration (8-week versus traditional 16-week timeframes) on student engagement, student perceptions of the learning experience and self-reported learning behaviors. Study participants were enrolled in an online graduate nurse educator program located in the northwest United States.
Methods
Data were collected using a background information form, a course evaluation form and the Community of Inquiry Questionnaire which measured teaching presence, social presence and cognitive presence. Data were analyzed using descriptive and inferential statistics.
Results
High mean scores on the questionnaire showed that a community of inquiry was established regardless of course duration. However, there were differences in terms of the social and teaching presence subscales but not in the cognitive presence subscale suggesting that students in the traditional course were better able to establish the type of rapport with each other that increased comfort and engagement with peer interactions. Independent t-tests revealed statistically significant differences in perceptions of time to complete course activities. Students in the 16-week course were more likely to report that they had adequate time to complete course teachings, think critically about course content, complete course assignments and thoughtfully engage in course discussion and that they performed their best on assignments.
Conclusions
The findings support the traditional course duration over an intensive 8-week format because it allows for students to build a better rapport and greater student engagement with the course materials and peers. The study reinforces previous work on distance education noting social presence and connectedness as essential to optimal online learning. Using the community of inquiry framework and best-practice pedagogies for online education in the design and development of online courses can contribute to greater collaboration and deeper learning.
We present the perspectives of Portuguese pre-service teachers about a formative strategy developed to promote learning about language and literacy education. The strategy was underpinned by theories about the pedagogical content knowledge (PCK), rehearsed (or simulated) agency, the epistemology of reflective practise and assessment for learning. It was implemented during a whole semester, after which pre-service teachers answered to a questionnaire focusing on their perceptions about their learning and the learning experience. The results of the quantitative and qualitative analysis of the collected data reveal positive and critical perceptions about the construction of PCK and agentic identities, evidencing the role of curricular analysis, rehearsed practice, reflection and assessment in the learning process. The final discussion, which highlights the possibilities and challenges of the strategy, aims to contribute to the construction of the Scholarship of Teaching and Learning of pre-service teachers after the Bologna Process.
The leading text that covers both the theory and practice of evaluation in one engaging volume has now been revised and updated with additional evaluation approaches (such as mixed methods and principles-focused evaluation) and new methods (such as technologically based strategies). The book features examples of small- and large-scale evaluations from a range of fields, many with reflective commentary from the evaluators; helpful checklists; and carefully crafted learning activities. Major theoretical paradigms in evaluation—and the ways they inform methodological choices—are explained. Readers learn effective strategies for clarifying their own theoretical assumptions; working with stakeholders; developing questions; using quantitative, qualitative, and mixed methods designs; selecting data collection and sampling strategies; analyzing data; and communicating and utilizing findings. The new companion website provides extensive recommended online resources and tools, organized by chapter.
Curriculum-Based Measurement of Oral Reading (CBM-R) is often used to monitor student progress and guide educational decisions. Ordinary least squares regression (OLSR) is the most widely used method to estimate the slope, or rate of improvement (ROI), even though published research demonstrates OLSR’s lack of validity and reliability, and imprecision of ROI estimates, especially after brief duration of monitoring (6-10 weeks). This study illustrates and examines the use of Bayesian methods to estimate ROI. Conditions included four progress monitoring durations (6, 8, 10, and 30 weeks), two schedules of data collection (weekly, biweekly), and two ROI growth distributions that broadly corresponded with ROIs for general and special education populations. A Bayesian approach with alternate prior distributions for the ROIs is presented and explored. Results demonstrate that Bayesian estimates of ROI were more precise than OLSR with comparable reliabilities, and Bayesian estimates were consistently within the plausible range of ROIs in contrast to OLSR, which often provided unrealistic estimates. Results also showcase the influence the priors had estimated ROIs and the potential dangers of prior distribution misspecification.
Educators increasingly need to evaluate schoolwide reform efforts; however, complex program evaluations often are not feasible in schools. Through a case example, we provide a heuristic for program evaluation that is easily replicated in schools. Criterion-referenced interpretations of schoolwide screening data were used to evaluate outcomes associated with participation in four-year-old kindergarten. Nonparametric analyses allowed for group comparisons across early literacy screening outcomes. Risk ratios demonstrated that four-year-old kindergarten participants were less likely to score “at-risk” on kindergarten and first grade screenings. The methods employed meaningfully addressed local program effectiveness questions. Further, they were easily determined and disseminated. Implications for extensions of the heuristic to other evaluation questions and data sources as well as limitations of the approach are discussed.
This chapter is a review and update of the so-called CIPP Model1 for evaluation. That model (Stufflebeam, 1966) was developed in the late 1960s as one alternative to the views about evaluations that were most prevalent at that time — those oriented to objectives, testing, and experimental design. It emerged with other new conceptualizations, especially those developed by Scriven (1966) and Stake (1967). (For a discussion of these historical developments, see Chapter 1 of this book.) The CIPP approach was applied in many institutions; for example, the Southwest Regional Educational Laboratory in Austin, Texas; the National Center for Vocational and Technical Education; the U.S. Office of Education; and the school districts in Columbus, Toledo, and Cincinnati, Ohio; Dallas, Forth Worth, Houston, and Austin, Texas; and Saginaw, Detriot, and Lansing, Michigan. It was the subject of research and development by Adams (1971), Findlay (1979), Nevo (1974), Reinhard (1972), Root (1971), Webster (1975), and others. It was the central topic of the International Conference on the Evaluation of Physical Education held in Jyvaskyla, Finland in 1976 and was used as the advance organizer to group the evaluations that were presented and discussed during that week-long conference. It was also the central topic of the Eleventh National Phi Delta Kappa Symposium on Educational Research, and, throughout the 1970s it was referenced in many conferences and publications. It was most fully explicated in the Phi Delta appa book, Educational Evaluation and Decision Making (Stufflebeam et al., 1971) and most fully implemented in the Dallas Independent School District. Its conceptual and operational forms have evolved in response to critiques, applications, research, and parallel developments; and it continues to be referenced and applied in education and other fields.
This article reported the concurrent, predictive, and diagnostic accuracy of a computer-adaptive test (CAT) and curriculum-based measurements (CBM; both computation and concepts/application measures) for universal screening in mathematics among students in first through fourth grade. Correlational analyses indicated moderate to strong relationships over time for each measure, with correlations between CAT and CBM measures across the three assessment periods low to moderate, with the strongest relationships between the CAT and CBM concepts/application measure. Relationships to the state assessment for math for third- and fourth-graders was found to be stronger for the CAT measure than for either the CBM computation or concepts/application measures, with the CAT measure the only significant predictor of the state assessment. Diagnostic accuracy indices found all measures to produce acceptable levels of specificity but limited levels of sensitivity. The study offered one of the first direct comparisons of CAT and CBM measures in screening for mathematics. Implications of using CAT and CBM measures in conducting screening in elementary mathematics were discussed.
Responsive evaluation builds upon the methods of informal evaluation in disciplined ways: getting personally acquainted with the evaluand, observation of activities, interviewing people who are in different ways familiar with the evaluand, searching documents that reveal what happened in the past or somewhere else. It calls for sustained effort to know quality and insufficiency, by different definitions.
A form which students can use to assess their class experiences was presented. A factor analysis based on evaluations filled out by 1,648 students revealed four factors which measured (a) the quality of the instructors’ presentations, (b) the evaluation process and the student-instructor interactions, (c) the degree to which the students were stimulated and motivated by the instructors, and (d) the clarity of the tests. A further analysis indicated that subscale scores which reflected the factor scores could be developed from the total item pool.
This report provides an overview of a selection of measurement error studies conducted on National Center for Education Statistics (NCES) surveys. Its intent is not to offer new analyses of program data, but to summarize information from internal memoranda, working papers, and adjudicated reports about errors of measurement that occur during five phases of survey operations: sample selection; data collection; data processing; estimation and analysis; and dissemination of results and postsurvey evaluation. The report illustrates the diversity of NCES efforts in this area and focuses on the major national surveys NCES conducts. The emphasis of the review is on reinterview studies, but other types of empirical studies of measurement error are discussed, including multiple indicators studies, record check studies, and cognitive studies. The following chapters are included: (1) "Introduction and Overview"; (2) "Profile of NCES Reinterview Studies"; (3) "Reinterview Studies: Simple Response Variance and Response Bias"; (4) "Reinterview Studies: Reliability and Validity"; (5) "Multiple Indicators' Studies"; (6) "Record Check Studies"; (7) "Cognitive Studies"; and (8) "Summary." An appendix presents a summary of a reinterview study on mode effects for the 1990-91 Schools and Staffing Survey. (Contains 60 tables, 1 exhibit, 31 figures, 4 tables in the appendix, and 102 references.) (SLD)
A 67-item Assessment Practices Inventory (API) was administered to 311 inservice teachers. The application of principal components analysis to the data yielded a 6-factor solution that explained 64% of the variance. The Rasch rating scale model was applied to the API to estimate item calibrations. The factor analyzed assessment categories were then ranked in order by difficulty based on mean logits. The distribution of mean logits ranged from -.35 to 0.78. Communicating assessment results was the easiest assessment category. Interpreting standardized test results, conducting classroom statistics, and using assessment results in decision making constituted the most difficult assessment categories. Nonachievement-based grading was more difficult than recommended grading practices, and performance assessment was more difficult than paper-pencil tests. The identification of the hierarchy of classroom assessment categories provided useful information for measurement training and teacher education in assessment. The findings justified ongoing research on grading practices, and supported the call in the assessment community for a shift of instructional emphasis from traditional objective tests to alternative assessments. (Contains 2 figures, 7 tables, and 53 references.) (Author/SLD)
Three national educational organizations articulated seven areas of teacher competency in student assessment. Among 555 teachers surveyed nationwide, the area of best performance was "administering, scoring, and interpreting test results," and the area of worst performance was "communicating test results." Development of inservice training materials is discussed. Includes standards. (SV)
Using standardized test scores to determine program effectiveness may contribute to further inequities.
