ArticlePDF Available

The structure of an online assessment of science and social studies content: testing optional formats of a general outcome measure

Authors:

Abstract and Figures

Research questions addressed the structure of an online form of content-focused general outcome measurement known as critical content monitoring. The assessment tool is designed to serve as an accurate and efficient measure learning performance and progress in science and social studies classes. Scores from single administrations of critical content monitoring probes that varied in content and length were correlated with results from a statewide accountability content test for a sample of American fifth-grade students. The magnitude of correlation was moderate for probes that included a single content (i.e., social studies) or a mix of two subjects (i.e., social studies and science). Comparable correlations were reported for probes that featured reduced time or increased questions. Limitations and research implications are discussed.
Content may be subject to copyright.
81
THE STRUCTURE OF AN ONLINE ASSESSMENT
OF SCIENCE AND SOCIAL STUDIES
CONTENT: TESTING OPTIONAL FORMATS OF
A GENERAL OUTCOME MEASURE
Paul Mooney, Kevin S. McCarter, Robert J. Russo, Danielle L. Blackwood
Louisiana State University, USA
Abstract
          
     
            



      
       


Key Words: Content Area Assessment, Formative Assessment, Technology
The Structure of an Online Assessment of Science and Social Studies Content:
Testing Optional Formats of a General Outcome Measure
            
            

      Embedded Formative Assessment,
         
        




 




Critical Content Monitoring
 
   

          

THE STRUCTURE OF AN ONLINE ASSESSMENT OF SCIENCE AND SOCIAL STUDIES CONTENT:
TESTING OPTIONAL FORMATS OF A GENERAL OUTCOME MEASURE
Paul Mooney, Kevin S. McCarter, Robert J. Russo, Danielle L. Blackwood
S O C I A L W EL F A R E INTERDISCIPLINARY APPROACH 2014 4 ( 1 )
82
     
th


        




          
            

      

          


            
       
    
          

    


           
     





       


         
           
    
          
           


          


          
           
             

83



          
        

            
     
          


      





  





       


  
             


                
          


             



Object of the research 

         
  

Research Rationale and Questions
          
          
          
           

THE STRUCTURE OF AN ONLINE ASSESSMENT OF SCIENCE AND SOCIAL STUDIES CONTENT:
TESTING OPTIONAL FORMATS OF A GENERAL OUTCOME MEASURE
Paul Mooney, Kevin S. McCarter, Robert J. Russo, Danielle L. Blackwood
S O C I A L W EL F A R E INTERDISCIPLINARY APPROACH 2014 4 ( 1 )
84
   

    

          

            


 
     
            





         
 

        
             


     
         
          
          

Method
Participants and Setting
         


             
   
    






            
        




85
Measures

integrated
i
Critical content monitoring.
               
   


        








    

iLEAPgrade 5 criterion-referenced test.  i
  


           
            
            
      

         

i   
   


i
th











THE STRUCTURE OF AN ONLINE ASSESSMENT OF SCIENCE AND SOCIAL STUDIES CONTENT:
TESTING OPTIONAL FORMATS OF A GENERAL OUTCOME MEASURE
Paul Mooney, Kevin S. McCarter, Robert J. Russo, Danielle L. Blackwood
S O C I A L W EL F A R E INTERDISCIPLINARY APPROACH 2014 4 ( 1 )
86
              
              
      

     







 


Results
Question 1: Social Studies


i

Question 2: Mixed Content







Question 3: Reduced Time

i
i

Question 4: Increased Questions

i
        i   

Discussion


 

              
  
 
              
87



             


        

r
               



               





Table 1.


Probe Mean SD  r

      

      

      
* p
Table 2.           
         

Probe Mean SD  r
      
   2.5   
      

      

      
* p
THE STRUCTURE OF AN ONLINE ASSESSMENT OF SCIENCE AND SOCIAL STUDIES CONTENT:
TESTING OPTIONAL FORMATS OF A GENERAL OUTCOME MEASURE
Paul Mooney, Kevin S. McCarter, Robert J. Russo, Danielle L. Blackwood
S O C I A L W EL F A R E INTERDISCIPLINARY APPROACH 2014 4 ( 1 )
88
  

          




      

Limitations


     
       
           
          



i
              
            

                 






      
             
               
   





        
            

      
            

         
            

89



           
          
       
        






    


          
     





        
            





              






       
     





      
        
          


       
THE STRUCTURE OF AN ONLINE ASSESSMENT OF SCIENCE AND SOCIAL STUDIES CONTENT:
TESTING OPTIONAL FORMATS OF A GENERAL OUTCOME MEASURE
Paul Mooney, Kevin S. McCarter, Robert J. Russo, Danielle L. Blackwood
S O C I A L W EL F A R E INTERDISCIPLINARY APPROACH 2014 4 ( 1 )
90

  
     

           
      

sample.
Conclusion
            
         

               
         
      
         


            


References
        
.
                
School
Psychology Review, 39
 
   
Effective teaching strategies that accommodate diverse learners 

               
  Assessment forEffective Intervention, 38 

           
     Assessment for Effective Intervention, 35  

              
Psychology in the
Schools, 47
   
        
Research-based strategies for improving outcomes academics (pp.

          Exceptional
Children, 52
 
Focus on Exceptional Children, 19
 Exceptional Children, 76 

91
    
        
    Assessment for Effective Intervention, 38   

       
             
Learning Disabilities Research & Practice, 16
 

Diagnostique, 20
 
   Journal of Learning
Disabilities, 38
               
School Psychology Review, 33
            
Exceptional Children, 57
            
Special Services in the Schools, 17
 
    Assessment for Effective Intervention,
31
             Collaborative
strategic reading: Strategies for improving comprehension
 
i
      iLEAP2010 Technical Summary. 

 
 
      
          
Beyond Behavior, 17
                  
Assessment for Effective
Intervention, 38
             
        Exceptional
Children, 80 
 
 Assessment for Effective Intervention, 35

    
Reading Research Quarterly, 47 
 
The Journal of Special Education, 46

     th 
  

 
Reading Research Quarterly, 42
            
Assessment for Effective Intervention,
38
THE STRUCTURE OF AN ONLINE ASSESSMENT OF SCIENCE AND SOCIAL STUDIES CONTENT:
TESTING OPTIONAL FORMATS OF A GENERAL OUTCOME MEASURE
Paul Mooney, Kevin S. McCarter, Robert J. Russo, Danielle L. Blackwood
S O C I A L W EL F A R E INTERDISCIPLINARY APPROACH 2014 4 ( 1 )
92
 Assessing vocabulary
 Assessment in special and inclusive education th

 
Psychology in the Schools, 42

 
 Interventions for achievement and behavior
problems
          
The Journal of Special Education, 44
 Embedded formative assessment
 
School Psychology Review, 36
THE STRUCTURE OF AN ONLINE ASSESSMENT OF SCIENCE AND SOCIAL
STUDIES CONTENT: TESTING OPTIONAL FORMATS OF A GENERAL OUTCOME
MEASURE
Summary
Paul Mooney, Kevin S. McCarter, Robert J. Russo, Danielle L. Blackwood
Louisiana State University, USA

         
               
     Embedded Formative Assessment    






Object of the research –

            
              



th
  

          

    


93


             
  
  
   
  
              
     
            


      
                 






               
          

integrated Louisiana
  i        

     


       

               


       

THE STRUCTURE OF AN ONLINE ASSESSMENT OF SCIENCE AND SOCIAL STUDIES CONTENT:
TESTING OPTIONAL FORMATS OF A GENERAL OUTCOME MEASURE
Paul Mooney, Kevin S. McCarter, Robert J. Russo, Danielle L. Blackwood
... as demonstrating evidence of high validity and those below .5 as demonstrating low validity. Mooney et al. (2014) evaluated critical content monitoring's instructional utility by determining linear relationships between scores for critical content monitoring and state-level content tests after varying the time, length, and makeup of the probe. In the present study, we calculated Pearson point and 95% CI estimates for three single critical content monitoring probe forms and the abbreviated online Stanford Achievement Tests-10th edition (SAT-10) science and social studies test: (a) the traditional 20 questions including content split equally between science and social studies content; (b) a single subject probe with 30 questions, not the 40 questions evaluated in Mooney et al. (2014) and (c) a mixed-content and lengthier (30-question) format. ...
... Mooney et al. (2014) evaluated critical content monitoring's instructional utility by determining linear relationships between scores for critical content monitoring and state-level content tests after varying the time, length, and makeup of the probe. In the present study, we calculated Pearson point and 95% CI estimates for three single critical content monitoring probe forms and the abbreviated online Stanford Achievement Tests-10th edition (SAT-10) science and social studies test: (a) the traditional 20 questions including content split equally between science and social studies content; (b) a single subject probe with 30 questions, not the 40 questions evaluated in Mooney et al. (2014) and (c) a mixed-content and lengthier (30-question) format. We hypothesized that the mixed-content probe correlations would be similar to the moderate magnitudes reported in Mooney et al. (2014) and would provide support to the supposition that assessment of student science and social studies content achievement might reasonably be conducted at the same time in school-based systems. ...
... In the present study, we calculated Pearson point and 95% CI estimates for three single critical content monitoring probe forms and the abbreviated online Stanford Achievement Tests-10th edition (SAT-10) science and social studies test: (a) the traditional 20 questions including content split equally between science and social studies content; (b) a single subject probe with 30 questions, not the 40 questions evaluated in Mooney et al. (2014) and (c) a mixed-content and lengthier (30-question) format. We hypothesized that the mixed-content probe correlations would be similar to the moderate magnitudes reported in Mooney et al. (2014) and would provide support to the supposition that assessment of student science and social studies content achievement might reasonably be conducted at the same time in school-based systems. We also hypothesized that a reduction from 40 questions would result in stronger descriptive correlations. ...
Article
Full-text available
The amount of research evaluating the technical merits of general outcome measures of science and social studies achievement is growing. This study targeted criterion validity for critical content monitoring. Questions addressed the concurrent criterion validity of alternate presentation formats of critical content monitoring and the measure’s predictive validity. Participants were fifth-grade students (N = 51) who completed five different forms of critical content monitoring probes as well as oral reading fluency and maze probes over three benchmarking periods. Criterion measures were the science and social studies subtests of the online abbreviated Stanford Achievement Test–10th edition. Concurrent correlation magnitudes for critical content monitoring ranged from .47 to .60. Predictive correlations for fall and winter ranged from .23 to .64. In three of four cases, commonality analyses findings favored critical content monitoring over oral reading fluency and maze as benchmarking choices. Study limitations and benchmark assessment framework implications are discussed.
... CCM probes were originally developed to be administered with 20 questions and over 5 min, similar to the present design of vocabulary matching. However, Mooney, McCarter, Russo, and Blackwood (2014) takers initially read definitional stems and then choose from vocabulary term choices. Each probe's content breakdown endeavors to approximate that of the curriculum, with terms pulled randomly and proportionally by curricular unit (see sample probe in Table 1). ...
... Technical adequacy research has addressed both science and social studies content. In science, results indicated moderate correlations (r = .36-.55); in social studies, a stronger correlation (.67) was reported (Mooney, McCarter, Russo, & Blackwood, 2013;Mooney et al., 2014). Mooney and Lastrapes (2017) replicated concurrent validity findings in science to Grades 4 to 6 and found stronger correlations for a 30-question probe (r = .44-.67 with a state test and .53-.76 with a nationally standardized test). ...
Article
The purpose of the research was to replicate commonality analysis for two measures: critical content monitoring and sentence verification technique. Participants were 967 fourth-, fifth-, and sixth-grade students across seven public primary schools in a southeastern U.S. district. The predictor variables were administered as benchmarks 3 times in an academic year. Predictor scores were compared with science content test and reading comprehension scores from the Stanford Achievement Test–Tenth Edition abbreviated online form and a statewide accountability test. Commonality analysis results indicated that scores from both critical content monitoring and sentence verification technique added unique variance to explanatory models, replicating previous findings. In most cases, critical content monitoring scores provided the greatest percentage of unique and common variance to model explanations.
... The results indicated moderate correlations for the 20 probes (r=.36 -.55). Mooney, McCarter, Russo, Blackwood (2014) extended the criterion validity findings, demonstrating a moderately strong correlation (.67) between a critical content monitoring social studies probe and the statewide content test. The social studies correlation was descriptively larger than the science correlation and comparable in magnitude to previous vocabulary matching findings (rs .64 to .70;Espin, ...
... was descriptively higher than the science correlation and comparable to the .68 linear relation with a state accountability test reported in Mooney et al. (2014) and .66 in this study (see Table 2). A pattern of critical content monitoring relations that were descriptively higher in social studies than in science was maintained across national and state assessments. ...
Article
Full-text available
The present research expanded validity findings for a structured formative assessment measure of content learning that was administered online and known as critical content monitoring. The study also evaluated the potential for additional measures, including sentence verification technique and written retell, to explain variance in student achievement in science and social studies classrooms. Participants were fifth-grade students (N=51) enrolled in a public primary school in the southeastern U.S. Three predictor variables (i.e. critical content monitoring, sentence verification technique and written retell) were correlated with content test scores from the nationally representative standardized achievement test (i.e. Stanford Achievement Test-Tenth Edition abbreviated online form) and a statewide accountability test. Pearson correlations for critical content monitoring and the Stanford tests across science (r=.55) and social studies (r=.63) were moderately strong and similar in magnitude with other reported correlations for academic language measures in the literature. Correlations for critical content monitoring were descriptively larger than those between the standardized tests and sentence verification technique and written retell. Commonality analyses indicated that both critical content monitoring and sentence verification technique added unique variance to explanatory models. Limitations and implications were discussed
... Similar data were used across and , and these coefficients were also averaged across state and commercial measures for the same content area. We did not combine coefficients if separate content areas (e.g., Mooney et al., 2014) or formats (e.g., were used in the same study and included both coefficients in the analysis. Twelve studies reported alternate form reliability coefficients. ...
Article
Meta-analysis was used to examine curriculum-based measurement in the content areas of social studies and science. Nineteen studies between the years of 1998 and 2020 were reviewed to determine overall mean correlation for criterion validity and examine alternate-form reliability and slope coefficients. An overall mean correlation of .59 was found for criterion validity; however, there was significant heterogeneity across studies suggesting curriculum-based measure (CBM) format or content area may affect findings. Low to high alternative form reliability correlation coefficients were reported across CBM formats between .21 and .89. Studies investigating slopes included mostly vocabulary-matching formats and reported a range from .12 to .65 correct items per week with a mean of .34. Our findings suggest that additional research in the development of these measures in validity, reliability, and slope is warranted.
... Second, teachers access multiple alternate forms of the probe since they are developed ahead of time. Indication of CCM's empirical effectiveness includes statistically significant and moderate in magnitude correlations with standardized content achievement tests across science and social studies content and testing formats (Mooney, McCarter, Russo et al., 2013;Mooney, McCarter, Russo, & Blackwood, 2014). In addition, when CCM has been compared with other general outcome measures, its correlations with standardized criterion measures have been descriptively stronger across both predictive (i.e., fall and spring benchmarks) and concurrent (i.e., spring) criterion validity circumstances Mooney, Lastrapes, Marcotte, & Matthews, 2016). ...
Article
We conducted conceptual replications for an online-administered general outcome measure known as critical content monitoring in science. Totally, 547 Grades 4 to 6 students from a rural Louisiana school district participated. Research questions addressed criterion validity, diagnostic accuracy, growth, and social validity concerns. Criterion measures included science and reading/literacy subtests from a state accountability test and a nationally standardized test. Findings indicated moderate to strong concurrent and predictive criterion validity correlations for the overall scores, fair diagnostic accuracy statistics for fall and winter and moderate for spring benchmark scores, variable growth scores across grades, and favorable social validity comments from teachers. Study limitations were listed. Research and practice implications were discussed.
Book
This volume presents a framework that expands the traditional concept of a vocabulary test to cover a range of procedures for assessing the vocabulary knowledge of second language learners.
Article
In this article, we delineate essential commonalities and distinctions between two approaches to measurement for instructional decision making. Specific subskill mastery measurement is explained using a case study, and salient features of this predominant model are described. Then, a major contrasting approach, the general outcome measurement model, is explained; a curriculum-based measurement case study is provided to illustrate general outcome measurement; and the essential features of this alternative model are reviewed. Finally, we describe how general outcome measurement represents an innovative approach to assessment by bridging traditional and contemporary paradigms.
Article
The present article includes 2 studies that were designed to expand the literature on the technical features of vocabulary matching. The purpose of both studies was to extend validity research on the general outcome measure of content knowledge. In Study 1, direct comparisons of predictor-criterion correlations related to selected standardized and curriculum-based instruments were made with a diverse sample of 153 sixth-grade students from 3 middle schools. Among other findings, direct correlation comparisons indicated that the relation between vocabulary matching and the state accountability test was statistically significantly stronger than those between other curriculum-based measures. In Study 2, linear mixed modeling was used to compare growth rates for a sample of 224 sixth-graders from the same 3 schools. Participants were administered 25 weekly probes over the course of a school year. Findings demonstrated growth rates that were significantly different from a zero growth hypothesis and variable across learners and probe type. Research implications for vocabulary matching are discussed.
Article
The literature base on using formative assessment for instructional and intervention decisions is formidable, but the history of the practice of formative assessment is spotty. Even with the pressures of high-stakes accountability, its definition is fuzzy, its adoption is inconsistent, and the prognosis for future use is questionable. A historical and organizational perspective explores plausible explanations for that inconsistency. These possible explanations include the standard literature in the research-to-practice gap and must also include the current policy environment surrounding educational accountability. A number of organizational and historical/cultural hypotheses suggest potential limits of structured formative assessment. From several perspectives, the practical question that may shape the future of special education is the identity of individuals at the school level who are responsible for coordinating the collection of formative-assessment data, for analysis, and for responses to the data.
Article
The purpose of this study was to evaluate technical adequacy features of an online adaptation of vocabulary matching known as critical content monitoring. Validity and reliability studies were conducted with a sample of 106 students from one school in fifth-grade science content. Participants were administered 20 parallel forms of the general outcome measure over a 2-week period. Criterion-related validity correlations with a statewide accountability test ranged from .36 to .55 for the 20 parallel forms. A pooled estimate of a common correlation between the state test and the probes was found to be .45. While correlation differences were not found, statistically significant differences in probe mean scores were identified across the body of parallel forms. Student commentary regarding the online assessment process was largely positive. Alternate-form reliability correlations ranged from .21 to .73, with a median correlation of .56. Limitations and implications are addressed.
Article
This article presents procedures and results from a 2-year project developing science key vocabulary (KV) short tests suitable for progress monitoring Grade 5 science in Texas public schools using computer-generated, -administered, and -scored assessments. KV items included KV definitions and important usages in a multiple-choice cloze format. A KV and item database was constructed for all Grades 3, 4, and 5 science student outcomes by scrutinizing basal and supplemental instructional materials used in Texas schools. The resulting 620 KV terms and 1,800 test items were formatted for computer administration in computer-generated probes.This study examines results from equivalent KV-science probes, administered to students with special needs on a weekly basis, over 5 or 6 weeks. It also compares results from KV probes versus standard multiple-choice science items in simpler pre-post administration. Results encourage further development of KV-based probes for progress monitoring in the content area of science.
Article
This study examined the technical adequacy of vocabulary-matching curriculum-based measurement (CBM) to identify and monitor the progress of 148 middle school students in social studies. In addition, the effectiveness of a reading comprehension intervention, Collaborative Strategic Reading (Klingner, Vaughn, Dimino, Schumm, & Bryant, 2001), was examined across 8 weeks with 15 low-achieving middle school readers who were monitored with content-area reading passages. Results indicated vocabulary-matching CBMs are valid and reliable indicators of performance in social studies. Significant differences were found for the intervention group on performance on pre to post vocabulary-matching CBM scores. Difference-in-Differences regression models were used to examine differences between groups on pre-post measures. No significant differences were found between groups in weekly change in scores or trend in performance on vocabulary-matching CBM.