The Dog ATE my Economics Homework! Estimates of the Average Effect of Treating Hawaii’s Public High School Students with Economics
ABSTRACT Hawaii is one of 27 states that do not require testing of public high school students regarding their understanding of economics. We report results for the first economics test administered to a large sample of students in Hawaii public high schools during the Spring 2004 semester. Our analysis focuses on evaluating the impact of a semester-long course in economics on student scores on a 20-question, multiple-choice economics test. We specify and estimate a regression analysis of exam scores that controls for other factors that could influence student performance on the exam. While student scores on the economics exam are relatively low, completion of an economics course and participation in a stock market simulation game each add about one point to student scores.
The Dog ATE my Economics Homework!
Estimates of the Average Effect of Treating
Hawaii’s Public High School Students with Economics*
University of Hawaii Economic Research Organization
University of Hawaii-Manoa
Sumner La Croix
Department of Economics
University of Hawaii-Manoa
Working Paper No. 10-1
January 22, 2010
Hawaii is one of 27 states that do not require testing of public high school students
regarding their understanding of economics. We report results for the first economics test
administered to a large sample of students in Hawaii public high schools during the
Spring 2004 semester. Our analysis focuses on evaluating the impact of a semester-long
course in economics on student scores on a 20-question, multiple-choice economics test.
We specify and estimate a regression analysis of exam scores that controls for other
factors that could influence student performance on the exam. While student scores on
the economics exam are relatively low, completion of an economics course and
participation in a stock market simulation game each add about one point to student
Keywords: economic education, high school economics, stock market simulation
JEL Codes: A20, A21, I21
* We thank Sang-Hyop Lee, Tim Halliday, Michael Kimmit, and participants in a seminar at the University of Hawaii for useful
comments. We thank Kristine Castagnero, Director of the Hawaii Council for Economic Education, for her early comments on this
research and support of this project. Gail Tamaribuchi, former Director of the University of Hawaii Center for Economic Education,
was instrumental in identifying high school teachers who were willing to administer the exam to their classes. We thank the public
high school teachers and students who took part in this project and the State of Hawaii Department of Education for allowing us to
administer a test to students in public high school classes. Finally, the views expressed in this paper are solely those of the authors
and do not necessarily reflect those of the University of Hawaii Dept. of Economics, the University of Hawaii Economic Research
Organization, or the Hawaii State Department of Education.
Over the last 50 years, U.S. public and private high schools have increasingly
incorporated economics courses into their curriculums. For the 2004-2005 academic
year, the High School Transcript Study (2007) reported that 45.8 percent of high school
graduates had completed an economics course.1 Standardized tests of nationwide
samples of high school seniors have been conducted since the early 1970s to measure
understanding of key economic concepts and ability to apply them to particular problems.
While average student scores have been relatively low, completion of an economics
course has substantially raised student performance (Walsted and Rebeck 2000, 2001).
In 2006, the National Center of Education Statistics conducted its first assessment of
economics knowledge for the National Assessment of Educational Progress (NAEP) by
testing 11,500 twelfth-grade students from 590 public and private high schools. Walstad
and Buckles (2008) report cross tabulations from the 2006 NAEP assessment that suggest
important demographic, socioeconomic, instructional, and aptitude-related determinants
of exam performance.2
Twenty-three states require high school students to be tested on their knowledge
of economics.3 There appears to be little systematic reporting of results from the states
requiring testing, while statewide tests in economics are almost never conducted in the
states without required testing. Hawaii is one of the states in which economics is well
established in the public high school curriculum and which has never administered a
standardized economics exam to its public high schools students or to students who
complete an economics class. The lack of testing is somewhat surprising, as over 90
percent of Hawaii public high schools typically offer a full-semester economics course
and 27 percent of high school seniors completed a semester course in economics in both
the 2004-2005 and 2006-2007 academic years.4 Test results for students within a
particular state and at particular high schools could, however, help administrators and
policymakers to assess the overall understanding of economics by high school graduates
and evaluate the impact of an economics course on a student’s economic knowledge.
We report results for the first economics test administered to a large sample of
students in Hawaii public high schools during the Spring 2004 semester. Our analysis
1 The National Center for Education Statistics (2007) reported that 66 percent of high school seniors in
2006 had taken either a general economics course or an advanced placement economics course.
2 The NAEP does not release state-by-state breakdowns due to the small number of students taking the
exam in some states.
3 Grimes et al. (2008) suggest that the increases in economics exam scores are smaller for students taking a
mandated economics course than for students taking the course as an elective.
4 Hawaii public high school students are required to complete four credits (8 semesters) of social studies
courses. Since 3 credits are taken up by required social studies courses, students complete their one
elective social studies credit with two semester-length courses chosen from social studies electives offered
by their high schools, e.g. psychology, geography, economics, consumer education, or European history.
focuses on evaluating the impact of a semester-long course in economics on student
scores on a 20-question, multiple-choice economics test. We specify and estimate a
regression analysis of exam scores that controls for other factors that could influence
student performance on the exam. As we discuss below, the State of Hawaii placed
severe restrictions on the quantity and type of questions that we could ask high school
students regarding their personal and family characteristics. As a consequence, some
variables that affect student’s selection of an economics elective and their performance
on our test could not be included in the regression analysis. Despite these limitations, we
are able to draw some conclusions from our regression results regarding the impact of an
economics course on student performance on our economics test.
II. Exam Methodology
We analyze results from a 20-question multiple-choice test on basic economics
administered to over 500 students enrolled in 19 public high schools in Hawaii during the
Spring 2004 semester. The 19 high schools were drawn from Hawaii’s 38 public high
schools based on the willingness of an individual teacher at each high school to
administer the exam to the teacher’s class. The exam was administered at high schools in
each of Oahu’s four school districts and from the neighbor islands of Hawaii, Maui,
Molokai, and Kauai. The exams were conducted in a high-school class but not in an
economics or a consumer education class. The exam did not count towards a student’s
grade in the class. No compensation was paid to students either for completion of the
exam or performance on the exam. Each class had some students who had completed an
economics course and some who had not.5
The 20-question multiple-choice exam was designed by the National Council for
Economic Education, an umbrella organization of state council of education. The
questions cover such topics as exchange, supply and demand, price controls, inflation,
national income accounting, and international trade. Each question had three possible
We asked the Hawaii State Department of Education (“Hawaii DOE”) for
permission to ask students 20 questions regarding their personal and family backgrounds.
The Hawaii DOE did not allow us to ask any questions pertaining to their family (e.g.
number of siblings, lives with both parents, each parent’s occupation, high school GPA,
ethnicity, and whether English was their native language or spoken at home. In fact, the
Hawaii DOE allowed us to ask just four questions pertaining to their personal
characteristics: (1) gender; (2) plans to attend a two-year college; (3) plans to attend a
four-year college; and (4) plans to take more economics after high school. The Hawaii
DOE denied permission to ask students for their high school GPA, ethnicity, and whether
English was their native language.6
5 In one high school, we learned after the test had been conducted that all students who took the economics
exam had completed an economics class. We dropped this high school from our sample.
6 The Hawaii Dept. of Education routinely allows such information to be provided to national testing
We also asked students whether they had completed (1) a semester-length course
in economics; (2) a partial course in economics; (3) a consumer education course; and (4)
the Hawaii Stock Market Simulation. We asked specifically about these two courses and
classroom exercises because each may teach some concepts and provide some
information that could improve performance on an economics exam.7 After reviewing
student responses for partial courses in economics, we found that the only students who
responded that they had completed a partial course in economics were students who had
also completed a semester-length course in economics. We conclude that students were
confused by this question and have not included this variable in our regression analyses.
The Hawaii Stock Market Simulation (SMS), administered by the Hawaii Council
on Economic Education, is an interactive educational program that teaches high school
students about U.S. securities markets. Student teams invest a hypothetical $100,000 in
stocks, bonds, and mutual funds over a 10-week period. Team winners are publicized in
the Hawaii media.8 The intent of the program is to encourage students to think about real
life investment and financial issues such as setting investment goals, their tolerance to
risk, and the trade-offs that must be considered as they determine how to save for their
IV. Sample Properties and Summary Statistics
We examined all test forms and identified 25 with particularly low scores that
were likely due to a lack of effort on the exam. A large number of questions were left
blank on some exams; others exhibited a pattern of answers inconsistent with efforts to
honestly answer each question. Three high schools each had four students with
questionable exams. Only 20 percent of these students had taken an economics course,
and only 8 percent (2 students) had participated in a stock market simulation. These
students were also more likely to be male (56 percent) and were less likely to have plans
to attend a four-year college. We decided to drop these exams from our regression
analysis. After dropping these exams, we find that student scores on the economics test
are normally distributed with no discernable heaping at the tails.
7 Our review of consumer education courses in Hawaii shows them to be focused on such issues such as
consumer budgeting, checkbook balancing, credit management, consumer protection, and personal saving
8 Two 10-week sessions are offered each school year. Students participate in divisions based on their
investor profile: Income Growth, Growth, and Aggressive Growth. Each division requires that students
maintain a certain asset allocation of stocks and bonds. For example, a student participating in the Income
Growth Division would build a portfolio that contains 50 percent common stock and/or equity mutual funds
and 50 percent bonds and/or bond funds. At the end of the simulation, the Hawaii Council on Economic
Education reviews each portfolio (based on the division in which the team is participating) and compares
its performance to an investor profile benchmark (i.e. Growth & Income) representing the proper asset
allocation of the 10 year treasury bond and the S&P500.
We compute summary statistics for (1) the entire sample of students; (2) students
without a full economics course; and (3) students with a full economics course. Thirty-
seven percent of students had completed a semester-length economics; 24 percent had
participated in the Stock Market Simulation; and 19 percent had completed a semester-
length class in consumer education. The summary statistics show that the students who
selected an economics class were somewhat different than the students who did not.
They were more likely to be male (53 percent vs. 42 percent), more likely to plan to
attend a four-year college (62 percent vs. 51 percent), less likely to attend a two-year
college (18 percent v. 26 percent) and more likely to have participated in the Stock
Market Simulation (42 percent vs. 14 percent). The difference in characteristics between
the two groups of students clearly indicates that students are self-selecting into the class,
and that we cannot view an economics class as a randomly assigned treatment.9
IV. Econometric Methodology
Our goal in estimating regressions on student test scores is to isolate the effect of
an economics course and a stock market simulation on student understanding of
economics. Since students are clustered within specific high schools (“strata” in the
econometrics literature), our econometric analysis uses a specific form of this more
general strata regression:
SCOREis= α + β1ECON _COURSEis+ β2STOCK _ MKT_SIMis+γICis+κSCs+qs+eis
where SCOREis is the number of questions correct on the economics test by student i at
αs is a school-specific intercept, and ECON_COURSEis and
STOCK_MKT_SIMis are the two treatment variables,
control variables, SCs is a vector of strata-specific control variables, qs is an unobserved
stratum effect, and eis is the error term for student i at school s.
Wooldridge (p. 133) noted that “the presence of the unobservable qs induces
correlation in the composite error term
µis= qs+εiswithin each stratum.” Estimating
individual and stratum effects in one regression leads to unbiased estimators for β, γ, and
κ but consistency and asymptotic normality cannot be demonstrated. Moulton (1992)
demonstrated that within-group correlation leads to upward biased standard errors for
ˆ γ . However, since our analysis focuses on estimated coefficients for individual
rather than strata characteristics, adding a set of strata dummies to the regression provides
a simple solution to the error correlation problem.10 This yields a more specific form of
9 Peterson (1992) presented evidence that failing to account for the self-selection bias will understate the
potential gain attributable to a course in economics.
10 Once strata dummies are added to the regression, adding additional variables covering strata
characteristics to the regression do not improve the efficiency of estimates for variables covering student
ICisis a vector of student-specific
SCOREis= αs+ β1ECON _CLASSis+ β2STOCK _SIMis+γICis+eis
αs is a stratum-specific intercept.
The dependent variable in all regression specifications—the student’s score on the
multiple choice exam, SCOREis—is count data, with a minimum value of 0 and a
maximum of 20. Estimated coefficients from a regression with a count-data dependent
variable often have superior properties when a Poisson or a negative binomial estimator
is used rather than OLS (Wooldridge, ch. 19). Standard goodness of fit tests indicated
that the negative binomial model was more appropriate than the Poisson model. Thus,
we estimate all regression specifications using both OLS and negative binomial
The severe restrictions imposed by the Hawaii Department of Education on the
amount of personal data that could be gathered from each student has the potential to bias
estimated coefficients on both treatment variables due to the usual problems stemming
from omitted control variables. If data were available, we would have included high
school GPA, ethnicity, and whether English was their native language as control
variables in each regression specification. The small number of student characteristic
variables also limits our ability to control for selection problems with the two non-
randomly assigned treatment variables.
Each regression includes three controls for student characteristics—gender, plans
to attend a two-year college, and plans to attend a four-year college. Our basic regression
SCOREis= αs+ β1ECON _COURSEis+ β2STOCK _ MKT_SIMis+γ1GENDERis
Finally, the students who took an economics class or participated in a stock
market simulation were not randomly assigned to these treatments but rather self-selected
into them. If more information on student (and their family’s) characteristics were
available, it might be possible for to use instrumental variable or matching estimation
techniques to address this problem. In lieu of this information, we construct a second
data set that only includes groups of students with identical characteristics (gender, post-
secondary education plans, and high school) and with at least one student with a different
value for the ECON_COURSE treatment variable. All other observations are dropped.
By dropping non-comparable observations from the data set, there is the potential for
selection bias to be reduced. We estimate the following regression using OLS and
negative binomial estimators with the common support data set:
SCOREis= αg+ β1ECON _COURSEig+ β2STOCK_ MKT_SIMig+eig
where αg is a group-specific intercept. Results from regressions with the common
support data set need to be considered cautiously, as a matching analysis will only
perform well if selection is solely on observables (Imbens and Wooldridge 2009). This
condition is clearly not met in our analysis, as we have strong priors that other variables
(e.g. family income, student GPA) are likely to affect student selection of an economics
class or a class with a stock market simulation.
Our econometric analysis begins (Table 2) with OLS estimates of specification
(3) and considers whether results are affected by adding an additional treatment variable
(CONS_ED) and an additional control variable (PLAN_ ECON). Next, we run the same
regression specifications with a negative binomial estimator because the dependent
variable (SCOREis) is count data (Table 3). Finally, we repeat both the OLS and negative
binomial estimations using the matched data set described above (Table 4).
Our empirical analysis of student test scores begins with an examination of results
from four different regression specifications estimated using ordinary least squares. In
each of the four specifications (Table 2, columns 1-4), control variables performed as
expected. Estimated coefficients on GENDER range between .73 and .75 and are
statistically significant at the five percent level in all specifications. These results are not
surprising, as they mirror those found in earlier national studies of performance of high
students on economics tests, in particular the recent NAEP assessment in 2006.
Estimated coefficients on TWO_YR_COLLEGE are positive, ranging from .42 to .44,
but are not statistically significant at the ten percent level in all specifications. Estimated
coefficients on FOUR_YR_COLLEGE are also positive, ranging from 1.33 to 1.35, and
are statistically significant at the five percent level. Both variables are measured against
the baseline of the group of students who have no plans for further education beyond high
school. These results support the finding by Walstad and Buckles (2008) that students
with higher post-high school aspirations are likely more academically inclined than
otherwise, and therefore perform better on the exam.
All OLS specifications were run with dummy variables for each student’s high
school to control for stratum effects.11 None of the estimated coefficients are statistically
significant at the ten percent level. This is a somewhat surprising finding, as our priors
indicate that the quality of instruction varied substantially across high schools, a factor
that should have produced a positive impact on test scores at these schools.
Two more OLS specifications (Table 2, columns 2 and 4) were run using an
additional control variable, PLAN_ ECON. Estimated coefficients on PLAN_ ECON
range from .06 to .14, but are not statistically significant at the ten percent level.
Since the control variables generally performed as expected (or were statistically
insignificant), we turn our focus to estimates of the effects of the two treatment variables,
ECON_COURSE and STOCK_MKT_SIM. Estimated coefficients on ECON_COURSE
range from 1.02 to 1.09, and are statistically significant at the five percent level in all four
specifications. We note that the mean of the four conditional estimates of the effect of an
11 One school dummy was omitted to prevent perfect collinearity.
economics course on a student’s score (1.05) is less than the difference in unconditional
means (1.50). Estimated coefficients on the second treatment variable,
STOCK_MKT_SIM ranged from .96 to 1.04, and are statistically significant at the five
percent level in all four specifications. Together, the two treatments raised student test
scores by just over two points. These results support findings by Walstad and Buckles
(2008) that participation in a stock market simulation reinforces economic concepts and
therefore improves performance on the test.
Estimated coefficients on a third treatment administered to some students,
CONS_ED, are negative (Table 2, columns 3 and 4). This is somewhat surprising as one
might expect some of the knowledge acquired in a consumer education course to be
useful on a test of economics knowledge.12 The estimates are, however, not statistically
significant at the ten percent level.
Student test scores on the economics test are count data, and OLS estimates with
count data could be biased. We experimented with a Poisson regression model, but ran
into issues with overdispersion, i.e., greater variance than might be expected in this type
of distribution, resulting in failures of standard goodness-of-fit tests.13 We followed the
Poisson model with a negative binomial regression, as it is often more appropriate when
there is overdispersion. A likelihood ratio test conducted on the negative binomial
regression results reinforces our earlier finding that the assumption of a Poisson
distribution is inappropriate for our data set. Table 3 reports marginal effects of
treatment and control variables on student score for the same four specifications used in
the OLS estimates. Inspection quickly reveals that results from the negative binomial
regressions are virtually identical to those from the OLS regressions.
We also run OLS and negative binomial regressions on two specifications (with
and without CONS_ED) with our matched data set (Table 4). Estimated coefficients for
ECON_COURSE range from 1.01 to 1.08 and are statistically significant at the five
percent level. These results are just about the same as those obtained from earlier OLS
and negative binomial regressions using the full data set. Estimated coefficients for
STOCK_MKT_SIM range from 1.03 to 1.11 and are statistically significant at the five
percent level. These results are slightly (.08 to .10) higher than those obtained from
earlier OLS and negative binomial regressions using the full data set.
Our empirical results for the effects of the stock market game and completion of
an economics course are robust across two different estimation methodologies (OLS and
negative binomial regression), estimation with matched and unmatched data, and
12 Walstad and Buckles (2008) find a similar result regarding the negative relationship between
participating in Junior Achievement and test scores.
13 A significant (p<0.05) test statistic from the goodness-of-fit test indicates that the Poisson model is
inappropriate. The large value for chi-square in our goodness-of-fit test was another indicator that the
Poisson distribution was not an adequate functional form.
inclusion of a third treatment variable. In sum, they indicate that the Hawaii public high
school students in our sample who completed an economics course scored about one
point higher on the twenty-question exam and that students who participated in the stock
market simulation also scored about one point higher on the exam.
Participation in the stock market game produced almost as much improvement on
the student test as completion of an economics course. This is somewhat surprising, as
the amount of student and teacher time devoted to this activity is far less than the
semester-long economics course.
How do our results inform us with respect to the impact of a course in economics
on a Hawaii public high school student? On the one hand, scores of Hawaii public school
students on our economics test mirror nationwide results on standardized economics
tests: they are not particularly high. Students who completed an economics course still
averaged just 59 percent on the exam. On the other hand, Walstad (2001) suggests that
the best opportunity for improving the economic understanding of youth occurs in high
school, and our empirical results show non-trivial improvement, as the course in
economics added about five percent of the total points possible to student scores. While
the effect is not large, it nonetheless shows that foundations are present in the Hawaii
public high school curriculum to improve student understanding further. Finally, our
analysis emphasizes contemporaneous effects, but there could also be (unmeasured)
effects on future performance, as Myatt and Waddell (1990) found that completion of a
high school economics course enhances performance in university-level economics.
Burnett, Kimberly, and Sumner La Croix (2009). “Economic Education’s Roller Coaster
Ride in Hawaii, 1965-2006.” Dept. of Economics University of Hawaii Working
Paper No. 09-1.
Grimes, Paul, Meghan Millea and M. Kathleen Thomas (2008). “District Level Mandates
and High School Students’ Understanding of Economics.” Journal of Economic
and Economic Education Research, May, 1 2008.
Imbens, Guido W., and Jeffrey M. Wooldridge (2009). “Recent Developments in the
Econometrics of Program Evaluation,” Journal of Economic Literature, 47(1)
Moulton, Brent R. (1992). “An Illustration of a Pitfall in Estimating the Effects of
Aggregate Variables on Micro Units,” Review of Economics and Statistics, 72(2)
Myatt, Anthony and Charles Waddell (1990). “An Approach to Testing the Effectiveness
of the Teaching and Learning of Economics in High School.” Journal of
Economic Education 21(3) Summer: 355-363.
National Center for Education Statistics (2005). America’s High School Graduates:
Results from the 2005 NAEP High School Transcript Study. Washington, D.C.:
Dept. of Education. Available online at http://nces.ed.gov/pubsearch/
pubsinfo.asp?pubid=2007467. Last accessed on 17 November 2008.
National Center for Education Statistics (2001). The High School Transcript Study: A
Decade of Change in Curricula and Achievement, 1990-2000. Washington, D.C.:
Dept. of Education. Available online at http://nces.ed.gov/pubsearch/
pubsinfo.asp?pubid=2004455 Last accessed on 17 November 2008.
National Center for Education Statistics (1998). The 1998 High School Transcript Study
Tabulations: Comparative Data on Credits Earned and Demographics for1998,
1994, 1990, 1987, and 1982 High School Graduates. Washington, D.C.: Dept. of
Education. Available online at http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid
=2001498. Last accessed on 17 November 2008.
National Center for Education Statistics (2007). The Nation’s Report Card: Economics
2006. NCES 2007-475. Washington, D.C.: U.S. Dept. of Education.
National Council on Economic Education (2008). Survey of the States: Economic and
Personal Finance Education in our Nation’s Schools in 2007: A Report Card.
New York: National Council on Economic Education.
Peterson, Norris A. (1992). “The High School Economic Course and its Impact on
Economic Knowledge.” Journal of Economic Education 23 Winter: 5-14.
Snyder, Thomas D and Charlene Hoffman. (2003) Digest of Education Statistics 2002.
National Center for Education Statistics, Report number NCES 2006-060.
Walstad, William B. (1992). “Economics Instruction in High School,” Journal of
Economic Literature 30(4) December: 2019-2051.
Walstad, William B. (2001). “Economic Education in U.S. High Schools,” Journal of
Economic Perspectives 15(3) Summer: 195-210.
Walstad, William and Ken Rebeck. (2001). “Assessing the Economic Understanding of
US High School Students,” American Economic Review 91(2): 452-457.
Walstad, William B., and Ken Rebeck (2000). “The Status of Economics in the High
School Curriculum,” Journal of Economic Education 31(1) Winter: 95-101.
Walstad, William B., and Stephen Buckles (2008). “The National Assessment of
Educational Progress in Economics: Findings for General Economics,” American
Economic Review 98(2) May: 541-546.
Wooldridge, Jeffrey M. (2004). Econometric Analysis of Cross-Section and Panel Data.
Cambridge: MIT Press.
Table 1: Summary Statistics: Student Test Scores and Characteristics
A. All Students Taking the Exam
B. Students Taking Exam with Economics Course
PART_ ECON .53
PLAN_ ECON .22
C. Students Taking Exam without Economics Course
PART_ ECON .00
PLAN_ ECON .19
Mean S.D. S.D.
Table 2: OLS Regressions on Student Test Scores:
Full Sample with School Dummy Variables
(1) (2) (3) (4)
yes yes yes yes
Table 3: Negative Binomial Regressions for Student Test Scores:
Full Sample with School Dummy Variables
(1) (2) (3) (4)
yes yes yes yes
Table 4: OLS and Negative Binomial Regressions for Student Test Scores:
OLS OLS NB NB
Note: ** indicates statistical significance at the five percent level.