Hands-on Experiments vs. Computer-based Simulations in Energy Storage Laboratories

Abstract

pp121-128 This cross-over study compares student laboratory work conducted in two different learning modes: first as a practical hands-on exercise and second using computer-based simulations. The research methodology was optimized to avoid other effects on the learning outcome. To evaluate the influence of the mode, short tests on knowledge gained during the previous experiment were conducted at the beginning of the next laboratory session. In 2016, forty students have taken part. Overall learning results of hands-on experiments were slightly better than those of simulated laboratories, but the difference in performance was not statistically significant. The study is continuing in 2017 with 30 participants. In addition to the knowledge tests, after each laboratory session the students were asked for their opinion in an online survey. A similar percentage of the students stated the execution of the experiments is beneficial for their future professional life. In the hands-on learning mode more students expressed they have acquired new knowledge. Although more students assessed the simulated laboratories as more challenging compared to hands-on experiments, more students mentioned obstacles while conducting the hands-on equivalents.

Full text

45th SEFI Conference, 18-21 September 2017, Azores, Portugal
Hands-on Experiments vs. Computer-based Simulations
in Energy Storage Laboratories
F. Steger1
a,b
Research Associate/Lecturer
E-mail: fabian.steger@thi.de
H.-G. Schweiger a
Professor
E-mail: hans-georg.schweiger@thi.de
a TH Ingolstadt, Germany
A. Nitsche
a
Research Associate
E-mail: alexander.nitsche@thi.de
I. Belski b
Professor
E-mail: iouri.belski@rmit.edu.au
b RMIT Melbourne, Australia
ABSTRACT
This cross-over study compares student laboratory work conducted in two different
learning modes: first as a practical hands-on exercise and second using computer-
based simulations. The research methodology was optimized to avoid other effects on
the learning outcome. To evaluate the influence of the mode, short tests on knowledge
gained during the previous experiment were conducted at the beginning of the next
laboratory session. In 2016, forty students have taken part. Overall learning results of
hands-on experiments were slightly better than those of simulated laboratories, but the
difference in performance was not statistically significant. The study is continuing in
2017 with 30 participants. In addition to the knowledge tests, after each laboratory
session the students were asked for their opinion in an online survey. A similar
percentage of the students stated the execution of the experiments is beneficial for
their future professional life. In the hands-on learning mode more students expressed
they have acquired new knowledge. Although more students assessed the simulated
laboratories as more challenging compared to hands-on experiments, more students
mentioned obstacles while conducting the hands-on equivalents.
Conference Key Areas: Open and Online Engineering Education, Engineering
Education Research, Sustainability and Engineering Education
Keywords: Hands-on experiment, Simulated experiment, Battery experiment,
Learning-modes comparison
1 Corresponding Author: F. Steger, fabian.steger@thi.de

45th SEFI Conference, 18-21 September 2017, Azores, Portugal
INTRODUCTION
It has been shown that hands-on student laboratory work can significantly influence
the outcomes of student learning [1]. Nevertheless, many universities and vocational
training institutions conduct laboratories as simulated experiments instead [2]. This is
due to the costs of proper laboratory equipment to engage students in effective
learning. Another reason is the increased amount of supervision to conduct hands-on
labs safely, especially when the object of the experiment is potentially dangerous, such
as lithium-ion battery cells [3].
1 RATIONALE
Numerous studies have presented mixed opinions on whether hands-on laboratory
work is more conducive to learning than the simulated laboratory [4, 5, 6].
Reflecting on the results of such studies, Ma and Nickerson concluded that many
studies did not allow the researchers to reach definite conclusions [2].
Making clear conclusion on learning effectiveness of different modes of laboratory
exercises is hampered by design of the abovementioned studies. For example,
students conduct hands-on experiments in groups whilst on campus, but are engaged
in simulated exercises over the web and individually [6]. As the learning mode is mixed
with other influences (in this case supervision, cooperative learning effects, distance
learning, instructional papers, synchronicity, etc.), such research compares the
combination of all aspects and is unable to specifically identify the difference in learning
effectiveness of hands-on and simulated experiments.
The present study compares learning outcomes of student laboratory work conducted
in two different modes: first as a practical hands-on exercise and second using
computer-based simulations. In order to provide more reliable insights, this study
optimized research methodology to avoid other effects on the learning outcome during
laboratory work. The student experiments were strictly designed in such a way that the
content and procedure were identical in each mode.
2 APPROACH
Each group was taught the same four content areas related to lithium-ion battery cells:
two as practical hands-on experiments and two as computer-based simulations. The
first group completed the even laboratory sessions as hands-on experiments and the
odd ones as computer-based simulations. Planned as a cross-over study, the second
group of students were taught the same topics by means of the opposite learning
mode. To evaluate the influence of the mode on student learning, short tests on
knowledge gained during the previous experiment were conducted at the beginning of
the following laboratory session. The mean group results were compared, to answer
the question: which mode has been more successful? Additionally, the students were
asked for their opinions on each laboratory session in an online survey.
2.1 Learning objectives of the laboratory
Electrochemical cells are a wide and rapidly evolving field. It was therefore decided to
limit the new laboratory to teach the most relevant transferable skills and knowledge
that are beneficial for the students’ future careers. First, they should strengthen their
understanding of the characteristic behaviour of battery cells. Second, there is a focus
on gaining practical knowledge of the most important parameters of cells and how to
determine these parameters self-sufficiently by means of appropriate experimental

45th SEFI Conference, 18-21 September 2017, Azores, Portugal
setups. With these competencies students are enabled to design experiments with
energy storage systems.
The identified learning objectives where grouped to four main content areas: (A)
contact and isolation resistance, (B) open-circuit voltage, (C) internal resistance and
power, and (D) energy of cells. Grouped in these areas, seven laboratory experiments
were designed in such a way that they could be conducted in both modes in the same
manner:
• Low Resistance Measurements (A1): Students discover that a multimeter is an
inaccurate tool for low ohmic measurements, in the mΩ range, and why such
measurement is a misuse of the multimeter. They learn how to use alternative
procedures for low ohmic measurements, including a four-wire measurement in
AC and DC.
• Contact Resistance (A2): Students conduct experiments with a variety of
contact resistance values of typical electrical connections in battery systems.
• Isolation Resistance (A3): Students learn to estimate the influence of moisture
on the isolation resistance.
• Open Circuit Voltage Curve (B): Students investigate the dependency of the
open circuit voltage of a cell on the state of charge. They use two different types
of lithium-ion cells.
• Internal Resistance (C1): Students learn to use AC- and DC- methods to
measure internal resistances, being aware of the temperature dependency of
battery cells. Students learn to approximate temperature changes caused by a
power loss inside a cell.
• Power (C2): Students investigate the maximum discharge rate of battery cells.
Students discover the dependency of maximum discharge power from state of
charge, pulse duration, and temperature.
• Energy and Capacity (D): Students determine the capacity of a lithium-ion cell
and learn about the factors influencing it. They learn to calculate the energy
efficiency of both charge and discharge cycles.
Instructions affect the learning outcome of an experiment, e.g. the guidance level can
strongly influence student exploration [7]. Thus, for each experiment only a single set
of instructions was created and then used in both laboratory modes.
2.2 Creating two comparable groups for the cross-over study
Over the last two years, students enrolled in the energy storages course, were split
into two comparable groups based on their prior practical experience to ensure a
similar group environment for the individual students.
Forty students were enrolled in the laboratory subject in summer-semester 2016. Thirty
students were enrolled in summer-semester 2017. The full semester group in the study
program "Electrical Engineering and Electric Mobility" at Technische Hochshule
Ingolstadt (THI) is participating in the mandatory laboratory. All students were asked
to join the study.
To conduct this educational research as a cross-over study, it was essential to
separate the enrolled students into two comparable groups. In this type of study,
differences of the compared groups’ average performances are detected and
equalized by statistics. With the goal of isolating the learning mode in the experiment,
we have to consider the in-group interaction in laboratories. Webb found that the same
student may have different experiences in different groups, with consequent effects on
his or her learning [8].

45th SEFI Conference, 18-21 September 2017, Azores, Portugal
It was assumed that students with more practical experience may perform better in
laboratories than their peers with a lesser practical background. Therefore, in order to
assess the level of students' practical experience a preliminary questionnaire was
developed. Each year, after analysis of the student responses, students were assigned
to two laboratory groups to ensure a similar mix of students with practical experience
in each group. [9]
Each student created a code-word that could be used to identify the same individual,
while keeping all participants anonymous. Later two lists with code-words were
publicized to inform the students which group they were assigned for the laboratory
sessions.
2.3 Conducting laboratories in content areas A to D
Each group completed experiments in the four main content areas, two as practical
hands-on experiments and two as computer-based simulations. The first group
completed the even experiments as hands-on experiments and the odd ones as
computer-based simulations. Planned as cross-over study, the second group of
students were taught the same topics by means of the opposite learning mode.
For the content area A in simulation-mode, a newly created simulation-website was
used. For areas B to D a black box simulation of the hands-on equipment [10] and the
battery cell was used. To minimize influences from the user interface to control the
experiments, the simulation was accessed through the same graphical user interface
as the real hands-on devices. The simulation model emulates all observed effects of
the real battery cell and the hands-on devices. The cell simulation model was
parametrized to match the outcome of the hands-on experiments.
Each group was split into smaller learning groups of three to five students. The
laboratory groups and the learning groups remained unchanged to limit any effects on
the result caused by changing cooperative learning.
The students worked autonomously in a supervised environment. All groups were
asked to prepare a written laboratory report for each content area before the following
session.
2.4 Online survey after conducting the experiments
The students where asked for their thoughts on each laboratory session in a short and
fully anonymous online survey. This survey is conducted in both laboratories of the
department (chemistry and energy storages) to improve the laboratories. Adequate
questions were evaluated to compare the learning modes:
(1) “By conducting the experiment I gained new insights today.” (Original: “Ich habe
heute durch den Versuch neue Erkenntnisse gewonnen.“) This was a yes/no answer
and coded 1 or 0.
(2) “At which point in the experiment did you have the biggest problem proceeding with
the experiment?” (Original: „An welcher Stelle im Versuch hatten Sie am meisten
Probleme voranzukommen?“) This was a free text question which was not compulsory.
For data evaluation, the information was coded to 1 if any problem was mentioned or
to 0 if students wrote nothing or were expressing they had no problems.
(3) “The procedure of the experiment is quite difficult (1) / feasible (0.5) / easy (0)”
(Original: „Die Versuchsdurchführung ist recht schwer (1) / machbar (0.5) / leicht (0)“)
The answers were coded in a three step Likert scale.

45th SEFI Conference, 18-21 September 2017, Azores, Portugal
(4) “The content of the experiment is also relevant for me outside the university; I can
imagine that it will be beneficial for my future professional life.” (Original: „Der Inhalt
des Versuchs hat auch außerhalb der TH Relevanz für mich; ich kann mir vorstellen,
im Berufsleben Gewinn aus dieser Versuchsdurchführung zu ziehen.“) The answers
were coded in a five step Likert scale: fully agree (1) / somewhat agree (0.75) / maybe
(0.50) / somewhat disagree (0.25) / disagree (0)
2.5 Testing the learning outcome
To evaluate the influence of the mode on the student learning, written tests on
knowledge gained during the previous experiment were conducted at the beginning of
the next laboratory session.
These tests lasted ten minutes and contained a mix of descriptive and multiple choice
questions, free answers, and drawings. A positive point system (similar to tests for
giving a mark) was used to evaluate the results.
The tests were conducted anonymously. Students were coded through the same self-
created code-word used in the questionnaire for grouping.
A priority while planning both semesters was to keep time gaps between experiment
and the corresponding test equal for both groups. For organizational reasons, this was
not possible at the first content area A in 2016 [9]. In 2017 the laboratories are on the
same weekday morning and afternoon, making it easier to keep the experiment – test
time gaps equal for both groups.
For the tests, the goal was to always use the computer lab to provide the same
environment sitting at a desk while completing the tests. The tests were evaluated and
rated using a point system. The average group results between the hands-on and
simulated modes were compared, to answer the question: which mode was more
successful to transfer the knowledge of the experiment?
3 FINDINGS
3.1 Learning outcome (ten minute tests)
The second run of the experiment is conducted with 30 students in summer semester
2017. In year 2016 with 40 students the range of individual scores was from 12% to
85% for hands-on, and from 12% to 88% for the simulated mode. The distribution of
all results was normal. As seen in Table 1, for three content areas (A to C) a weak
effect towards benefits of the hands-on mode was measured. There was a slight trend
demonstrating hands-on laboratory sessions led to a better knowledge acquisition
compared to simulated experiments. Content area D showed no difference between
the modes. [9]
Overall learning results of hands-on experiments were slightly better than those of
simulated laboratories (weak effect, Cohen's d = 0.22), but the difference
in performance was not statistically significant (p=0.09>0.05).
3.2 Student feedback
The student feedback results are based on all results of the first iteration and the results
of content area A of the current ongoing laboratory in 2017.
(1) In hands-on mode more students expressed they gained new insights (81% vs.
55%, Cohen's d = 0.56, medium effect). Pearson’s correlation between mode and new
insights was 0.28 and significant (p=0.005).

45th SEFI Conference, 18-21 September 2017, Azores, Portugal
(2) A similarly large share of the students mentioned problems while conducting the
hands-on experiments or their simulation equivalents (42% hands-on vs. 43% in
simulation); Cohen's d = -0.02 is showing that there was no effect.
(3) The engagement in the simulated experiments was stated to be a small amount
(8% of scale) more difficult (0.38 hands-on vs. 0.46 simulation). Cohen's d = -0.33
demonstrates a weak effect.
Table 1. Results of 2016 [9], Effect hands-on vs. simulated
Content
Area Sample
Size
Percentage
of points
Mean Value
hands-on
Percentage
of points
Mean Value
simulated
Percentage of
points
Std.
Deviation
both modes
Effect size
(Cohen's d)
"hands-
on led to
better learning
outcome"
A
37
38%
33%
16%
0.28
B
39
54%
49%
16%
0.32
C
35
52%
47%
18%
0.30
D
37
45%
45%
15%
0.04
all
148
47%
44%
17%
0.22
(4) Students who conducted the experiments in the hands-on mode rated the execution
of the experiments a little more beneficial for their future professional life (0.58 hands-
on vs. 0.55 simulation, Cohen's d = 0.19, very weak effect).
No significant correlations between these items were found except between (1) and
(4), where Pearson’s correlation was 0.41 (p<0.001) and Spearman’s rho was 0.38.
Looking at both modes separately, the correlation in the simulation mode was stronger
(Pearson's r 0.52, p<0,001; Spearman's rho 0.52; N=49) compared to the hands-on
mode (Pearson's r 0.23, p=0,115; Spearman's rho 0.14; N=48).
Due to a low response rate for this questionnaire (31%) in the first year (2016), in the
second year (2017) the students were recompensed for their time to fill out a
questionnaire. Additional 5% of points were added to the grading of the laboratory
protocol, which makes it easier to reach the minimum level to pass the subject. In
experiment A, which had already been conducted at the writing of this publication, the
return rate increased to 77%.
4 CONCLUSIONS
4.1 Learning outcome
This study showed that the described methodology is applicable to focus on the
comparison of two learning modes. By excluding many other influences on the
comparison of the learning outcome, the results show only a small difference. The
slightly better learning results of the hands-on mode are not significant. Some of the
excluded factors might have greater impact on student learning than estimated
previously. To get statistically significant results more data collection is necessary. This
study is going on through 2018 at THI and the methodology will be tested at more
universities with different types of students (e.g. international students, summer
schools).

45th SEFI Conference, 18-21 September 2017, Azores, Portugal
4.2 Online survey
The contrast between both modes in the student’s subjective opinion about gained
knowledge (1) is much more significant than in the objective results of the ten minute
tests. The students’ opinions show advantages of the hands-on mode: although one
cannot determine the students understood more in this mode, one can conclude they
gain more confidence in performing similar tasks. In a limited capacity, self-confidence
can be considered beneficial for the students' further development by increasing their
intrinsic motivation.
The correlation between (1) and (4) shows that students who claimed that they gained
new insights also tend to believe that the execution of the experiment will help them in
their future professional life. The weaker correlation between (1) and (4) in the hands-
on mode suggests that even when the students think they gained additional insights,
the students do not consider all of the insights relevant for their profession. Identifying
these insights may be beneficial for the improvement of the experiments. Therefore,
future surveys will ask for the most beneficial insights gained and whether they
consider them useful for their future work outside the university.
Regardless of the learning mode, more than forty percent of the students think they do
not benefit in their professional life (4) from the experiments. This is quite disappointing
and leads to the possibility of asking for missing content students estimate as important
for their future profession.
On one hand less students mention problems with the hands-on experiments (2), while
on the other hand they feel the simulated variants to be more difficult (3). When taking
a deeper look at the answers from the students, it becomes clear that these results are
not correlated. A common mentioned problem was the lack of available measurement
devices. Also problems tied to the use of the software could be considered as neutral,
as the same software was used in both modes. The difference between the described
levels of difficulty of the learning modes (3) raises the question for these reasons. At
the time of writing we do not have a final conclusion on this weak effect. For the future
evaluation of the study we will ask the students to describe the difficulty in a free text
answer.
The actual methodology does not allow one to find correlations between individual
learning outcome and student’s feedback, as the standard online feedback form is not
asking for the self-created code-word that was used in the tests. We plan to implement
the new questions and this missing information to a new questionnaire in paper form
and use it for the next iteration.
5 ACKNOWLEDGMENTS
This research was approved by the Faculty of Electrical Engineering and Computer
Science of TH Ingolstadt and the College Human Ethics Advisory Network of RMIT,
Melbourne.
Many thanks go to the laboratory engineer Sönke Barra for his assistance while
planning, preparing, and conducting the experiments. The devices used in the
laboratory were built from finance of the Faculty of Electrical Engineering and
Computer Science at THI.
This research was not possible without students, generously ready to take part in the
study to improve the learning outcome of future groups.

45th SEFI Conference, 18-21 September 2017, Azores, Portugal
REFERENCES
[1]
Magin, D. J., Churches, A. E., and Reizes, J. A. (1986), Design and
experimentation in undergraduate mechanical engineering, Proc. of the
Conference on Teaching
Engineering Designers, Institution of Engineers,
Sydney, pp. 96-100.
[2]
Ma, J., & Nickerson J. V. (2006), Hands-On, Simulated, and Remote
Laboratories: A Comparative Literature Review, ACM Computing Surveys, Vol.
38, No 3, Article 7
[3]
Dahn, Jeff and Ehrlich, Grant M. (2011), Lithium-Ion Batteries, Linden's
Handbook of Batteries 4th Edition, McGraw-Hill, Town, pp. 26.1-26.79.
[4]
Engum, S. A., Jeffries, P., & Fisher, L. (2003), Intravenous catheter training
system: Computer-Based edu
cation versus traditional learning methods.
American J. Surgery 186, 1, pp. 67-74.
[5]
McAteer, E., Neil, D., Barr, N., Brown M., Draper, S., & Henderson, F. (1996),
Simulation software in a life sciences practical laboratory. Computers and
Education 26, 1-3, pp. 102-112.
[6]
Edward, N. S. (1996). Evaluation of computer based laboratory simulation.
Computers and Education 26, 1-3, pp. 123-130.
[7]
Chamberlain, J. M., Lancaster, K., Parson, R. & Perkins K.K., (2014), How
guidance affects student engagement with an interactive simulation. Chemistry
Education Research and Practice 15, p. 628.
[8]
Webb, N. M. (1989), Peer interaction and learning in small groups. International
Journal of Educational Research, Volume 13, Issue 1, p. 21-39
[9]
F. Steger, A. Nitsche, H.-G. Schweiger and I. Belski, Teaching Battery Basics
in Laboratories: Comparing Learning Outcomes of Hands-on Experiments and
Computer-based Simulations, Proc. of the 27th Austral
asian Association for
Engineering Education (AAEE) Annual Conference, Coffs Harbour, 2016.
[10]
F. Steger, A. Nitsche, H.-G. Schweiger and I. Belski, Teaching Energy Storages
by means of a Student Battery Cell Test System, Proc. of the 45th European
Society for Engineering Education (SEFI) Annual Conference, Angra do
Heroísmo, 2017.