Engineering education and non-education research: a scientometric comparison of 7 countries

Abstract

Over the last decade there has been a growing interest in the global evolution of engineering education research (EER) as a field of inquiry and a variety of approaches have been adopted to study this process.
Studies mapping engineering education research in different parts of the globe have mostly been human-curated and thus limited to relatively small samples. Recent advances in computer data analysis permit machine-curated study of larger data sets and this paper adopts such an approach.
The study assembles scientometric data on EER publications in Australia and compares it
with that of 6 European countries: 4 Nordic countries (Denmark, Sweden, Finland, and
Norway) and 2 Southern European countries (Portugal and Spain). This is achieved by
identifying 651 authors that published in 13 leading EER journals in the period 2018-2019
and then analysing their entire research output throughout their careers in both educational and non-educational publications - 32934 publications in all.
There are notable differences in the career evolution and EER output across the 7 countries and these in turn influenced the h-index values of the researchers in our sample. For Australia, as in the cases of Finland, Norway and Spain, engineering academics published over three times more non-educational than educational. This in turn affected their h-index values. In addition, our data suggest that Australian educators, along with those in Portugal, Sweden and Spain, are typically 6 to 8.5 years on average into their publishing careers when they publish their first educational work whereas in the case of Denmark, Finland and Norway this tends to occur earlier in their careers.
Scientometric findings acquired through analysis of large bodies of data, as in this study, can have a valuable role in informing both institutional and national policy decisions regarding support for engineering education research and can also help individual engineering educators in planning their own research career.

Full text

Engineering education and non-education research: a
scientometric comparison of 7 countries
Andrew Valentinea; Bill Williamsb,c
University of Melbournea, TU Dublinb, CEG-IST, Instituto superior Técnico, Universidade de Lisboac
Corresponding Author Email: bwbillwilliamsbw@gmail.com
ABSTRACT
CONTEXT
Over the last decade there has been a growing interest in the global evolution of engineering
education research (EER) as a field of inquiry and a variety of approaches have been
adopted to study this process.
PURPOSE OR GOAL
Studies mapping engineering education research in different parts of the globe have mostly
been human-curated and thus limited to relatively small samples. Recent advances in
computer data analysis permit machine-curated study of larger data sets and this paper
adopts such an approach.
APPROACH OR METHODOLOGY/METHODS
The study assembles scientometric data on EER publications in Australia and compares it
with that of 6 European countries: 4 Nordic countries (Denmark, Sweden, Finland, and
Norway) and 2 Southern European countries (Portugal and Spain). This is achieved by
identifying 651 authors that published in 13 leading EER journals in the period 2018-2019
and then analysing their entire research output throughout their careers in both educational
and non-educational publications - 32934 publications in all.
OUTCOMES
There are notable differences in the career evolution and EER output across the 7 countries
and these in turn influenced the h-index values of the researchers in our sample. For
Australia, as in the cases of Finland, Norway and Spain, engineering academics published
over three times more non-educational than educational. This in turn affected their h-index
values. In addition, our data suggest that Australian educators, along with those in Portugal,
Sweden and Spain, are typically 6 to 8.5 years on average into their publishing careers when
they publish their first educational work whereas in the case of Denmark, Finland and
Norway this tends to occur earlier in their careers.
CONCLUSIONS
Scientometric findings acquired through analysis of large bodies of data, as in this study, can
have a valuable role in informing both institutional and national policy decisions regarding
support for engineering education research and can also help individual engineering
educators in planning their own research career.
KEYWORDS
Scientometrics, citation analysis, engineering education research, Australia

Proceedings of REES AAEE 2021 The University of Western Australia, Perth, Australia, Copyright © Andrew Valentine and Bill
Williams, 2021.
Introduction
Over the last decade interest in the evolution of engineering education research (EER) has
been growing and a variety of approaches have been adopted to study this process. Froyd
and Lohman (2014) used criteria for defining the field of science education research
(Fensham, 2004) to point out that while engineering education has been seen as an area of
interest for educators since the end of the 19th century, over the last two decades there have
been significant indicators of a transition to an interdisciplinary, more scholarly field of
scientific inquiry into engineering education. Borrego and Bernhard (2011) have compared
Northern and Central European approaches to EER with those of the U.S. using a framework
from the European didaktik tradition, which focuses on answering the w-questions of
education. Borrego and Olds (2011) employed an analysis of National Science Foundation
funded projects as a way of characterizing development in EER in the US while Williams and
Alias (2011) used a scientometric approach to track the evolution of EER in Malaysia.
Neto and Williams (2011) analysed historical studies of the European Journal of Engineering
Education (EJEE) to provide insights on the European context. Other studies looked at
specific European national contexts (Williams, Wankat and Neto, 2018; Edström et al. 2016;
van Hattum-Janssen, Williams and Oliveira, 2015; Nyamapfene and Williams, 2017).
Strobel and colleagues at Purdue University applied bibliometric analyse to gauge the
presence of interdisciplinarity in EER (2012) and the growth of loose networks within the
EER community (2011).
The present study examines data gathered by using a quantitative scientometric approach to
understand the characteristics of EE researchers who were affiliated with tertiary institutions
in Australia. A small set of data from the Australian context was earlier reported based on
analysis of three EER journals (Valentine, 2020) whereas the present study considers data
from 13 publications. This allows us to create a more granular profile of Australian EER
output. To put the data in context we compare the Australian figures with those previously
collected by the authors relating to EER publication patterns of researchers in two European
contexts: researchers based in two Southern European countries (Valentine and Williams,
2021a) and those in four Nordic ones (Valentine and Williams 2021b).
Methodology
Data Sources
Data were gathered from the Scopus API (http://api.elsevier.com and
http://www.scopus.com) during January-March 2021 using the pybliometrics Python library
(Rose and John, 2019). Data was gathered over several months due to limitations of the
Scopus API.
A comprehensive list of EER publications from each of the respective countries was required.
To create this list, thirteen research journals relevant to the field of engineering education
(EE) were consulted (Table 1). For each journal, the list of all authors who had published at
least one article between 2019-2020 (inclusive) was considered. The tertiary institutions of
each author were checked, and this was used to establish which countries the author was
affiliated with.
Comprehensive details for each author were then retrieved from Scopus. This included their
full publication history. For subsequent analysis, only articles, conference papers, reviews,
book, and book chapters were included. Other publication types such as editorials, letters,
erratum or notes were excluded. Key details of each publication were captured including
document title, source title (e.g. JEE), document publication year, document type (e.g.
article), author keywords, subject category, citation count (note that this can change over

Proceedings of REES AAEE 2021 The University of Western Australia, Perth, Australia, Copyright © Andrew Valentine and Bill
Williams, 2021.
time; this is a limitation of the study), and DOI. A total of 32934 publications until the end of
2020 were captured for the 651 authors.
Journal
Finland
Spain
Portugal
Denmark
Norway
Australia
Sweden
Advances in Engineering Education
0
0
3
0
0
0
0
Australasian Journal of Engineering Education
0
1
0
0
0
29
0
Education for Chemical Engineers
2
100
7
8
0
2
0
European Journal of Engineering Education
14
22
27
7
4
43
20
Global Journal of Engineering Education
1
2
0
0
0
5
0
IEEE Transactions on Education
1
19
2
2
0
12
2
International Journal of Electrical Engineering
Education
1
31
0
0
0
4
0
International Journal of Engineering Education
3
218
10
4
3
11
9
International Journal of Engineering Pedagogy
2
1
11
0
0
1
0
International Journal of Mechanical Engineering
Education
0
0
0
0
1
1
0
Journal of Engineering Education
0
0
0
4
0
0
0
Journal of Engineering Education
Transformations
0
0
0
0
0
2
0
Journal of Professional Issues in Engineering
Education and Practice (now Journal of Civil
Engineering Education)
0
8
0
0
1
8
0
Total (duplicates removed)
23
397
58
23
9
111
30
Table 1: Engineering education journals where authors from each country were sourced from
(note it was possible that authors may have published in multiple journals)
Data Analysis
Publications were subsequently classified as being either educationally focused or non-
educationally focused. The purpose of this was to build an understanding of how educational
and non-educational publications contribute to the research track record of each author.
Because this involved analysis of thousands of publication records, it was not feasible to do
this manually. A computer aided approach was therefore required to assist with automating
the process. Accordingly, an algorithm was created, using a combination of keyword search
and Scopus data fields.
An extensive manual scoping search involving several iterations (and testing) was
undertaken to identify suitable Scopus fields and keywords (this is similar to how a scoping
search is implemented for systematic literature reviews).
A publication was deemed to be educationally focused if:
1. any of the following Scopus fields:
'authkeywords', 'subject_areas', or 'publicationName'
included any of the following terms
○ 'education', 'student', 'teach', 'tutor', 'novice', 'MOOC', 'ASEE', 'SEFI'
OR
2. the Scopus ‘title’ field included the term ‘learn’
○ AND the term ‘learn’ appeared at once outside the term ‘machine learn’
The inclusion of criterion 2 was necessary because “learn” was identified as a term that was
absolutely essential for some papers to be correctly flagged as educational (i.e. there were
no other terms which may have worked). However, an issue arose where papers in “machine
learning” were then often flagged as educational when they were not (this is also why “learn”
was restricted to the ‘title’ field). To try and address this issue, it was required that ‘learn’

Proceedings of REES AAEE 2021 The University of Western Australia, Perth, Australia, Copyright © Andrew Valentine and Bill
Williams, 2021.
appeared at least once in the title outside the context of the term ‘machine learn’. This
increased the accuracy, but some machine learning publications were still incorrectly flagged
as being educationally focused.
To test the efficacy and accuracy of this algorithm (compared to human judgement), a
random subset of 1000 publications were manually coded by the authors as being either
educationally focused or non-educationally focused. This was then compared to the output of
the algorithm.
● 400 papers from the Portugal, Spain authors were checked
○ there was a 99.7% agreement between human judgement and the algorithm
● 300 papers from the Denmark, Finland, Norway, Sweden authors were checked
○ there was a 97.3% agreement between human judgement and the algorithm
● 300 papers from the Australian authors were checked
○ there was a 98.3% agreement between human judgement and the algorithm
There was an overall 98.6% agreement between authors and the algorithm (11 false
positives, and 4 false negatives). This was deemed to be reasonable accuracy for analysing
the larger dataset and making conclusions (with the acknowledged limitation that about 1.4%
of publications may be incorrectly flagged).
Following this, information for each of the 651 authors was then established, including:
● the number of years the author had been publishing, and when they published their
first educational paper;
● the distribution of the publications by document type including articles, conference
papers, book chapters, books, and reviews;
● the percentage of publications which were educationally focussed;
● the number of citations on educational and non-educational publications;
● the author’s overall h-index, and that of their educational publications, and non-
educational publications.
Results
Ratio of educational and non-educational publications per country
Country
Population
(million)
Educational
Publications
Non-educational
Publications
Total
Publications
Australia
26
1377
4924
6301
Denmark
6
318
663
981
Finland
6
334
1066
1400
Norway
5
98
307
405
Portugal
10
667
1690
2357
Spain
47
4479
15909
20388
Sweden
10
493
609
1102
Total
7766
25168
32934

Proceedings of REES AAEE 2021 The University of Western Australia, Perth, Australia, Copyright © Andrew Valentine and Bill
Williams, 2021.
Table 2: The number of publications which are educationally focused and non-educationally
focused, per country
Overall, authors from each country published more non-educational publications than
educational publications (Table 2). While Sweden published slightly more non-educational
publications compared to educational publications, some other countries had published over
3 times as many non-educational publications (Australia, Finland, Norway, Spain) as
educational publications.
Average author percentage of publications which are of each document type
Country
Number of
Authors
Type of Document
Article
Book
Chapter
(Book)
Conference
Paper
Review
Total
Australia
111
Educational
28.2%
0.1%
2.1%
15.8%
1.5%
47.7%
Non-educational
27.5%
0.2%
1.5%
21.3%
1.7%
52.3%
Denmark
23
Educational
27.9%
0.3%
5.7%
24.8%
0.9%
59.7%
Non-educational
24.0%
0.1%
6.2%
7.1%
2.9%
40.3%
Finland
23
Educational
32.9%
0.0%
0.8%
29.4%
0.1%
63.2%
Non-educational
14.9%
0.0%
2.4%
18.7%
0.8%
36.8%
Norway
9
Educational
21.9%
0.0%
4.6%
11.3%
0.8%
38.6%
Non-educational
22.8%
0.0%
2.9%
35.2%
0.5%
61.4%
Portugal
58
Educational
18.8%
0.0%
1.2%
19.8%
8.7%
48.4%
Non-educational
29.1%
0.1%
2.6%
18.5%
1.4%
51.6%
Spain
397
Educational
24.1%
0.0%
0.6%
9.3%
0.3%
34.3%
Non-educational
43.4%
0.0%
1.6%
18.9%
1.7%
65.7%
Sweden
30
Educational
29.2%
0.6%
1.4%
34.0%
0.6%
65.7%
Non-educational
18.5%
0.0%
0.7%
14.0%
1.1%
34.3%
Table 3: The mean percentage of authors’ publications which are educationally focused for
each document type, per country
Table 3 shows that authors from Denmark, Finland and Sweden publish on average more
educational papers at 59.7%, 63.2%, and 65.7% of their overall total, respectively.
Conversely, authors from Australia, Norway, Portugal and Spain publish less educational
papers at 47.7%, 38.6%, 48.4% and 34.3% of their overall total on average, respectively.
h-index
For each country, the h-index of non-educational publications is higher than the h-index of
educational publications (Figure 1). While the difference between mean values is relatively
large for Australia, Denmark, Norway, Portugal and Spain, it is closer for Finland and
Sweden. The differences between h-index of non-educational publications and h-index of
educational publications for each country was evaluated for statistical significance using the
paired samples t-test with IBM SPSS 26. It was found that there was a statistically significant
difference for Australia (t=-4.244, df=110, p<0.001), Norway (t=-2.468, df=8, p=0.039),
Portugal (t=-3.553, df=57, p<0.01), and Spain (t=-13.221, df=396, p<0.001), but not
Denmark, Finland, or Sweden. Considering all 651 authors, while the h-index of educational

Proceedings of REES AAEE 2021 The University of Western Australia, Perth, Australia, Copyright © Andrew Valentine and Bill
Williams, 2021.
publications was significantly correlated with the overall h-index (Pearson Correlation=0.196,
p<0.001), the h-index of non-educational publications was a lot more strongly correlated with
the overall h-index (Pearson Correlation=0.956, p<0.001).
Figure 1: Mean h-index for each author per country for (i) all publications, (ii) educational
publications, and (iii) non-educational publications
Evolution of Publication Careers
Years into Career
Until First
Educational
Publication
Australia
(N=111)
Denmark
(N=16)
Finland
(N=21)
Norway
(N=9)
Portugal
(N=58)
Spain
(N=397)
Sweden
(N=30)
Total
0
43
13
15
2
18
119
13
223
1-5
17
4
3
5
9
79
4
121
6-10
19
3
2
2
8
59
4
97
11-15
16
1
1
0
11
57
5
91
16-20
9
0
1
0
7
44
2
63
21+
7
2
1
0
5
39
2
56
Table 4: Mean number of years into a researcher’s career before an educational publication is
published (counting from the date of their first research publication) (N is number of authors)

Proceedings of REES AAEE 2021 The University of Western Australia, Perth, Australia, Copyright © Andrew Valentine and Bill
Williams, 2021.
Figure 1: Mean number of years into a researcher’s career before an educational publication is
published (counting from the date of their first research publication)
Table 4 and Figure 1 demonstrate that many authors begin their careers publishing
educational research, while others commence educational research at a later time during
their career. Figure 2 shows the mean of years which authors from each country take until
publishing their first educational publication, while Table 5 also shows the median number of
years. Median values of 0 may be attributed to the small sample sizes of these countries.
Number of years
until educational
publication
Australia
(N=111)
Denmark
(N=16)
Finland
(N=21)
Norway
(N=9)
Portugal
(N=58)
Spain
(N=397)
Sweden
(N=30)
Mean
6.44
4.26
3.43
3.56
8.26
8.17
7.60
Median
4.00
0.00
0.00
2.00
6.00
6.00
4.00
Table 5: Number of years into a researcher’s career before an educational publication is
published (N is number of authors)
Figure 2: Mean number of years into a researcher’s career before an educational publication is
published
Limitations
Although we believe these data provide a valuable snapshot that allows us to compare the
publishing patterns in these countries, as the number of authors is relatively small in some

Proceedings of REES AAEE 2021 The University of Western Australia, Perth, Australia, Copyright © Andrew Valentine and Bill
Williams, 2021.
cases this can reduce the generalisability of the findings. We note for example that whereas
Valentine and Williams, 2021b studied data from 12 EER journals, for analysis in this paper
we added a further journal (Education for Chemical Engineers) to provide a larger sample;
while this led to similar overall findings there are some minor differences between the results
here and those of the smaller sample.
Conclusions
With regard to the ratio of education focused and non-education focused publications (Table
3), there is considerable variation between the 7 countries: authors from Denmark, Finland
and Sweden on average publish more educational papers whereas those from Norway and
Spain publish significantly more non-educational. In the case of Australia and Portugal, there
is a small preponderance of non-educational publications. These data help us begin to
characterise the current research culture in each country viz a viz research publication by
engineering educators.
Taking population differences into account, output from Australian EER scholars is broadly
similar to that of the other 5 EU countries. Globally, Spain appears to be something of an
outlier, even taking into account the fact it has the largest population: it publishes a large
number of journal articles, almost exclusively in two technically focussed journals, while
conference publications from Spain are rather lower than those of the other countries. This is
probably due to a nationally defined career progression system there that strongly privileges
journal publications in both educational and non-educational fields (Valentine and Williams,
2021a).
The above publication patterns in turn affect the h-index of the 651 authors included in our
study. In addition, our data suggest that non-educational publications play more of a role in
determining the h-index than educational ones as they tend to acquire more citations. This
reflects a generalized phenomenon that was noted in the 1970s by citation analysis pioneer
Ernest Garfield – founder of the ISI system and credited with being the initiator of the journal
impact factor concept – when he observed that “citation potential can vary significantly from
one field to another.” (Garfield, 1979). In general engineering education articles tend to have
much lower citation rates than those in specialized engineering fields. This can be seen in
the impact factor of journals: for example, the most cited journal in the field of EER, Journal
of Engineering Education, has a 2020 impact factor of 3.146 while those of the three highest
ranked in the field of Mechanical Engineering are Nature Materials 43.84, Materials Science
and Engineering: R: Reports 36.21 and Advanced Materials 30.85.
The mean number of years until educational publication is in the range 6 to 8.5 years for
Australia, Portugal, Spain and Sweden while engineering educators in Denmark, Finland and
Norway on average begin earlier in their academic careers. This may be due to the
increasing number of PhD programs in engineering and STEM education provided in these
countries: the Engineering/STEM Education Graduate Programs online resource curated by
the University of Arizona lists 4 programs in Sweden and one in Denmark but none for the
other countries. These results merit further study.
To conclude, scientometric findings acquired through analysis of large bodies of data, as in
this study, can have a valuable role in informing institutional and national policy decisions
regarding support for engineering education research and can also help individual
engineering educators in planning their own research career.
References
Borrego M. and Bernhard J., (2011), “The Emergence of Engineering Education Research as
an Internationally Connected Field of Inquiry.” Journal of Engineering Education 100: 14 –
47.

Proceedings of REES AAEE 2021 The University of Western Australia, Perth, Australia, Copyright © Andrew Valentine and Bill
Williams, 2021.
Borrego M. and Olds B. (2011). Analysis of Trends in United States National Science
Foundation Funding for Engineering Education: 1990 – 2010. Research in Engineering
Education Symposium, 175 – 183.
Edström K., Kolmos A., Malmi L., Bernhard J., and Andersson P., (2016). A bottom-up
strategy for establishment of EER in three Nordic countries–the role of networks. European
Journal of Engineering Education, 1-16.
Engineering/STEM Education Graduate Programs online resource,
http://engineeringeducationlist.pbworks.com/w/page/27610307/Engineering%20Education%2
0Departments%20and%20Programs%20%28Graduate%29 consulted 25 Sep. 2021
Fensham P. J. (2004) Defining and identity: The evolution of science education as a field of
research, New York, NY, Springer
Froyd J. E & Lohmann J. R., (2014). Chronological and ontological development of
engineering education as a field of scientific inquiry. Cambridge Handbook of Engineering
Education Research, 3- 26. http://dx.doi.org/10.1017/CBO9781139013451.003
Garfield, E. (1979). Is citation analysis a legitimate evaluation tool? Scientometrics 1(4), 359-
375.
Neto P. and Williams B. (2017). The European Journal of Engineering Education as a venue
for engineering education research publication: a meta view. In 45th SEFI Conference,
Azores, Portugal Sep 2017.
Nyamapfene A. and Williams B. (2017). "Evolution of Engineering Education Research as a
Field of Inquiry in the UK", 7th Research in Engineering Education Symposium (REES 2017):
Research in Engineering Education, Bogota, Colombia.
Rose, M. E., and John R. K. (2019). Pybliometrics: Scriptable Bibliometrics Using a Python
Interface to Scopus. SoftwareX, 10: 100263
Strobel, J. M. J., Radcliffe, D. F., Rajan, P., Nawz, S., Luo, Y., & Choi, J. H. (2011). AC 2011-
2836: Loose Networks and The Community of Engineering Education Research: A Definition
by Bibliometric Standards - mapping the domain of engineering education research. ASEE
Annual Conference and Exposition. https://doi.org/10.18260/1-2-18302
Strobel, J. M., Radcliffe, D. F., Yu, J. H., Nawaz, S., Luo, Y., & Choi, J. H. (2012). Is the
Engineering Education community becoming more interdisciplinary? ASEE Annual
Conference and Exposition, Conference Proceedings. https://doi.org/10.18260/1-2-21621
van Hattum-Janssen N, Williams B., and Nunes de Oliveira J.M., (2015). Engineering
Education Research in Portugal, an Emerging Field. International journal of engineering
education, 31(2), 674-684.
Williams B. and Alias M. (2011). Strategic Pathways to Engineering Education Research:
case study of a top-down initiative. Proceedings of the Research in Engineering Education
Symposium (REES 2011).
Williams B., Wankat P. C, and. Neto P, (2018). Not so global: a bibliometric look at
engineering education research. European Journal of Engineering Education, 43(2), 190-
200.
Valentine, A. (2020). Do Australian engineering education researchers publish more
educational or non-educational research? A bibliometric analysis. In 31st AAEE Conference,
Sydney, Australia, Dec 2020.
Valentine, A., Williams, B., (2021a) Evolution of Engineering Education Research in Portugal
and Spain: a scientometric study. Proceedings of CISPEE, 2021, Lisbon, Portugal 21 -23
June 2021.

Proceedings of REES AAEE 2021 The University of Western Australia, Perth, Australia, Copyright © Andrew Valentine and Bill
Williams, 2021.
Valentine, A., Williams, B., (2021b) Engineering Education Research in the Nordic Countries:
Scientometric Insights into Publication and Career Patterns, Proceedings of the 49th Annual
Conference of the European Society for Engineering Education (SEFI), Berlin, 13 -16
September 2021.
Copyright statement
Copyright © 2021 Andrew Valentine and Bill Williams: The authors assign to the Research in Engineering Education Network
(REEN) and the Australasian Association for Engineering Education (AAEE) and educational non-profit institutions a non-
exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and
this copyright statement is reproduced. The authors also grant a non-exclusive licence to REEN and AAEE to publish this
document in full on the World Wide Web (prime sites and mirrors), on Memory Sticks, and in printed form within the REEN
AAEE 2021 proceedings. Any other usage is prohibited without the express permission of the authors.