Content uploaded by Andrew Valentine
Author content
All content in this area was uploaded by Andrew Valentine on Dec 06, 2022
Content may be subject to copyright.
Research profiles of Australian computing education
authors: A scientometric analysis
Andrew Valentine
Computing and Information Systems
The University of Melbourne
Melbourne, Australia
0000-0002-8640-4924
Eduardo Araujo Oliveira
Computing and Information Systems
The University of Melbourne
Melbourne, Australia
0000-0001-5063-8860
Bill Williams
CEGIST
Universidade de Lisboa
Lisbon, Portugal
0000-0003-1604-748X
Abstract—Computing education (CE) is a growing, but well-
established field of research. However, relatively little is known
about the research profiles of CE researchers: whether they
tend to publish more educational or non-educational papers and
when during their career they tend to commence CE research.
Using a scientometric approach and data from Scopus, 189 CE
authors from Australia were identified who had published in the
field between 2018 and 2021. Their research publication history
was then retrieved, and each publication was classified as
educational or non-educational using a computer aided
approach. It was found that CE researchers have diverse
research profiles; well established researchers tended to have
fewer educational papers, new researchers tend to have more
educational papers, and that it is becoming more common to
start a research career doing CE research. This has implications
for how the research field may be viewed by university
computing departments.
Keywords—Research profiles, computing education,
scientometric
I. INTRODUCTION
Computing education (CE) is a growing field of research
in Australia. Here, 'computing education' refers to formal
tertiary education activities covering computer science,
software engineering, information technology, and
information systems. In 1970 soon after the ACM Special
Interest Group Computer Science Education (SIGCSE) was
formed, there were only 7 members from Australia, while
between 2010-2019 there was an average annual membership
of 41.4 from Australia [1].
Globally, Australian CERs are relatively prolific
contributors to the field. Outside of the United States, 3 of the
top 10 institutions which publish the most computing
education research worldwide are from Australia [2]. Analysis
of over 4500 publications from three conferences (ICER,
ITiCSE, SIGCSE TS) between 1970–2017 identified 173
unique authors from Australia, ranked 4th globally [3]. In
2019 “Computing Education" was finally codified as a
recognised Field of Research by the Australian Government
[4], highlighting a milestone which further solidified the
legitimacy of this research field.
However, academics within computing and engineering
university departments can face challenges that their research
on teaching, learning, or education is often seen as having less
legitimacy (compared to research on technical topics) by
faculty leadership and academic colleagues [5, 6, 7, 8].
The impact is that CERs are more likely to face problems
when it comes to performance reviews or applications for
promotion [7], because this can often be heavily reliant on
publication records. It has long been known that authors in
educational fields tend to publish papers at a slower rate than
technical fields and that educational publications tend to
attract fewer citations [8], which means that CERs often
cannot match the publication output of non-CERs in their
department. Educational researchers often “felt that they had
to maintain some technical research even when they found it
“boring" [7], to improve their publication records. This has a
negative impact on the advancement of CE as a field of
research, which is of concern. Where the research
performance of CERs is evaluated, it makes more senses to do
so comparative to other CERs (possibly outside the faculty).
A. Research questions and contribution to the literature
A challenge is that there is a lack of research which looks
at the overall publication histories of Australian CERs
(including the publications outside CE) to determine how
multi-disciplinary their research careers are. This is important
to understand, because it is not known how much of Australian
CERs output relates to education, and how much relates to
technical fields. Understanding this will help to build a general
profile (or profiles) of Australian CE authors.
• RQ1: How much of Australian CERs research output
relates to education, and how much relates to
technical fields?
• RQ2: What is the distribution of Australian CERs
research output by publication type?
• RQ3: When during their career to Australian CERs
do tend to commence educational research?
• RQ4: How are the h-indexes Australian CERs
influenced by their education and technical research?
II. METHODOLOGY
This study used a scientometric approach (focused on
bibliometric techniques [10, 11] to address the research
questions. This study focuses on specific indicators that are
widely used in bibliometric (and scientometric) research [10]
such as number of publications [11], citation analysis [11],
distribution of publications by type, and h-index [10]. The
process relied on identifying relevant authors who were
affiliated with an institution in Australia and had recently
published in a computing education related journal or
conference. An overview of the process is shown in Fig. 1.
A. Select Research Database
Information about authors and their publication histories
were gathered from the Scopus API using the pybliometrics
Python library [12] during May 2022. Scopus was used
because it was necessary to use a large multi-disciplinary
research database which contained comprehensive records of
each authors' publications (including those outside computing
education), that could be retrieved through an API.
Fig. 1. Process of how information was gathered and summarised
B. Identify Computing Education Publication Venues
Between 2018-2021
Authors who published in relevant journals and
conferences (where indexed) during 2018-2021 were
considered. A period of 4 years was used to ensure that most
Australian authors who had recently published computing
education research would be captured in the sample of authors
(e.g. who may be considered as research active in the field). A
scoping search was conducted to identify journals and
conferences where computing education-related papers were
a primary focus (e.g. not venues mainly focused on general
higher education or technology in education). The purpose of
this was to identify as many relevant authors as possible, while
focusing on only computing education specific venues. The
identified journals and conferences are described in the
following section.
C. Identify All Publications With at Least One Australian
Author
For each venue, the authors of each publication (several
thousand overall) between 2018-2021 (inclusive) were
considered. Some conferences were not indexed in Scopus
each year, which meant that only publications from a subset
of years could be considered. In this investigation, only
publications which had at least one Australian author were
recorded and included as part of future analysis.
The Conference and journal publication venues where
computing education researchers were sourced from between
2018-2021 included (Number authors; years not indexed in
Scopus): ACM ICER Conference (9), ACM ITICSE
Conference (53), ACM SIGCSE Conference (15), ACM
SIGITE Conference (0), ACM/IEEE ICSE: JSEET (3; 2018-
2020), ACM/IEEE ICSE: SEET (9; 2018,2021), Australian
Computing Education Conference (15; 2018-2020),
International Conference on Software Engineering Education
and Training (1; 2018-2019), Koli Calling Conference (8),
IEEE EDUNINE Conference (2), IEEE EDUCON
Conference (3), IEEE FIE Conference (6), IEEE TALE
Conference (31), Information Systems Education Conference
(1; 2018), Western Canadian Conference on Computing
Education (0),Computer Science Education Journal (5), ACM
Transactions on Computing Education (1). IEEE Transactions
on Education (1), Journal of Information Systems Education
(2), Journal of Information Technology Education:Research
(5), Journal of Information Technology
Education:Innovations in Practice (1), Journal of Information
Technology Education:Discussion Cases (0).
D. Exclude Engineering Papers From IEEE Venues
Publications in IEEE outlets were then considered
separately to the non-IEEE outlets, because it was necessary
to only include authors of IEEE papers about computing
education, and not engineering education. There were 139
publications from IEEE outlets with at least one Australian
author. A conservative viewpoint was adopted that all IEEE
publications were considered engineering publications by
default and were only selected as being computing
publications if the paper clearly focused on computing
education from the title, author keywords, and abstract. A
screening process identified characteristics of IEEE papers
identified that presence of certain keywords could accurately
determine if a paper was about computing or engineering
education. If an IEEE paper mentioned any of the following
keywords in the title, author keywords, or abstract, it was
deemed to be in computing education; ‘'computing education',
'programming', 'computer science', 'software engineering',
'information technology', 'ICT education', 'software
development', 'code', 'coding’. Papers without these keywords
were deemed to be about engineering education and were
discarded. 43 of the 139 publications were deemed to be about
computing education and would be included in further
analysis.
E. Retrieve List of All Australian Authors
In total, there were 171 publications which had at least one
Australian author, 43 from IEEE outlets, and 128 from all the
other outlets. Within the 171 publications there were 370
unique authors, of which 193 were from Australia.
F. Retrieve Full Publication History of Each Author From
Research Database
The entire research publications for each of the 193
authors were retrieved from Scopus giving a total of 10377
publications. For subsequent analysis, only journal articles,
conference papers, reviews (which includes many review
studies or systematic literature reviews), and book chapters
were considered. Other types of publications such as
editorials, letters, or notes were excluded. 490 publications
and 4 authors (who only published educational editorials in
one of the source journals) were excluded, meaning 9857
publications and 189 authors were included in the final
analysis.
G. Classify Papers as Educational or Non-educational
Focus
For further analysis it was necessary to classify each
publication as either having a focus related to education, or a
focus that was not related to education. Due to the large
number of publications, it was infeasible to do this completely
manually. Therefore, a computer aided approach was used.
This required creating an algorithm which was able to
categorise each paper, with high accuracy. This took place in
several stages.
First, an extensive manual scoping search was undertaken to
identity a suitable set of criteria which could be applied to the
publication details retrieved from Scopus, to classify each
publication. This took place in an iterative manner, with the
set of criteria being refined over several iterations. The set of
criteria was based on identifying the presence of certain
keywords within specific Scopus data fields.
A publication was deemed to be educationally-focused if:
Search strings
• Scopus fields: publication title, author keywords,
abstract, venue title
• Keywords for computing (OR operator):
computing, programming, development,
computer, software, information technology,
ICT, code, engineering, program
• Keywords for education (OR operator):
education, authentic learning, learning analytics,
learning outcomes, authentic teaching, teacher,
mentor, tutor, student, lms, pedagogy
• Keywords for venues (OR operator): ascilite,
tale, sefi, aaee, informatics in education,
computer science education, technology
education, computing education, computers and
education, higher education, educational
psychology
Data analysis
• Publication title (AND operator): Keywords for
computing AND keywords for education
• Author keywords (AND operator): Keywords
for computing AND keywords for education
• Abstract (AND operator): Keywords for
computing AND keywords for education
• Venue title: Keywords for venues
The set of strings from computing had to be combined with
the set of strings from education using the AND operator to
avoid the presence of a number of papers about machine
learning (or other similar terms) which introduced ambiguity
to the analysis. Our analysis was performed in R.
To evaluate the accuracy of the algorithm, a random sample
of 500 papers was extracted from the total of 9857 papers.
Then each paper in the sample was manually classified by the
first author as being either educational or non-educational
focused. The results of this were then compared to how the
algorithm classified each paper. Overall, there was a 96.4%
agreement rate, which was deemed as reasonable accuracy,
with the understanding that about 4% of papers may be
incorrectly classified.
H. Summarise Papers
Information about each author was then established
including:
• Ratio of educational publications to non-educational
publications
• Distribution of publications by type including
articles, conference papers, book chapters, and
reviews
• Number of years the author had been publishing, and
how long they had been publishing educational
papers
• Distribution of the publications by authors
• Number of years before first educational publication
• Author’s overall h-index, and that of their
educational publications, and non-educational
publications
III. RESULTS
A. RQ1: How Much of Australian CERs Research Output
Relates to Education, and How Much Relates to
Technical Fields?
Overall, the 189 Australian authors had a total of 9857
publications. Considering the entire publication set (not
individual authors), there were 2139 educational publications
and 7718 non-educational publications. Across the entire set
of individual authors, 21.8% of the researchers’ publications
were educationally focused, and 78.2% were non-
educationally focused.
Looking into more details about authors' publications, Tables
I and II presents, respectively, the number of authors that
published non-educational and educational papers. Table I
shows 159 Australian authors published non-educational
papers. Among them, 59 published between 0-10 non-
educational papers, and 23 authors published over 91 papers.
In contrast, Table II shows 189 authors published educational
papers. 133 of them published between 0-10 educational
papers and only 2 authors published over 91 papers.
23 authors were responsible for publishing 4471 non-
educational papers while 2 authors alone published 282
educational papers. 15% (or 29) of Australian authors were
responsible for publishing 61.3% (or 1277) of the educational
papers (21-higher).
TABLE I. D
ISTRIBUTION
O
F
T
HE
N
UMBER
O
F
A
UTHORS
,
C
ATEGORIZED
B
ASED
O
N
H
OW
M
ANY
N
ON
-
E
DUCATIONAL
P
UBLICATIONS
T
HEY
H
AVE
Number Of Non-
Educational
Publications By An
Individual Author
Number Of
Authors With This
Many Non-
Educational Papers
(N=189)
Combined Number
Of Non-
Educational
Publications By
Authors (N=7718)
0 30 0
1-10
59 212
11-20 20 396
21-30 14 331
31-40 10 349
41-50 7 323
51-60 8 438
61-70 7 452
71-80 7 513
81-90 4 333
91+ 23 4471
TABLE II. D
ISTRIBUTION
O
F
T
HE
N
UMBER
O
F
A
UTHORS
,
C
ATEGORIZED
B
ASED
O
N
H
OW
M
ANY
E
DUCATIONAL
P
UBLICATIONS
T
HEY
H
AVE
Number Of
Educational
Publications By An
Individual Author
Number Of
Authors With This
Many Educational
Papers (N=189)
Combined Number
Of Educational
Publications By
Authors (N=2139)
0-10
133 463
11-20 27 399
21-30 12 296
31-40 7 251
41-50 3 135
51-60 3 161
61-70 1 63
71-80 0 0
81-90 1 89
91+ 2 282
A t-test was conducted to compare mean differences on
number of publications between educational and non-
educational publications. There was significant difference
between the groups (t=-9.3459, p-value < 2.2e-16). This result
show us both groups (publications of educational and non-
educational papers) have significance differences in the mean
values of that variable. Fig. 2 and Fig. 3 shows distribution for
non-educational publications (M=48.54.24, SD=78.6)
descriptively performed better than educational publications
(M = 11.32, SD = 19.7), which is not a surprising result for us.
Fig. 2. Distribution of non-educational publications per author
Fig. 3. Distribution of education publications per author
B. RQ2: What is the Distribution of Australian CERs
Research Output by Publication Type?
Table III presents the number publications which are
educationally focused, organised by type. As shown,
educational publications are most commonly published
conference proceedings, followed by articles. Non-
educational publications follow the same trend.
C. RQ3: When During Their Career to Australian CERs do
Tend to Commence Educational Research?
On average, researchers took 4.78 years before they
published their first educational paper. However, this number
was influenced by some researchers with long careers who
only published an educational paper late into their career.
Table IV showed that a very large number of researchers
publish educational research right at the start of their career.
D. RQ4: How are the h-indexes Australian CERs
Influenced by Their Education and Technical Research?
Table V shows the overall h-index of authors is primarily
driven by non-educational publications, compared to
educational publications. While the correlation between h-
index of educational publications and overall h-index is
relatively high (Pearson correlation coefficient=0.60), the
correlation between h-index of non-educational publications
and overall h-index is notably higher (Pearson correlation
coefficient=0.94).
TABLE III. DISTRIBUTION OF ALL THE EDUCATIONAL AND NON-
EDUCATIONAL PUBLICATIONS, CATEGORZED BY PUBLICATION TYPE
Publication
Type Number Of Non-
Educational
Publications (N=7718)
Number Of
Educational
Publications
(N=2139)
Total (Mean of authors) Total (Mean of authors)
Article 2896 (15.3) 701 (3.7)
Book Chapter 230 (1.2) 49 (0.3)
Conference
Proceeding 4432 (22.4) 1345 (7.1)
Review 160 (0.8) 44 (0.2)
TABLE IV. NUMBER OF AUTHORS, CATEGORISED BASED ON TIME
INTO CAREER UNTIL FIRST EDUCATIONAL PUBLICATION
Number Of Years Into
Research Career Until First
Educational Publication
Number Of
Authors (N=189)
0 years 83
1-5 years 42
6-10 years 34
11-15 years 15
16-20 years 8
21+ years 7
TABLE V. COMPARISON OF H-INDEX OF PUBLICATIONS
Statistic H-Index Of All
Publications
H-Index Of
Educational
Publications
H-Index Of Non-
Educational
Publications
Min. 1 1 1
1st Qu. 4 2 2
Median 8 3 5
Mean 10.5 4.74 7.45
3rd Qu. 14 6 10
Max. 49 32 49
IV. DISCUSSION
The overall number of Australian authors' publications of
non-educational papers were significantly higher than the
number of publications of papers in computing education. As
much as the number of publications in computing education
has been increasing since 2010, our findings suggest a small
number of authors have been responsible for most of the
publications in this field in Australia (Table II). This effect can
be explained by the fact many authors have an already well-
established research track records within their fields, making
it hard to transition to computing educational area. While
many CERs are now starting their career doing educational
research (although there are no CE specific PhD programs in
Australia), many CERs commenced educational research later
during their research careers. Further investigations could
look at publications from Australian early career researchers.
A larger number of publications in computing education was
also found at conference papers. This finding is consistent
with Table III, which shows a larger number of conferences is
available in the field. Although not surprising, similar results
were also found for non-educational publications. Our results
are consistent with analyses and findings from [14] which
discusses that computer science, as a discipline, often values
conferences as a publication venue more highly than any other
academic field of study.
Comparing the results of this study to that of [14] who
investigated the research profiles of engineering education
researchers (EERs), we can see that while CERs tend to
publish more in conferences, EERs tend to publish in journals
more than conferences. It is also evident that the h-index of
both CERs and EERs are both primarily driven by their non-
educational publications.
We also identified in our results that non-educational papers
are cited at a much higher rate than educational papers. This
reflects a generalized phenomenon that was noted in the 1970s
by citation analysis pioneer Ernest Garfield – founder of the
ISI system and credited with being the initiator of the journal
impact factor concept – when he observed that "citation
potential can vary significantly from one field to another" [9].
In general, computer education articles tend to have lower
citation rates than those in specialized computing fields. This
can be seen in the impact factor (IF) of journals: for example,
whilst IEEE Transactions on Education has a 2020 impact
factor of 2.1 and ACM Transactions on Computing Education
has an IF of 1.5, those of the three highest ranked in the field
of computer science are IEEE Transactions On Pattern
Analysis And Machine Intelligence 16.4, Nature Machine
Intelligence 15.5 and Information Fusion 13.0.
A. Implications and Suggestions for Academics
The findings of this research have several implications for
individual researchers and educational institutions.
We encourage individual researchers involved in teaching
computing subjects to explore more and new publication
venues in computing education as a way to evaluate their own
teaching initiative. In the long term, those evaluations could
turn into better content delivery, promotion of active learning,
authentic teaching and learning and sense of belonging among
students, for example. This could be particularly relevant to
early career researchers who just completed PhD and are now
initiating a career in teaching. There's a clear opportunity for
them to publish more in the field, which will contribute to their
scientific career but also to learn and develop new teaching
skills. In addition, the relation between computing education
research and teaching may be used as a starting point for
improving learning experiences in educational institutions
who's attention have been mostly focused in research.
We also suggest to university departments that where
necessary, that the research performance of computing
education researchers is compared relative to others in the
field with a similar number of publications and ratio of
educational to non-educational publications (to control for
confounding factors), rather than comparing computing
education researchers to colleagues in their department who
do technical research.
B. Limitations
We identify two main limitations in our study. First, the
study is limited to the consideration of Australian computing
education researchers. It is possible that the findings presented
here may not be easily generalised to other countries. Previous
work similar to this but in the field of engineering education
demonstrated that the characteristics of authors’ research
changed considerably depending on the country being
considered [14], and it is possible this is also true for
computing education researchers. Second, the method of
sourcing Australian CER also has limitations. While the
primary computing education publication venues were
included, it is possible that some CER may not have published
in these venues. Conference proceedings of some years were
not indexed in Scopus; this is particularly notable for the fact
that the Australian Computing Education conference
(arguably one of the most important to consider in this study)
was not indexed in 2018-2020, meaning some authors may be
absent. However, the range of venues considered and sample
of authors was reasonable to build a reliable profile of
Australian CERs. The data used in this study was gathered
from Scopus; if another data source was used the indexed
papers may vary slightly, meaning the findings may differ
slightly.
V. CONCLUSION
This paper presents an investigation into the research
profiles of Australian computing education authors. Results
show that Australian authors have been mainly and
consistently publishing in non-educational venues in the past
few years. Twenty-nine Australian authors were responsible
for nearly 60% of all publications in computing education in
the last decade, suggesting this may be a good field to pursue
as a scientific career. Our findings have implications for
individual researchers (especially those involved in teaching
activities) and educational institutions as there is a clear
opportunity to perform more research in computing education,
which will contribute to individuals' scientific career and
impact teaching and learning experiences.
REFERENCES
[1] B. A. Becker, A. Settle, A. Luxton-Reilly, B. B. Morrison, and C.
Laxer, “Expanding opportunities: Assessing and addressing
geographic diversity at the sigcse technical symposium,” in
Proceedings of the 52nd ACM Technical Symposium on Computer
Science Education, 2021, pp. 281–287
[2] S. Lunn, M. Marques Samary, and A. Peterfreund, “Where is computer
science education research hap-pening?” in Proceedings of the 52nd
ACM Technical Symposium on Computer Science Education, 2021,
pp. 288–294.
[3] J. Zhang, A. Luxton-Reilly, P. Denny, and J. Whalley, “Scientific
collaboration network analysis for computing education conferences,”
in Proceedings of the 26th ACM Conference on Innovation and
Technology in Computer Science Education V. 1, 2021, pp. 582–588.
[4] A. B. of Statistics. Australian and new zealand standard research
classification (anzsrc), 2020 —australian bureau of statistics. [Online].
Available: https://www.abs.gov.au/statistics/classifications/australian-
and-new-zealand-standard-research-classification-anzsrc/latest-
release
[5] R. Adams, J. Siddiqui, and B. Williams, “Swelling the ranks—three
studies of scholars’ pathways to engineering education research,” in
Proceedings of the Research in Engineering Education Symposium
(REES 2013), 2013.
[6] S. Dart, K. Blackmore, K. Willey, A. Gardner, S. Jose, R. Sharma, T.
Sloan, and L. Jolly, “Moving from crime and punishment to success
and reward: Transitioning from technical to educational research,” in
Proceedings of the 8th Research in Engineering Education Symposium,
REES 2019: Volume 1. South African Society for Engineering
Education (SASEE), 2019, pp. 329–338.
[7] L. Kavanagh, L. O’Moore, and L. Jolly, “Mad as hell and not taking it
any more?: job satisfaction amongst engineering educators in australian
universities,” in Proceedings of the 23rd Annual Conference of the
Australasian Association of Engineering Education (AAEE2012).
2012, pp. 600–608.
[8] .C. Cretchley, S. L. Edwards, P. O’Shea, J. Sheard, J. Hurst, and W.
Brookes, “Research and/or learning and teaching: a study of australian
professors’ priorities, beliefs and behaviours,” Higher Education
Research & Development, vol. 33, no. 4, pp. 649–669, 2014.
[9] E. Garfield, “Is citation analysis a legitimate evaluation tool?”
Scientometrics, vol. 1, no. 4, pp. 359–375, 1979.
[10] N. Donthu, S. Kumar, D. Mukherjee, N. Pandey, and W. M. Lim, “How
to conduct a bibliometric analysis: An overview and guidelines,”
Journal of Business Research, vol. 133, pp. 285–296, 2021.
[11] S. Haustein and V. Larivi`ere, “The use of bibliometrics for assessing
research: Possibilities, limitations and adverse effects,” in Incentives
and performance. Springer, 2015, pp. 121–139.
[12] M. E. Rose and J. R. Kitchin, “pybliometrics: Scriptable bibliometrics
using a python interface to scopus,” SoftwareX, vol. 10, p. 100263,
2019.
[13] Vrettas and M. Sanderson, “Conferences versus journals in computer
science,” Journal of the Association for Information Science and
Technology, vol. 66, no. 12, pp.2674–2684, 2015.
[14] A. Valentine and B. Williams, “Engineering education and non-
education research: a scientometric comparison of 7 countries,” in
Proceedings of the REES AAEE 2021 Conference. AAEE, 2021, pp.
1–9.
