Evaluating geovisualization for spatial learning analytics

Abstract

Contemporary systems for supporting digital learning are capable of collecting a wide range of data on learner behaviours. The emerging science and technology of learning analytics seeks to use this information to improve learning outcomes and support institutional assessment. In this work we explore the potential for the spatial dimension in learning analytics, and we evaluate a prototype geovisualization system designed to support what we call spatial learning analytics. A user evaluation with geographers and educators was conducted to characterize the usability and utility of our prototype spatial learning analytics system. By helping us understand what our prototype system does and does not do well, we are able to suggest a variety of new ways in which future spatial learning analytics systems can be developed.

Full text

Evaluating geovisualization for spatial learning analytics
Anthony C. Robinson
a
, Cary L. Anderson
b
and Sterling D. Quinn
c
a
GeoVISTA Center, Department of Geography, The Pennsylvania State University, University Park, PA, USA;
b
Joseph M. Katz Graduate School of Business, University of Pittsburgh, Pittsburgh, PA, USA;
c
Department of
Geography, Central Washington University, Ellensburg, WA, USA
ABSTRACT
Contemporary systems for supporting digital learning are capable of
collecting a wide range of data on learner behaviours. The emerging
science and technology of learning analytics seeks to use this
information to improve learning outcomes and support
institutional assessment. In this work we explore the potential for
the spatial dimension in learning analytics, and we evaluate a
prototype geovisualization system designed to support what we
call spatial learning analytics. A user evaluation with geographers
and educators was conducted to characterize the usability and
utility of our prototype spatial learning analytics system. By
helping us understand what our prototype system does and does
not do well, we are able to suggest a variety of new ways in
which future spatial learning analytics systems can be developed.
RÉSUMÉ
Les systèmes actuels utilisés pour faciliter l’apprentissage
numérique sont capables de collecter un très grand nombre de
données sur le comportement des apprenants. Les sciences et
techniques émergentes en analyse de l’apprentissage cherchent à
utiliser ces informations pour améliorer les résultats
d’apprentissage et pour aider l’évaluation institutionnelle. Dans ce
travail nous explorons le potentiel de la dimension spatiale dans
l’analyse de l’apprentissage et nous évaluons un prototype de
système de géovisualisation conçu pour faciliter ce que nous
appelons l’analyse spatiale d’apprentissage Une évaluation des
utilisateurs par des géographes et des éducateurs a été menée
pour caractériser l’utilisabilité et l’utilité de notre prototype de
système d’analyse spatial d’apprentissage. En nous aidant à
comprendre ce que notre système fait correctement et ce qu’il ne
fait pas correctement, nous sommes capable de proposer une
variété de nouvelles façons qui pourront être intégrées dans le
développement de futurs systèmes d’analyse spatial
d’apprentissage.
ARTICLE HISTORY
Received 5 September 2019
Accepted 23 February 2020
KEYWORDS
Geovisualization; learning
analytics; user evaluation
Introduction
Understanding the roles of geography in learner engagement can help educators develop
high quality courses and global learning experiences that account for geographic
© 2020 International Cartographic Association
CONTACT Anthony C. Robinson arobinson@psu.edu, acr181@psu.edu
Supplemental data for this article can be accessed https://doi.org/10.1080/23729333.2020.1735034
INTERNATIONAL JOURNAL OF CARTOGRAPHY
https://doi.org/10.1080/23729333.2020.1735034

differences and leverage them to support improved learning outcomes. Our work con-
nects closely to recent calls for the development of the spatially-enabled smart campus
(Janelle, Kuhn, Gould, & Lovegreen, 2014; Skupin, 2013), a vision for a flexible learning
environment that includes location technology and science at its core to support learners
working together as well as new forms of educational content that adapt automatically to
different scales and places of focus. In order to achieve the vision of a spatially-enabled
campus, we must first understand the situations in which geography matters in terms
of modifying modes of educational engagement and influencing improved in outcomes.
Our work to design and evaluate MapSieve, a geovisual analytics system designed to
support spatial learning analytics, represents progress toward addressing this gap in our
scientific knowledge.
Current learning analytics approaches provide insights on assessment performance
and student interactions, but do not shed light on the roles that geography may play.
Therefore, this project seeks to answer the question: How can geographic information
help us characterize, explain, and improve learning in online courses? We hypothesize
that leveraging geographic information in conjunction with traditional learner engage-
ment metrics in what we call spatial learning analytics systems can (1) help instructors
improve the quality of their teaching, (2) help students connect with each other and
their instructors, and (3) support the analytical goals of those who work on institutional
assessment.
In this article we introduce and evaluate a geovisualization system for supporting
spatial learning analytics. This system includes coordinated spatial and attribute visualiza-
tions that are coupled with a computational method for enhancing the utility of faceted
search in geographic visualization by pre-computing to uncover which combinations of
learner behaviour attributes (e.g. forum posts, grades on quizzes, etc …) (A) have results
at all to visualize, and (B) which of those combinations include significant spatial hotspots.
We implement this approach in a geovisual analytics system we call MapSieve which is
designed to support analysis of learner activity data from a Massive Open Online
Course (MOOC). Through a user evaluation we show how geovisualization can reveal inter-
esting combinations that analysts can use to understand learner behaviours and to make
judgments about pedagogical effectiveness. We conclude with ideas for further extension
of cartographic and spatial analysis approaches to learning analytics and highlight its
potential strengths and weaknesses vis-à-vis the new realities associated with big
spatial data, new sources of learner data, and supporting the development of spatially-
enabled campuses.
Background
Learning analytics
The emerging science of learning analytics has been theorized in many ways, but the most
commonly applied model is the five-step process proposed by Campbell, DeBlois, and
Oblinger (2007). Their model consists of five stages: capture, report, predict, act, and
refine. Subsequent developments in this domain have defined learning analytics as the
‘…measurement, collection, analysis, and reporting of data about learners and their con-
texts, for purposes of understanding and optimizing learning and the environments in
2A. C. ROBINSON ET AL.

which it occurs’(Siemens & Baker, 2012). Researchers in the computational and social
sciences have blended concepts from learning analytics into the co-emergent fields of
educational data mining (Romero & Ventura, 2007) and social learning analytics, respect-
ively (Shum & Ferguson, 2012).
Current platforms for learning analytics include a wide range of commercial systems
that are part of or can be plugged into a contemporary Learning Management System
(LMS). Such solutions can focus on a variety of analytical needs; for example, Hobson’s
Starfish is designed to support student advising and retention, while Civitas’Illume and
Inspire systems focus on institutional assessment and improving individual class experi-
ences respectively.
The development of MOOCs has also helped to inspire new forms of learning analytics
systems. Beginning in the early 2010s, the Massive Open Online Course model of distance
learning rose to popular prominence via MOOC platforms like Coursera, edX, and Udacity.
Major universities around the world participated in launching courses via these platforms,
and considerable media attention was placed on the potential for free online courses to
reach massive new audiences. In subsequent years, MOOC platforms have sought to
profit from their learner populations and the presence of features that support massive-
ness and openness has waned. Recent work by (Bennett & Kent, 2017) characterizes the
rise and evolution of MOOCs over time, and (Adams, 2018) explores and critiques the evol-
ution of Geography MOOCs more specifically.
Whether or not we are indeed in a post-MOOC or at least beyond the peak-MOOC era,
the challenges associated with effectively teaching students at a massive scale in MOOCs
has prompted the need for new analytical methods that can help instructors identify key
trends among their student cohorts when it is functionally impossible for every single
student to be individually evaluated. MOOC data include website logs of video and
content interactions, contributed assessment scores and assignments, as well as forum dis-
cussions. For example, Chen et al. (2015) have shown how a visual analytics approach can
be applied to identify and characterize peaks in MOOC student activity as indicated by
lecture video interaction logs. Their PeakVizor system grounds these peaks in activity by
linking them with flowlines to a world map which aggregates student activities to a
country level. Their efforts hint at what may be possible by better leveraging the geo-
graphic components of learner activity, showing that even the simple connection
between where students are located and what parts of lecture videos they watch has
the power to help instructors understand how different parts of a massive global cohort
are engaging with the course content. Related work by Wu, Yao, Duan, Fan, and Qu
(2016) to explore the social connections among MOOC students has also included a loca-
tional component (again at the country level) to highlight the development of study
groups and to characterize their interconnections. Much more remains to be known
about the potential utility of geographic information in learning analytics –these
examples along with the research progress we share in this article suggest that this com-
ponent deserves more attention than it has received to date.
The prevalence of learner activity data is increasing rapidly as universities and colleges
around the world adopt new digital learning management systems that can easily collect
and produce these data. According to the United States Department of Education (2014),
the proportion of students taking fully online courses is growing every year, currently com-
prising 13% of all enrolments in U.S. higher education, and more than 26% of students take
INTERNATIONAL JOURNAL OF CARTOGRAPHY 3

blended online/resident courses. Extrapolated to a global context, it is clear that millions of
learners are already working within learning management systems. In both online and
blended courses student location information can be more difficult for educators to
characterize and assess compared to what can be assumed and witnessed directly in an
on-campus course.
In this research we seek to align the goals of learning analytics with those of visual ana-
lytics, and specifically focus on the potential for geovisual analytics as a new lens through
which learning can be characterized and understood using spatially-referenced learner
activity data. Visual analytics is the science of analytical reasoning as facilitated by interac-
tive visual interfaces (Thomas & Cook, 2005), and geovisual analytics focuses on analytical
reasoning with visual interfaces for spatial and spatio-temporal phenomena (Andrienko
et al., 2007). We note several similarities between the five-step model of learning analytics
(Campbell et al., 2007) and a key theoretical framework for describing sensemaking in
visual analytics (Pirolli & Card, 2005). The sensemaking loop as proposed by Pirolli and
Card proposes an iterative process in which analysts collect information, schematize it,
hypothesize about its meaning, and present conclusions. The five-step model for learning
analytics does not include the stage of schematization, but this activity is implied by its
focus on reporting, prediction, and action.
Spatial learning analytics
Current learning management systems (LMS) can capture discussion forum activity and
assessment performance, but existing learning analytics methods do not explore where
learning is taking place. For example, we do not incorporate knowledge about the
broad spectrum of places in which students are engaging with the course (library,
coffee shop, airport, etc. …), or how those places of engagement may change over time
(through the course of a day, on weekends, etc …). In a fully-online class, this could
include places around the world, and in the context of a Massive Open Online Course,
the number and diversity of these locations is substantial. Space-time patterns are likely
to be extremely diverse, as learners engage with the course contents in different places
at different times. We do not know what types of locations are commonly associated
with learner engagement during the morning, versus the evening, or the impact of week-
ends and holidays. A key advantage of online learning is that students are empowered to
interact with each other and the course content at any time, in any place, but we know
surprisingly little about which times and which places students choose, and contemporary
LMS and associated learning analytics tools do not yet provide insights on spatial or spatio-
temporal patterns, and perhaps more importantly, we do not yet understand the potential
pedagogical roles for which spatial and spatio-temporal information may be helpful for
teaching and learning.
In terms of pedagogy, contemporary LMSs support a wide range of pedagogical
approaches, including behaviourist, cognitivist, and constructivist methods (Hodges &
Grant, 2015). The latter mode receives a great deal of attention in LMS design, as is evi-
denced by the proliferation of project-based, collaborative, active, and problem-based
tools. Support for behaviourist and cognitivist approaches is also well-served by contem-
porary LMS environments via support for various types of reinforcement and for explicitly
measuring outcomes. New research is needed to identify opportunities for leveraging
4A. C. ROBINSON ET AL.

spatial learner data in behaviourist, cognitivist, and constructivist pedagogical frameworks,
as all three are widely used in instructional design, and elements of each are often blended
by educators to create a specific course (Ertmer & Newby, 2013). For example, an instructor
leading a large online class with several hundred students may decide to adjust collabora-
tive group assignments by taking into account where and when particular students typi-
cally interact with the LMS. Instructors may also be able to leverage location-enabled
analytical tools that help them identify subgroups of students who are performing in
similar ways on tests and perhaps convene those students in a special study session at
a time and place that readily fits students’observed behaviours.
The role of location information in the context of learning analytics has received little
attention to date. In one of the few articles in which it has been mentioned, Becker
(2013) describes location as one of the three key data types to be collected to support
learning analytics: timing, location, and population. Becker claims that location in learning
analytics refers to the where and how students access a learning space. In the example
that follows, Becker categorizes a web forum as a relevant location associated with a pro-
totypical online learning activity. Becker appears to leave open the possibility that a rel-
evant learning location could be in physical or virtual space.
Others have invoked the example of location tracking via mobile devices to contextua-
lize the rise of data capture and collection around learning management systems (Wolf-
gang & Hendrik, 2012) and to highlight the potential usages of mobile learning systems
(Aljohani & Davis, 2012) to support learning in out-of-classroom locations. The presence
of location data in learning systems is also acknowledged in discussions about the
range of ethical and privacy challenges posed by developments in learning analytics
(Pardo & Siemens, 2014). In the geographic education literature some have begun to
experiment with the spatial components associated with student behaviour in digital
learning systems, such as Treves, Viterbo, and Haklay (2015) who logged and visualized
the movements of students on virtual field trips to explore learning outcomes.
We are aware of one prior invocation of spatial learning analytics by (Cowling, Hillier, &
Birt, 2018). In their work, Cowling et al. explore the potential for measuring the positions of
digital objects in mixed-reality environments and define spatial learning analytics around
making comparisons regarding those location manipulations in immersive learning
environments.
Understanding space and place in learning analytics can go well beyond basic pos-
itional tracking and comparison. For example, geographic information can also be
extracted from discussion posts and assignment contents using natural language proces-
sing techniques (Robinson, 2015). In combination, it may be possible to link positional
information with other forms of geographic information to provide a rich context by
which educators can situate and interpret the results of more traditional forms of learning
analytics.
Leveraging learning data with geovisual analytics
Our research focuses on evaluating the usability and utility of geovisual analytics as a
means for extending and enhancing the state of the art in learning analytics. The
origins of geovisual analytics can be found in early work to develop cartographic
science and technology in support of interactive mapping, leading to the
INTERNATIONAL JOURNAL OF CARTOGRAPHY 5

conceptualization of geovisualization (MacEachren, 1994). With geovisualization, digital
maps are constructed to support user interaction, and their analytical aims can include
exploratory visual analysis as well as more traditional types of map-driven communi-
cation (DiBiase, 1990; Gahegan, 2005). Since the mid-2000s, the field of geovisualization
within cartography has extended further to include geovisual analytics, which aims to
support analytical reasoning with interactive visual interfaces to spatial and spatio-tem-
poral data (Andrienko et al., 2007;Kraak,2008; Robinson et al., 2017). The aims of geo-
visual analytics go beyond geovisualization in that it seeks to expose and predict
patterns, support analysis of big spatial data through tight integration with compu-
tational methods, and to provide explicit support for human reasoning processes (Robin-
son et al., 2017).
Recent work by cartographers in geovisual analytics has resulted in new approaches
for understanding complex human mobility patterns (Andrienko & Andrienko, 2011;
Benke, Sheth, Betteridge, Pettit, & Aurambout, 2015; Kveladze, Kraak, & Van Elzakker,
2015), characterizing the role of location in social media streams (MacEachren et al.,
2011; Morstatter, Kumar, Liu, & Maciejewski, 2013; Pezanowski, Maceachren, Savelyev,
&Robinson,2017), and supporting spatio-temporal analysis of political and social
change (Burns & Skupin, 2013; Nelson, Quinn, Swedberg, Chu, & MacEachren, 2015;
Peuquet, Robinson, Stehle, Hardisty, & Luo, 2015). In addition to the development of
new geovisual analytics approaches for problem solving in domains like these, a concur-
rent stream of GIScience research has focused on evaluating the utility and usability of
such approaches, with the aim of developing new guidelines for designing effective
systems (Coltekin, Pettit, & Wu, 2015;Griffin & Bell, 2009;Rothetal.,2017). To date
there have been few examples in the geovisual analytics literature that focus on the edu-
cational context, perhaps because LMS-derived learner engagement data has just begun
to emerge as a viable data source for analysis. We target this particular gap with the work
we present here.
MapSieve: a prototype spatial learning analytics system
To explore the potential utility of spatial dimensions in learner data, we designed and
implemented a geovisual analytics system we call MapSieve to ingest, process, and visu-
alize spatially-referenced learner data from a Cartography MOOC taught on the Coursera
platform (Robinson et al., 2015).
In a typical demographic analysis use case with exploratory geovisualization tools, an
analyst will cross-filter multiple displays in order to identify and probe spatial patterns.
While this approach can help support inductive and abductive reasoning on spatial
data (Gahegan, 2005), it relies on the analyst to make choices about which combinations
to explore, and we know in advance that there are usually far too many potential combi-
nations for any one analyst to evaluate on their own. The science of spatial analysis and
geocomputation has developed a wide range of approaches to help solve this problem,
and indeed, many contemporary geovisual analytics systems couple computational
approaches to data reduction and clustering with interactive visual interfaces to explore
and reason about the resulting patterns.
MapSieve is designed to support coordinated-multiple view geovisualization and is
coupled with a brute-force computational technique intended to help users quickly
6A. C. ROBINSON ET AL.

uncover and explore multivariate facet combinations in learner data that exhibit spatial
clustering. In the sections that follow we explain the design of our computational
method for spatially-enhanced faceted search as well as the views included in the Map-
Sieve interface.
Spatially-enhanced multivariate facet analysis
The use of faceted search is quite common in contemporary commercial websites from
which users are expected to winnow a huge range of possibilities down to a few in
order to make purchasing decisions. Faceted search supports these tasks by allowing
users to leverage categorical information about items to select combinations of cat-
egories that fit their desires (Hearst et al., 2002). For example, faceted search can
allow a user to search an online travel site for Hotels + Rating Above 4 Stars + Has
Swimming Pool + Offers Free WiFi + Costs < $100 Nightly to find a specificplaceto
stay. Faceted search is also commonly found in visual analytics systems and has
received substantial attention in prior visualization and human–computer interaction
research (Clarkson, Desai, & Foley, 2009;Lee,Smith,Robertson,Czerwinski,&Tan,
2009;Smithetal.,2006).
In most cases, the faceted search technique requires that the data in question feature a
hierarchical categorization. However, multivariate analysis is also possible, where facets
may not necessarily be independent from one another (Ben-Yitzhak et al., 2008) and
may be coupled to parametric search techniques (Tunkelang, 2009). In previous research
the multivariate faceted search technique has been demonstrated to analyse images (Yee,
Swearingen, Li, & Hearst, 2003) and videos (Matejka, Grossman, & Fitzmaurice, 2014), as
well as spatiotemporal and attribute data in social media (Pezanowski et al., 2017).
We implement multivariate faceted search in MapSieve using a computational tech-
nique developed and tested in previous research. Our spatially-enhanced multivariate
facet analysis method combines faceted search with a brute force computational
approach that searches for facet combinations that include spatial hotspots (Robinson
& Quinn, 2018). This approach allows users to avoid dead-end search paths (which
can be common in multivariate facets due to the correlation of attributes to each
other) and highlights the combinations for which a potentially interesting spatial
pattern may be present. In previous research we introduced the design of this
method and explored its potential usefulness through case studies. In this article we
evaluate its role as part of a larger interactive geovisual analytics system and provide evi-
dence from user evaluation research regarding its potential utility and usability for sup-
porting spatial learning analytics.
Study data
The data we explore in this study are comprised of multiple variables collected from
recorded student activity in a MOOC taught in 2013 on the Coursera platform (Robinson
et al., 2015). We spatially-referenced student activity data by geocoding each student
using the Internet Protocol (IP) address associated with their MOOC enrolment record.
Geocoding utilized the commercial MaxMind, Inc. service for locating IP addresses. Out
of 49,392 students enrolled in the class, 35,783 were successfully assigned a location.
INTERNATIONAL JOURNAL OF CARTOGRAPHY 7

Students were then aggregated into 2-degree wide hexbins. The Coursera platform logs
student activities of various types, including forum posts, demographic survey data,
assignment activity, and assessment results. These data are stored across multiple SQL
tables that we joined together to make further analysis possible. In this study we focus
on five types of learning data, (1) final grade earned, (2) the number of course page
views, (3) the number of course forum posts, (4) the number of quiz attempts, and (5)
the number of video lecture plays. Together, these learning activity data help describe
key elements of student engagement in a MOOC. Even with a relatively small set of vari-
ables like this, a very large number of potential cross-filter combinations are possible,
making it a good candidate for an initial case study in applying geovisualization to learning
analytics.
Geovisualization in MapSieve
The MapSieve prototype system we have developed uses a coordinated set of visualization
components. It includes dynamic query sliders, a thematic map of the world showing the
number of students counted in hexbins for each query, a table view with summary infor-
mation about demographic characteristics, and a facet combination browser that provides
a visual overview of facet combinations and allows users to select a combination to
explore (Figure 1). These views work in combination with each other via cross-filtering.
Users may also select regions on the map to view detailed data for a subset of the
hexbins on the map.
The technology driving MapSieve includes a web client built using a combination of
OpenLayers (https://openlayers.org/) and JQuery (https://jquery.com/). The client inte-
grates with a basemap service built using GeoServer (http://geoserver.org/) and a
PostGIS (https://postgis.net/) extended PostgreSQL (https://www.postgresql.org/) data-
base containing the raw data tables that originated from the Coursera MOOC LMS.
Figure 1. The MapSieve interface features multiple coordinated views to support interactive analysis of
spatial learner data.
8A. C. ROBINSON ET AL.

Utility and usability evaluation
Methodology
To evaluate the utility and usability of MapSieve to support spatial learning analytics we
designed an experiment that included task analysis and survey components. The study
was designed to first ask users to complete prototypical spatial learning analytics tasks
before asking participants to reflect on their experiences and characterize MapSieve’s
utility and usability. Two major tasks were developed. Task 1 asked users to explore
facet combinations that we had previously identified as having interesting spatial patterns.
Users were prompted to use the faceted query tool to select these patterns and then
answer questions about student engagement and their hypotheses for what might
explain the geographic patterns that emerge. Task 2 asked participants to attempt to
develop at least three new insights on their own and to document which facet combi-
nation they believed best describes each pattern. Following these tasks, open-ended ques-
tions prompted participants to characterize the potential utility of MapSieve for
instruction, informing the design of online courses, and for conducting institutional assess-
ment in a university.
The task analysis portion of this study was followed by five-point Likert scale rating
questions focused on usability and utility metrics for MapSieve. Usability metrics were
implemented based on the System Usability Scale (Brooke, 1996) and utility metrics
were developed by the authors to probe the ways in which MapSieve provides support
for understanding student engagement, spatial patterns in learner data, its ability for sup-
porting analytical reports, and its ability for prompting/testing hypotheses about spatial
patterns in learner engagement. The complete test instrument with all task information
and survey question details is provided in Appendix A. In addition, an orientation guide
was provided to all participants to complete before the start of the experiment in order
to explain the basic features of MapSieve (Appendix B).
Participants
A total of twenty-four participants were recruited to take part in this study. We recruited par-
ticipants using email lists targeting instructional design, online teaching faculty, and geogra-
phy user groups at our university. Our intention was to gain perspectives from both
geographers as well as education experts on the potential utility and usability of the MapSieve
prototype. A twenty-dollar compensation was provided to encourage participation in this
experiment. The demographic profile of our resulting participant pool is shown in Figure 2.
Usability evaluation results
Survey responses on Usability Metrics for MapSieve help contextualize its relative
strengths and weaknesses in terms of its user experience (Figure 3). Users agreed more
often than not that they could use MapSieve without technical support from another
person, that its interface was consistent, that people could learn to use it quickly, and
that it is simple. The usability results then begin to trend to a more neutral or negative
stance when it comes to its overall integration, the confidence users have in working
with the tool, and its general ease of use. We note that no users chose the most negative
INTERNATIONAL JOURNAL OF CARTOGRAPHY 9

possible rating for any of these usability metrics. The usability results suggest that we
should focus on improving the integration of MapSieve’sdifferent pieces, provide
additional explanatory guidance in the interface to improve user confidence in what
they’re seeing, and to focus on interface improvements that will make it more likely
that users can work with the system without receiving training.
Utility evaluation results
The results from survey questions on aspects of MapSieve’s utility offer insights on its
potential to solve problems in spatial learning analytics. Each metric garnered an
average rating of ‘Agree’on our rating scale, and little variation is found between the
top and lowest ranked utility metrics (Figure 4). However, we note that the most strongly
supported metric was that MapSieve would be helpful when generating analytical reports
Figure 3. A summary of MapSieve usability ratings from our user study.
Figure 2. A visual summary of the demographic characteristics of participants in our study of the Map-
Sieve prototype.
10 A. C. ROBINSON ET AL.

to share with others, and the least supported metric was that it would help users under-
stand the spatial patterns relevant to types of student engagement. As in our usability
assessment, no ratings indicating strong disagreement were associated with any of the
utility metrics. These results indicate that MapSieve is scoring reasonably well on key
utility metrics, and in comparison with the usability ratings, it looks as though interface
usability rather than utility deserves greater attention in future MapSieve development.
Qualitative evaluation feedback
Participants provided text responses to task analysis prompts and questions about the
current and potential future utility of MapSieve. Additionally, we provided the opportunity
for users to tell us anything else on their minds after the conclusion of the study. Based on
these text responses, the following sections highlight areas of improvement we can make
to MapSieve in terms of its interface and analytical support, as well as the ways in which
users see the potential for MapSieve to support key spatial learning analytics goals to
understand engagement, improve pedagogy, and support institutional assessment. An
anonymized subset of the raw survey data table is provided in Appendix C which includes
all of the open-ended text responses from our participants. In the interest of concision, we
summarize the key themes here regarding interface improvements and analytical
improvements, and representative quotes are provided in the subsection describing par-
ticipant feedback on potential use cases for spatial learning analytics.
Interface improvements
Open-ended feedback from users indicated several areas for potential MapSieve interface
improvements. For example, users highlighted the need for MapSieve to update the
display in real-time as faceted queries are constructed, rather than having to click a
button to run each query. Several users also noted having trouble with basic panning
and zooming on the hexbin map, where both interactions behaved inconsistently com-
pared to their experiences with other web maps. Others suggested that we should add
the ability to click and set specific value thresholds for the faceted search parameters to
support more fine-grained questions.
Analytical improvements
Suggested analytical improvements to MapSieve included the need to support temporal
as well as spatial multivariate analysis. For example, to explore the extent to which student
Figure 4. A summary of MapSieve utility ratings from our user study.
INTERNATIONAL JOURNAL OF CARTOGRAPHY 11

engagement may be changing over time, and whether or not there is a potential spatial
signal associated with that change. Several participants indicated a desire to have more
contextual data to help them understand and explain patterns. For example, to include
descriptive population statistics, educational attainment information, language data,
and other types of attributes that could help explain the patterns seen in MapSieve. A
related request was to include basic descriptive statistics for learner variables. Finally, mul-
tiple users expressed a desire to compare multiple faceted query results, suggesting that a
small-multiple map design may be a good option to consider in conjunction with the facet
combination approach we have implemented in MapSieve.
Participants provided a great deal of feedback through open-ended responses to
prompts we gave them to explore specific facet combinations. Their complete responses
are available in Appendix C and we summarize the results here for the sake of brevity.
These results confirmed that users were able to detect and explain the patterns we had
explicitly prompted them to explore. When we asked users to explore on their own and
then suggest interesting facet combinations and associated hypotheses, the results were
more mixed, with about half of the users providing a facet combination number they
found noteworthy, and a higher proportion providing a question or hypothesis associated
with the learner data without any listed facet combination. This suggests that while Map-
Sieve may be prompting the kind of analytical engagement we had hoped for, that the
spatial facet combination technique is not necessarily working as intended for all of our
users. In open-ended comments a few users suggested that they wanted to compare mul-
tiple facet combinations more directly –echoing comments provided in other areas of our
survey where we asked users to suggest improvements to MapSieve more generally.
Envisioning support for spatial learning analytics
Several of our survey questions asked participants to reflect on potential use cases for
MapSieve and to suggest areas in which we could improve its utility. In terms of using
MapSieve to better understand student engagement in a course, many participants
suggested that it could help them see different educational outcomes related to language
and cultural factors. As one participant described, ‘…it would help in identifying where
students have a particular pattern of engagement, and adjust accordingly for that audi-
ence (like changing the requirements for forum posts to make them more accessible).’
One participant was less hopeful, however, stating,
I might notice general differences in how different cultures engage with the materials, but I’d
need help thinking about what to do with that. It might be more helpful when aggregated to
the level of a curriculum or a program, rather than an individual course.
We also asked participants to envision a way in which a tool like MapSieve could be used to
improve the design of a course. Participant responses to this prompt tended to focus on how
this type of system could help them create interventions to target specificregionalsubgroups
in a MOOC. For example, one response said, ‘I would think MapSieve could help designers
create forms of assessment that could be adapted for culture, language and interest.’
Another said, ‘It could potentially be used to better direct student and support resources
based on each specific class.’One participant suggested an inversion of our assumed use
case, and said, ‘One way for it to help build a better online course is to give students advice
before they actually start the course according to the student’s location and past grades.’
12 A. C. ROBINSON ET AL.

Finally, we asked participants to think about using a tool like MapSieve from the per-
spective of an institutional analyst. Participants suggested that institutions could use
spatial learning analytics to help direct recruiting efforts, to provide improved advising
support for students from around the world, and to provide reporting to faculty based
on institution-wide data to suggest pedagogical interventions. For example, one partici-
pant stated that spatial learning analytics could, ‘Help(ing) students and faculty predict
what kinds of engagement will promote a higher grade (and hopefully learning).’
Another participant thought it would help, ‘…narrow down target countries or regions
to advertise their online courses / program(s) to attract more students.’One person
suggested that it would be a useful framework for supporting study groups; ‘It may be
interesting, given the oft believed idea that learning occurs best in groups, to see if stu-
dents in areas with other students could meet up to do coursework.’
Discussion and limitations
Our research to design and evaluate a prototype geovisualization for supporting spatial
learning analytics has shown that educators and geographers are capable of using such
a system with minimal training, that relatively simple interactive tools can support basic
spatial learning analytics tasks, and that there are a variety of interesting future directions
in which we should experiment with spatial learning analytics to understand learner
behaviour, improve teaching, and support institutional assessment. Along the way we
have also identified a variety of specific usability and utility improvements that can be
made to the MapSieve tool and to consider when developing future geovisualization
environments for spatial learning analytics.
While the use of spatially-enabled query facets was something we explicitly prompted
in multiple tasks and survey questions, it did not feature prominently in participant
responses in questions where we did not ask about it directly. We do not assume this
means the method is therefore already ideally-designed, but rather that it supported
the tasks we prompted participants to complete, and that it did not pose significant usabil-
ity issues above and beyond the rest of the MapSieve interface. One of the few direct com-
ments on the technique outside of questions where we specifically asked them to talk
about query facets suggested that we need to make it easy to compare multiple facet
combinations. What we have learned from this aspect of our study is that we need to
further evaluate its usefulness by comparing what people find with facet combinations
versus what those may find in a version of MapSieve that does not include this feature.
Overall usability ratings were generally good for MapSieve, and no participants found
the need to provide a strong negative rating for any of the usability heuristics. However,
we note the need to focus on improving the coordination between MapSieve elements,
fixing bugs associated with panning and zooming the map, and in making small visual
refinements to the user interface to improve legibility and aesthetic qualities. Users also
asked us to make it possible for query combinations to generate results in real time to
obviate the need to click a button to run a query.
Utility ratings for the MapSieve prototype were also generally positive, once again with
no responses indicating a strong negative rating on any of our utility metrics. While there
was not a great deal of noteworthy difference between the best supported and least sup-
ported utility metric, we note that the metrics on exploratory utility earned slightly higher
INTERNATIONAL JOURNAL OF CARTOGRAPHY 13

ratings overall as compared to the metrics we used to explanatory utility. Users want to
know more about what constitutes student engagement, and they also want to drill
down further into the spatial patterns that may emerge from learner engagement data.
In open-ended feedback at the end of our survey, one user expressed a desire to better
understand the spatial cluster detection element of our query facet tool. Another user
expressed a desire to integrate time into the MapSieve framework to understand the dyna-
mism in patterns of engagement.
In this research we acknowledge the limitation associated with the fact that we have
chosen a single course context in which to evaluate a spatial learning analytics approach.
We were also limited in terms of the usable geographical scale for which location data
existed for our case study. It would be possible with higher precision geocoding and a
greater density of students in smaller areas to explore patterns below the regional level
(for example, if we were exploring data from a single college campus rather than a
course taught to people around the world), but we were not able to accomplish that
with this project.
Conclusions and future research
In this work we have described the design and evaluation of a prototype spatial learning
analytics geovisualization. The MapSieve application couples a computational approach
using spatially-enabled query facets to an interactive map and to support guided explora-
tion in learner data. Our evaluation results suggest that the prototype system we created
scores well in terms of its overall usability and utility in support of spatial learning analytics.
Our study participants provided guidance through their rating responses and open-ended
responses regarding multiple ways in which we can improve the usability and utility of
MapSieve and have suggested areas to focus on in the future to improve support for
spatial learning analytics more generally. Few spatial learning analytics systems have
been described in the literature, and we are not aware of similar work that has evaluated
this type of geovisualization with educators and geographers. Our progress here advances
what we know about supporting one aspect of the previously proposed spatially-enabled
smart campus (Skupin, 2013), showing how spatial data science may be applied to learner
data itself to improve teaching, course design, and institutional assessment. It also illumi-
nates the potential to leverage data from open geospatial education (Belgiu, Strobl, & Wal-
lentin, 2015) in order to improve learning outcomes.
In future work we wish to address the challenges our study participants raised regard-
ing the need for a spatial learning analytics system to explicitly incorporate time. A wide
range of factors may play into the decision for a learner to engage with an online course at
a given time of the day, including work considerations and access to the internet. If we
were to continue using the query facet approach we have adopted here for MapSieve,
we would need to further extend that method to handle and represent spatiotemporal
data. We hypothesize that in addition to linear representations of time, that including
configurable calendar views would also be quite useful, as we can reasonably expect
that patterns will be present in daytime activity vs. night activity, as well as between
work days and the weekend.
Many of our users also reported a desire to see additional demographic data along with
the ability to explore it across multiple spatial scales. In our work here we used data from a
14 A. C. ROBINSON ET AL.

single MOOC, aggregated to a single hexagonal grid. In future research we will need to
address the challenges associated with comparing multiple courses, analysing traditional
online classes that have much smaller student populations than MOOCs, fusing together
learner data with a deeper set of demographic variables, and supporting geovisualization
across a range of user-configurable spatial scales. Developing and evaluating new carto-
graphic representations to support analysing big spatial data in other contexts will likely
translate into the spatial learner analytics context as well.
One of our study participants encouraged us to think about the use of spatial learning
analytics from the student perspective. We know very little about what students them-
selves expect to leverage in terms of geographic information in personalized learning
systems that are driven by learning analytics approaches. How can students be made
aware of classmate presence in an LMS? It is often hard for online students to feel the pres-
ence of their classmates in quite the same way as it is experienced in a classroom or lab
setting, and we believe there are ways to leverage spatial information about LMS inter-
action that may help students achieve this feeling at a distance. We also need to know
how can we help students form ad hoc groups to study and collaborate together at a dis-
tance. Furthermore, it is possible that self-awareness of patterns highlighting where/when
they are engaging with an LMS may be helpful to students. Finally, we need to engage
students, faculty, and administrators in frank discussions on managing the spatial dimen-
sions of privacy associated with the expansion of learning analytics methods and systems.
One idea we would like to explore is to aggregate campus location data from an LMS into
types of locations, and explore a generalized map of those location types. For example, all
events taking place in a library, dorm, classroom, or laboratory could be categorized as
such and visualized as part of a location category, rather than emphasizing an explicit
location reference that would exacerbate location privacy issues. Understanding what
types of learner engagement happen at library locations versus laboratory settings
could be quite useful to educators and institutional analysts and eschew the need to
focus on the exact locations of individuals. Therefore, we suggest that the spatial dimen-
sion of spatial learning analytics need not be explicit coordinates.
At the institutional level, analysts are expected to report on key metrics for course com-
pletion, engagement, progress toward degree milestones, and many other aspects of
learner activity. Such metrics are often driven by strategic planning goals and accreditation
concerns. How might institutional analysts leverage geographic information to better
understand adult learners in online programmes who may be participating around the
world? At the institutional level, knowing which places on campus are attracting lots of
interaction with the campus LMS at specific times may improve the provisioning of
student support services or scheduling useful open hours for a library. We need to evaluate
the potential utility of geospatial information more broadly in the context of learning ana-
lytics, as contemporary learning analytics approaches do not routinely consider geo-
graphic context.
As we reflect on the results of our initial study on the potential of spatial learning ana-
lytics, we see the need for a new research trajectory that brings together cartographers,
geographers, and educators to identify key opportunities for extracting and visualizing
geographic insights in the context of learning analytics, design and develop prototype
spatial learning analytics systems for analysing learner activity, and to evaluate these
systems with the appropriate stakeholders.
INTERNATIONAL JOURNAL OF CARTOGRAPHY 15

Acknowledgements
We wish to thank Scott Pezanowski at the GeoVISTA Center for providing technical assistance to
support the implementation of the MapSieve prototype.
Disclosure statement
No potential conflict of interest was reported by the author(s).
Funding
This work was supported in part by a Research Initiation Grant from the Center for Online Innovation
in Learning at Penn State University.
Notes on contributors
Dr. Anthony C. Robinson is Associate Professor, Director for Online Geospatial Education pro-
grammes and Assistant Director for the GeoVISTA research centre in the Department of Geography
at Penn State University. Dr Robinson’s research focuses on the science of interface and interaction
design for geographic visualization. He currently serves as the Co-Chair of the Commission on Visual
Analytics for the International Cartographic Association.
Cary L. Anderson, M.S. Cary is a doctoral candidate in Marketing in the Joseph M. Katz Graduate
School of Business at the University of Pittsburgh. Her research focuses on the influence of see-
mingly-incidental design factors in maps and other visual graphics on user cognition, emotion,
and behaviour.
Dr. Sterling D. Quinn is Assistant Professor and GIS Program Director in the Department of Geogra-
phy at Central Washington University. His research interests include crowdsourced geographic data,
free and open source GIS, and the politics of online maps.
ORCID
Anthony C. Robinson http://orcid.org/0000-0002-5249-8010
Cary L. Anderson http://orcid.org/0000-0003-1702-9590
Sterling D. Quinn http://orcid.org/0000-0002-4900-8885
References
Adams, P. C. (2018). Geographies of media and communication III: Academic communications and
the digital communication environment. Progress in Human Geography,43(4), 739–748. doi:10.
1177/0309132518763849
Aljohani, N. R., & Davis, H. C. (2012, October 16–18). Learning analytics in mobile and ubiquitous learn-
ing environments. Proceedings of the 11th international conference on mobile and contextual
learning, MLearn 2012, Helsinki, 70–77. Retrieved from http://ceur-ws.org/Vol-955/./papers/
paper_70.pdf
Andrienko, G., Andrienko, N., Jankowski, P., Keim, D., Kraak, M. J., MacEachren, A. M., & Wrobel, S.
(2007). Geovisual analytics for spatial decision support: Setting the research agenda.
International Journal of Geographical Information Science,21(8), 839–857.
Andrienko, N., & Andrienko, G. (2011). Spatial generalization and aggregation of massive movement
data. IEEE Transactions on Visualization and Computer Graphics,17(2), 205–219.
Becker, B. (2013). Learning analytics: Insights into the natural learning behavior of our students.
Behavioral & Social Sciences Librarian,32(1), 63–67. doi:10.1080/01639269.2013.751804
16 A. C. ROBINSON ET AL.

Belgiu, M., Strobl, J., & Wallentin, G. (2015). Open geospatial education. ISPRS International Journal of
Geo-Information,4(2), 697–710.
Benke, K. K., Sheth, F., Betteridge, K., Pettit, C. J., & Aurambout, J. P. (2015). Application of geovisual
analytics to modelling the movements of ruminants in the rural landscape using satellite tracking
data. International Journal of Digital Earth,8(7), 579–593. doi:10.1080/17538947.2013.872703
Bennett, R., & Kent, M. (Eds.). (2017). Massive open online courses and higher education | what went
right, what went wrong and where to next? (1st ed.). Retrieved from https://www.taylorfrancis.
com/books/e/9781315594248
Ben-Yitzhak, O., Golbandi, N., Har’El, N., Lempel, R., Neumann, A., Ofek-Koifman, S., …Yogev, S. (2008).
Beyond basic faceted search. Proceedings of the 2008 International Conference on Web Search and
Data Mining,33–44. doi:10.1145/1341531.1341539
Brooke, J. (1996). SUS: A “quick and dirty”usability scale. In P. W. Jordan, B. Thomas, B. A.
Weerdmeester, & A. L. McClelland (Eds.), Usability evaluation in industry (p. 252). London: Taylor
and Francis.
Burns, R., & Skupin, A. (2013). Towards qualitative geovisual analytics: A case study involving places,
people, and mediated experience. Cartographica: The International Journal for Geographic
Information and Geovisualization,48(3), 157–176. doi:10.3138/carto.48.3.1691.
Campbell, J. P., DeBlois, P. B., & Oblinger, D. G. (2007). Academic analytics: A new tool for a new era.
EDUCAUSE Review,42(4), 40–57.
Chen, Q., Chen Y., Liu, D., Shi, C., Wu, Y., & Qu, H. (2015). PeakVizor: Visual analytics of peaks in video
clickstreams from massive open online courses. IEEE Transactions on Visualization and Computer
Graphics,99,1–1. doi:10.1109/TVCG.2015.2505305
Clarkson, E., Desai, K., & Foley, J. (2009). Resultmaps: Visualization for search interfaces. IEEE
Transactions on Visualization and Computer Graphics,15(6), 1057–1064. doi:10.1109/TVCG.2009.176
Coltekin, A., Pettit, C., & Wu, B. (2015). Geovisual analytics: Human factors. International Journal of
Digital Earth,8(8), 595–598. doi:10.1080/17538947.2015.1047173
Cowling, M. A., Hillier, M., & Birt, J. R. (2018). Integrating mixed reality spatial learning analytics into
secure electronic exams. ASCILITE 2018 conference proceedings - open oceans: Learning without
borders, 330–334. Retrieved from https://research.bond.edu.au/en/publications/integrating-
mixed-reality-spatial-learning-analytics-into-secure-
DiBiase, D. (1990). Visualization in the earth sciences. Earth and Mineral Sciences, Bulletin of the College
of Earth and Mineral Sciences, The Pennsylvania State University,59(2), 13–18.
Ertmer, P. A., & Newby, T. J. (2013). Behaviorism, cognitivism, constructivism: Comparing critical fea-
tures from an instructional design perspective. Performance Improvement Quarterly,26(2), 43–71.
doi:10.1002/piq.21143
Gahegan, M. (2005). Beyond tools: Visual support for the entire process of GIScience. In J. Dykes, A. M.
MacEachren, & M. J. Kraak (Eds.), Exploring geovisualization (pp. 83–99). London: Elsevier.
Griffin, A. L., & Bell, S. (2009). Applications of signal detection theory to geographic information
science. Cartographica: The International Journal for Geographic Information and
Geovisualization,44(3), 145–158. doi:10.3138/carto.44.3.145
Hearst, M., Elliott, A., English, J., Sinha, R., Swearingen, K., & Yee, K.-P. (2002). Finding the flow in web
site search. Communications of the ACM,45(9), 42–49. doi:10.1145/567498.567525
Hodges, C., & Grant, M. (2015). Theories to support you: Purposeful use of learning management system
features. Berlin: Association for the Advancement of Computing in Education, 481–486.
Janelle, D., Kuhn, W., Gould, M., & Lovegreen, M. (2014). Advancing the spatially enabled smart campus:
Specialist meeting final report (Report). Santa Barbara: Center for Spatial Studies, University of California.
Kraak, M.-J. (2008). Editorial: From geovisualisation toward geovisual analytics. The Cartographic
Journal,45(3), 163–164. doi:10.1179/174327708X315183
Kveladze, I., Kraak, M.-J., & Van Elzakker, C. P. J. M. (2015). The space-time cube as part of a GeoVisual
analytics environment to support the understanding of movement data. International Journal of
Geographical Information Science,29(11), 2001–2016. doi:10.1080/13658816.2015.1058386
Lee, B., Smith, G., Robertson, G. G., Czerwinski, M., & Tan, D. S. (2009). Facetlens: Exposing trends and
relationships to support sensemaking within faceted datasets. Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems, 1293–1302. doi:10.1145/1518701.1518896
INTERNATIONAL JOURNAL OF CARTOGRAPHY 17

MacEachren, A. M. (1994). Visualization in modern cartography: Setting the agenda. In A. M.
MacEachren & D. R. F. Taylor (Eds.), Visualization in modern cartography (pp. 1–12). Retrieved
from https://www.sciencedirect.com/bookseries/modern-cartography-series/vol/2/suppl/C
MacEachren, A. M., Jaiswal, A., Robinson, A. C., Pezanowski, S., Savelyev, A., Mitra, P., …Blanford, J.
(2011). SensePlace2: Geotwitter Analytics Support for Situation Awareness. IEEE conference on
visual analytics science and technology.
Matejka, J., Grossman, T., & Fitzmaurice, G. (2014). Video lens: Rapid playback and exploration of large
video collections and associated metadata. Proceedings of the 27th Annual ACM Symposium on
User Interface Software and Technology, 541–550. doi:10.1145/2642918.2647366
Morstatter, F., Kumar, S., Liu, H., & Maciejewski, R. (2013). Understanding Twitter data with
TweetXplorer. Proceedings of the 19th ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining, 1482–1485. doi:10.1145/2487575.2487703
Nelson, J., Quinn, S., Swedberg, B., Chu, W., & MacEachren, A. (2015). Geovisual analytics approach to
exploring public political discourse on Twitter. ISPRS International Journal of Geo-Information,4(1),
337–366.
Pardo, A., & Siemens, G. (2014). Ethical and privacy principles for learning analytics. British Journal of
Educational Technology,45(3), 438–450. doi:10.1111/bjet.12152
Peuquet, D. J., Robinson, A. C., Stehle, S., Hardisty, F. A., & Luo, W. (2015). A method for discovery and
analysis of temporal patterns in complex event data. International Jounal of Geographical
Information Science,29(9), 1588–1611.
Pezanowski, S., Maceachren, A. M., Savelyev, A., & Robinson, A. C. (2017). Senseplace3: A geovisual
framework to analyze place-time-attribute information in social media. Cartography and
Geographic Information Science, 44. doi:10.1080/15230406.2017.1370391
Pirolli, P., & Card, S. (2005). The sensemaking process and leverage points for analyst technology as
identified through cognitive task analysis. International conference on intelligence analysis.
Presented at the McLean, VA.
Robinson, A. C. (2015). Exploring class discussions from a massive open online course (MOOC) on car-
tography. In J. Brus, A. Vondrakova, & V. Vozenilek (Eds.), Modern trends in cartography: Selected
papers of CARTOCON 2014 (pp. 173–182). doi:10.1007/978-3-319-07926-4_14
Robinson, A. C., Demšar, U., Moore, A. B., Buckley, A., Jiang, B., Field, K., …Sluter, C. R. (2017).
Geospatial big data and cartography: Research challenges and opportunities for making maps
that matter. International Journal of Cartography,1–29. doi:10.1080/23729333.2016.1278151
Robinson, A. C., Kerski, J., Long, E. C., Luo, H., DiBiase, D., & Lee, A. (2015). Maps and the geospatial
revolution: Teaching a massive open online course (MOOC) in geography. Journal of Geography
in Higher Education,39(1), 65–82. doi:10.1080/03098265.2014.996850
Robinson, A. C., & Quinn, S. D. (2018). A brute force method for spatially-enhanced multivariate facet
analysis. Computers, Environment and Urban Systems,69,28–38. doi:10.1016/j.compenvurbsys.
2017.12.003
Romero, C., & Ventura, S. (2007). Educational data mining: A survey from 1995 to 2005. Expert Systems
with Applications,33(1), 135–146. doi:10.1016/j.eswa.2006.04.005
Roth, R. E., Çöltekin, A., Delazari, L., Filho, H. F., Griffin, A., Hall, A., …van Elzakker, C. P. J. M. (2017). User
studies in cartography: Opportunities for empirical research on interactive maps and visualiza-
tions. International Journal of Cartography,3(Suppl. 1), 61–89. doi:10.1080/23729333.2017.1288534
Shum, S. B., & Ferguson, R. (2012). Social learning analytics. Educational Technology & Society,15(3), 3–26.
Siemens, G., & Baker, R. S. J. d. (2012). Learning analytics and educational data mining: Towards com-
munication and collaboration. Proceedings of the 2nd International Conference on Learning
Analytics and Knowledge, 252–254. doi:10.1145/2330601.2330661
Skupin, A. (2013). Spatially enabled smart campus. Center for Spatial Studies Specialist Meeting, 1–3.
Santa Barbara, CA.
Smith, G., Czerwinski, M., Meyers, B., Robbins, D., Robertson, G., & Tan, D. S. (2006). Facetmap: A scal-
able search and browse visualization. IEEE Transactions on Visualization and Computer Graphics,12
(5), 797–804. doi:10.1109/TVCG.2006.142
Thomas, J. J., & Cook, K. A. (2005). Illuminating the path. New York: IEEE Press.
18 A. C. ROBINSON ET AL.

Treves, R., Viterbo, P., & Haklay, M. (2015). Footprints in the sky: Using student track logs from a “bird’s
eye view”virtual field trip to enhance learning. Journal of Geography in Higher Education,39(1), 97–
110. doi:10.1080/03098265.2014.1003798
Tunkelang, D. (2009). Faceted search. Synthesis Lectures on Information Concepts, Retrieval, and
Services,1(1), 1–80. doi:10.2200/S00190ED1V01Y200904ICR005
United States Department of Education. (2014). Integrated Postsecondary Education Data System
(IPEDS), Spring 2013 and Spring 2014, Enrollment component. Retrieved from https://nces.ed.
gov/programs/digest/d14/tables/dt14_311.15.asp
Wolfgang, G., & Hendrik, D. (2012). Translating learning into numbers: A generic framework for learn-
ing analytics. Journal of Educational Technology & Society,15(3), 42–57.
Wu, T., Yao, Y., Duan, Y., Fan, X., & Qu, H. (2016). Networkseer: Visual analysis for social network in
MOOCs. 2016 IEEE Pacific Visualization Symposium (PacificVis), 194–198. doi:10.1109/PACIFICVIS.
2016.7465269
Yee, K.-P., Swearingen, K., Li, K., & Hearst, M. (2003). Faceted metadata for image search and browsing.
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 401–408. doi:10.
1145/642611.642681
INTERNATIONAL JOURNAL OF CARTOGRAPHY 19