Content uploaded by Irena White
Author content
All content in this area was uploaded by Irena White on Jul 28, 2017
Content may be subject to copyright.
Page | 1
Modelling the complexity of technology adoption in higher
education teaching practice
Irena White, Heather Smigiel
Flinders University, Australia
James Levin
University of California San Diego, USA
The study reported in this paper examines the inter-relationships between organisational roles
during the process of sustaining the diffusion of e-learning innovations in higher education
teaching practice. Through this process, new ways of teaching and learning with digital
technologies become adopted by a mainstream group of academics in similar university
teaching roles. Unlike top-down implementations of enterprise-wide e-learning management
systems that succeed because they are mandated by university policies, e-learning
innovations that originate in higher education teaching practice are generally bottom-up
initiatives that mostly fail to achieve mainstream adoption. Previous studies have viewed
technology adoption in teaching practice as a simple linear process. These studies have relied
on traditional case study and survey research methods to identify individual and institutional
actors and causal factors in this process. The methods used in these studies do not explain the
inter-relationships of actors and factors in what is a dynamic, non-linear, complex process.
This study is the first to investigate this problem from a complexity perspective. The study
uses computer modelling to simulate and explore the inter-relationships between
organisational roles within university systems that enable and inhibit mainstream bottom-up
adoption of e-learning innovations that originate in higher education teaching practice.
Introduction
The continuing lag in adoption of e-learning innovations into mainstream higher education teaching
practice is emerging as a growing concern amongst universities (Bates & Sangrà, 2011; Laurillard, Oliver,
Wasson & Ulrich, 2009; Selwyn, 2013). The 2017 New Media Consortium Horizon Report Higher
Education Edition warns that "if institutions do not already have robust strategies for integrating these now
pervasive approaches, then they simply will not survive" (Adams Becker et al., 2017, p. 2). This concern is
particularly evident where large investments are being made by universities in technology infrastructures.
These investments are occurring in an effort to remain competitive in a growing global education
marketplace, fuelled by expectations of an increasingly digitally literate population of students (Johnson et
al., 2016). To meet the demands of this highly leveraged, competitive and emerging digital education
landscape, government policy recommendations are directing universities to find more scalable solutions
to the adoption of e-learning innovations (U.S. Department of Education, Office of Educational
Technology, 2017, p. 74). How to scale up university adoption of e-learning innovations continues to
present challenges for educational researchers, as noted by Sabelli and Harris (2015):
Getting innovations to scale is an increasingly important mandate for educational research,
yet also a vexing challenge for researchers who have attempted to take this on. A common
perspective on scaling considers it fundamentally as an issue of how to take interventions
that have been shown to work in a small number of settings and transfer them to a larger
number of settings (Sabelli & Harris, 2015, p. 13).
The process of scaling up the adoption of new interventions is commonly referred to in the research
literature as the diffusion of innovations (DoI) (Rogers, Medina, Rivera, & Wiley, 2005; White, 2010). This
term has been popularised through Everett Rogers’ seminal model of diffusion of innovations (Elgort, 2005)
and DoI theory (Rogers, 2003). In his final published paper, Rogers et al., (2005) defines the sustainable
diffusion of an innovation as occurring when “critical mass is reached at the point where there are enough
adopters that further diffusion becomes self-sustaining” (p. 7). Most e-learning innovations that originate
in higher education teaching practice fail to reach this critical mass point which Markus (1987) describes
simply as “the way we do things around here” (p. 506) and Pacansky-Brock (2015) portrays as achieving
mainstream adoption. Just as in the early pioneering years of e-learning, during the 1960s, e-learning
innovations are still failing to be adopted by the mainstream of academics in teaching roles in the
Page | 2
universities where these innovations originate (Nicholson, 2007; Reid, 2012). This is occurring even when
the innovations are proven, through rigorous evaluation studies, to be effective in teaching and learning.
The failure to achieve mainstream adoption of proven e-learning innovations in universities has been
recognised for over 20 years in the research literature as a global problem (Nichols, 2008). This has led to
numerous research studies. However, these studies have continued to view DoI from a traditional linear
perspective based on scaling up from small to large implementations. It is a view that contrasts with the
proposal by Rogers et al., (2005) for a hybrid theory of DoI and Complex Adaptive Systems (CAS). This
study addresses the limitations of previous studies by adopting an interpretive complexity perspective that
applies and extends Rogers et al., (2005) hybrid DoI CAS hybrid theory by using computer modelling
(Levin, 2015) within an interpretive interactionism (Denzin, 2001) research design.
From causality to complexity
Over the past two decades, educational researchers around the world have focussed on the individual and
institutional actors and the factors in the diffusion of e-learning. The actors represent the various e-learning
stakeholders in universities. The factors represent both the drivers for success and the challenges or barriers
that respectively enable and inhibit the DoI process. These previous studies have reported a range of diverse
findings and conclusions about the causes and effects of both top-down management-driven
implementations of e-learning systems and bottom-up adoption of e-learning innovations that originate in
higher education teaching practice. The majority of these investigations have been conducted as case
studies, ranging from one-off single cases in one or several universities to 43 cases in a 2006 to 2012
longitudinal study in one university (Csete & Evans, 2013). In addition to citing case studies, published
articles on the subject of technology adoption in universities also report results of surveys and interviews
and, in some cases, provide no more than a few examples and anecdotes to support their findings and
discussions. The causal factors reported in these articles derive from experiences and influences that have
been found to play a part in technology adoption decision making by university teaching staff. Previous
studies range from exploring the perceptions, beliefs and attitudes of individuals in university teaching roles
towards the adoption of new technologies in their teaching practice (Alexander, 2006; Smigiel, 2013) to
examining the roles of institutional structures, systems, policies and practices in implementing technology
adoption (Gunn, 2010; Salmon & Angood, 2013; Csete & Evans, 2013). Some studies combine an
investigation of the roles of both individual and institutional actors (Elgort, 2005; Sharpe, Benfield &
Francis, 2006; Birch & Burnett, 2009; Gunn & Herrick, 2012; Henderson, 2015). Causal factors examined
in these studies include the role and perceptions of students (Smigiel, 2013, Henderson, 2015), the
pedagogical impact of teaching and learning processes (Elgort, 2005; Birch & Burnett, 2009), funding
availability (Gunn & Herrick, 2012) and the features of e-learning products (Alexander, 2006; Gunn &
Herrick, 2012). This focus on isolating the causal factors and actors in the DoI process “has been criticised
for over-simplifying what is often a complex organisational change process” (Nutley, Davies & Walter,
2002, p. 13). In an extensive review of published literature from around the world, Casanovas (2010)
concludes that previous studies “focus on factors and prescribed practices, but not on the human interactions
during the transition from individual adoption until institutionalization” (p. 73). The limitations of viewing
DoI as a simple linear process of scaling up from small to larger numbers based on cause and effects studies
has led to recommendations for further research to examine the process of technology adoption from a
complexity perspective (Rogers et al., 2005). From this perspective, understanding the “relationships
among members of a system” (Rogers et al., 2005, p. 3) in which DoI is viewed as a non-linear “complex
emergent phenomena” (Kiesling, Günther, Stummer & Wakolbinger, 2012, p. 1) provides a new and
challenging opportunity for further investigation.
The research questions addressed in this study emerge from a view of several layers of complexity that
depicts universities as complex educational systems (Jacobsen, 2015) made up of “diverse but
interconnected elements” (Rossiter, 2006, p. 261) in which DoI is viewed as a complex process that operates
within a complex adaptive system (Rogers et al., 2005). Rossiter (2006) adds to this view by also suggesting
that complexity is “an integral dimension of e-learning” (p. 245). Snyder (2013) suggests that questions
about complexity "take the viewpoint of individual (or institutional) actors’ effect on the wider system
rather than the reverse" (p. 9). The five guiding questions in this study apply this multi-perspective
viewpoint by examining: (1) university actors as diverse elements in a complex educational system; (2) the
critical success factors in the sustainable diffusion of innovations as diverse elements in a complex emergent
process; (3) the association between factors, in a complex process, with the roles of actors, in a complex
system; (4) the interactions in the inter-relationships between factors and actors as diverse interconnected
elements in a complex system and process; (5) implications for organisational change suggested by the
Page | 3
interaction of factors and actors in a complex process within a complex system. These questions inform the
main research question which seeks to identify actionable insights rather than the causal factors found in
previous studies of technology adoption in higher education teaching practice. Cooper (2012) defines the
term actionable insights as having the “potential for practical action rather than either theoretical
description or mere reporting” (p. 4) and concludes that “too frequently, management reports fail to provide
this level of clarity and leave actionable insights as missed opportunities" (Cooper, 2012, p. 4). There is
some debate about how complexity theory can provide these insights. Castellani (2014) argues that
“complexity theory is not so much a substantive theory, as much as it is an epistemologically explicit
attempt to model social life in complex systems terms” (p. 10). The main research question in this study
adopts this complexity system perspective by exploring how modelling the inter-relationships between
organisational actors and factors in technology adoption within higher education teaching practice can elicit
actionable insights. In proposing a modelling approach to studying education as a complex system, Levin
and Jacobson (2017) argue that existing quantitative and qualitative methodologies used in educational
research are insufficient for understanding the nonlinear dynamics of education when viewed as a complex
system. By seeking to model interactions that are nonlinear and dynamic in the relationships between
critical success factors and university system actors, the questions guiding this research reflect a complex
systems perspective while building on the results of previous studies.
Connecting the actors and the factors
Recommendations from educational researchers and policy makers suggest the need to investigate both the
relationships between the DoI actors in a university system and the DoI factors identified in previous
studies. For example, Stepanyan, Littlejohn and Margaryan (2010) propose that "a deeper understanding
of the factors of sustainability and, most importantly, their inter-relationship” (p. 30) is necessary for future
studies of DoI. They add, three years later, that an “insight into multiple stakeholder perspectives, could
provide better pointers toward future e-learning sustainability" (Stepanyan, Littlejohn & Margaryan, 2013,
p. 98) from both an individual and institutional level. The view of these researchers is also supported in the
2014 report of an investigation by the European Commission Directorate for Education and Training study
on innovation in higher education. This report recommends further research is needed about “the roles of
the key stakeholders in implementing innovation" (Brennan et al., 2014, p. 1) and concludes with a policy
recommendation to “clarify the roles of the different actors" (Brennan et al., p. 7) in this process. An
Australasian Council on Open, Distance and E-learning (ACODE) research study concludes its
investigation of 15 case studies of bottom-up adoption of e-learning products in Australian and New
Zealand universities by observing:
It seems possible that if the universities concerned had a clearer understanding of their role
in the development and support of elearning innovations, some of the challenges around
sustainability might be discussed and addressed at a strategic level throughout the process
from development to product maturity (Gunn & Herrick, 2012, p. 15).
Furthermore, Gunn and Herrick (2012) recommend that “universities consider and clarify the roles of key
individuals, practitioners and departments in the support, evaluation and adoption of new elearning
products” (p. 2). They suggest the need to investigate questions “around the institutional structures and
processes where the innovators work” (Gunn & Herrick, 2012, p. 16). This view is supported by Kiesling
et al., (2012) who suggest that “more research is also needed on the structure of social systems, which plays
a key role in diffusion processes” (p. 43). This reflects similar calls for this type of research to be conducted
around the world (Bui, 2015; Singh & Hardaker, 2014).
There are only a few available published case studies of bottom-up adoption of e-learning innovations in
Australian and New Zealand universities. Of these, the largest study was conducted by Gunn and Herrick
(2012) for a project funded by ACODE. Other smaller studies have been conducted by the Office for
Learning and Teaching (OLT) http://www.olt.gov.au/ and its predecessor bodies between 2004 and 2016
when funding for new OLT projects ceased. The OLT projects investigated case studies of mostly top-
down technology adoption in Australian universities and were concerned largely with the describing the
dissemination rather than the diffusion of e-learning innovations. These case studies describe the
organisational structure within a university system as being made up of both individual and institutional
actors. The individual actors in this study represent the innovators and adopters who are in academic
teaching roles and the institutional actors represent management and professional development and/or
technical support roles with responsibilities for e-learning within their university. Staff in institutional
Page | 4
management roles and support services together with individual lecturers who are innovators and adopters
of new ways of teaching and learning with digital technologies, each play a part in the sustainable diffusion
of e-learning innovations. Roberston (2008) describes the diffusion of e-learning innovations in universities
as occurring across three systems: macro, meso and micro. Within higher education these three systems are
respectively described as “the organisational activity system – largely represented by management … the
technological activity system – largely represented by information technology specialists … [and] the
pedagogic activity system – represented by those with primary responsibility for teaching and learning”
(Roberston, 2008, p. 821). Within universities each these three activity levels within a university system
plays a different role in the collaboration required for the diffusion of innovations in education (White,
2010). Pacansky-Brock (2015) suggests that a new way is needed to connect these diverse roles in the DoI
process:
Our models of faculty support are out-dated remnants of machine-age thinking and we are
missing rich opportunities for collaborative solutions. We must begin to understand each
higher education institutions [sic] as members of a complex ecosystem. Each is an organic
system that is in a continuous state of change (Pacansky-Brock, 2015, para 5).
There appear to be no previous studies that have investigated the DoI process from the perspective of this
complex and constantly changing higher education technology landscape. This may be largely because the
skills and resources required to visualise this complex ecosystem have previously required a highly
specialised knowledge of mathematical equations and computational modelling tools (Levin & Jacobson,
2017; Rogers et al., 2005). Over the past decade, computer simulation modelling applications have become
more freely available and provide user interfaces that make it possible to more easily build models that
explore non-linear interactions within, for example, institutional structures (Levin & Jacobson, 2017). The
aim of this study is to provide further evidence that computer simulation models can be used to visualise
and interpret the interactions and inter-relationships between actors and factors in complex systems with
complex problems and thus lead to actionable insights.
Using Multi-Mediator Modelling
The multi-mediator modelling (MMM) computer simulation used in this study builds on the results of
proof-of-concept research reported in Levin and Datnow (2012) and most recently in Levin and Jacobson
(2017). The coding and concepts in the MMM tool and framework were developed by Professor James
Levin at the University of California San Diego (UCSD) La Jolla Department of Education Studies. The
MMM tool uses code from NetLogo, “a free multi-platform agent-based model-building environment
developed by Wilensky (1999) and his colleagues at Northwestern University” (Levin, 2015, p. 3). More
information about NetLogo can be found at http://ccl.northwestern.edu/netlogo/.
MMM originates from agent based modelling (ABM) which has been described by Axelrod (2005), an
American political scientist, as a “third way of doing science” (p. 1). Tubaro and Casilli (2010) define the
features of ABM in the following terms:
ABM uses computational techniques to simulate dynamic interactions between individual
entities in a given social context. Emphasis is not on variables as in statistical models, but on
'agents' (Smith and Conrey, 2007) that are endowed with attributes and behavioral [sic] rules,
and act on the basis of some decision-making criterion or heuristic – an epistemological
posture sometimes illustrated by the catchy slogan 'from factors to actors' (Tubaro & Casilli,
2010, p. 61).
As a research method, ABM is derived from the complexity sciences. Jacobson (2015) suggests that “the
use of computer modelling, particularly ABMs, can provide research and policy insights about complex
educational systems” (p. 310). The application of MMM extends traditional social science research
methods, such as case studies and surveys that are commonly used in educational research, by providing
“analytics and information that goes beyond traditional quantitative and qualitative educational research
approaches” (Jacobson, 2015, p. 310). Agent based models have increasingly been adopted in diffusion
research “as intuition aids that facilitate theory-building and as tools to analyze real world scenarios, support
management decisions and obtain policy recommendations" (Kiesling et al., 2012, p. 1). In a similar way,
the MMM tool developed by Levin (2015) is applied in this study to build on emerging theories of DoI and
complexity (Rogers et al., 2005). The MMM framework in this study has been modified for modelling real
Page | 5
and possible scenarios of sustainable diffusion of e-learning innovations to inform university change
management strategies.
Levin (2015) describes the functions and features of the MMM computer simulation tool as providing a
framework in which
the concepts in the domain being modeled are represented by labeled circles, each of which
has an activity level that is partially determined by impact from other concepts within the
model and partially determined by outside context, represented by globe icons … the activity
level of each concept and context node is indicated visually by its size (Levin, 2015, p. 3).
In this study, the creation of connections between these MMM nodes occurs during interviews with
volunteer participants who apply scenarios from their experience of technology adoption in their university
to the model. Running the model, once these connections are made, allows an interpretation of changes to
the size of the factors and levels of influence associated with different actor roles. A completed model that
illustrates this effect is shown in Figure 1. This example is from one of five pilot interviews for this study
conducted in 2016. The findings from these interviews will not contribute to the final research data collected
for this study which received ethics approval following the pilot phase.
Figure 1. A Multi-Mediator Model showing connected actors, factors and levels of influence
In Figure 1, the concepts being modelled are shown as labelled dots that represent critical success factors
in the sustainable diffusion of an e-learning innovations. Please note that the factors in the model appear
the same size before the connections between them are made and the model is run. The factors that appear
in the model in Figure 1 were drawn from a preliminary analysis of 15 Australasian university case studies
conducted by Gunn and Herrick (2012). Each factor relates to one of four actors in the model, defined by
the quadrants. The actor labels for each quadrant represent two levels within the organisational structure of
a university: the individual level (micro innovators and micro adopters) in the lower two quadrants and
institutional level (macro management roles and meso professional and/or technical support services) in the
upper two quadrants. Each actor and factor in the model plays a role in the complex process of technology
Page | 6
adoption. These factors and actors provide the starting point for completing each model. During an
interview the factors in the model are connected and levels of influence are applied to reflect a scenario
from a lived experience of technology adoption provided by the interview participant. The factors
connected by arrows indicate the direction of enabling interactions in the scenario and the factors connected
by barred lines indicate inhibiting interactions. The levels of influence of specific factors within an
interviewee’s scenario are indicated by attaching and adjusting the size of the globe icons before the model
is run. The levels of influence act as the context nodes in the model. Figure 1 shows the resulting size of
the factors and associated levels of influence after the model is run. The relative size of factors and levels
of influence indicates their activity level in the inter-relationships between actors: the greater the size the
more a factor and influence plays a role in the scenario.
The process of creating the model framework shown in Figure 1, started with the researcher’s analysis of
themes in published reports of case studies. The outcome of this analysis was then applied to create the
common framework in the model used at the start of each interview which showed the factors as even sized
dots within each of the actor quadrants. As Levin and Datnow (2012) suggest, this first step in the data
modelling process is useful for drawing “themes out of qualitative case studies of educational change” (p.
199). In their own research, Levin and Datnow (2012) applied data from a case study using a prototype of
the MMM tool to demonstrate that case studies can inform the development of dynamic models of complex
interactions in educational change processes. During the interviews conducted for the pilot phase of this
study, the placement of some of the factors in the initial model were changed by interviewees as they
directed the development of the model to reflect scenarios from their own experience and explored
alternative possibilities. In some cases, additional factors were also added as the modelling process
unfolded.
Viewed on their own, ABM models can appear simplistic and limited in interpreting social complexity
(Tubaro & Casilli, 2010). However, when used as part of an interactive interpretive interview process, these
models can provide a dynamic diagrammatic representation that acts as a graphic elicitation stimulus for
gaining deeper insights about complex systems and problems. The advantages of using this graphic
elicitation method are described by Crilly, Blackwell and Clarkson (2006) as follows:
Diagrams are effective instruments of thought and a valuable tool in conveying those
thoughts to others. As such, they can be usefully employed as representations of a research
domain and act as stimulus materials in interviews. This process of graphic elicitation may
encourage contributions from interviewees that are difficult to obtain by other means (Crilly
et al., 2006, p. 3).
Jacobson (2015) suggests that using an agent-based computer simulation modelling tool like MMM,
provides a "simplicity-complexity epistemic view" (p. 311) of complex systems that leads to insights based
on simple rules rather than producing complex causal explanations. In this study, the insights revealed by
interviewees, after modelling a scenario from their own lived experience, are applied in exploring and
testing possible alternative connections in the model to depict an ideal scenario of bottom-up adoption of
e-learning innovations in their university. Used in this way, MMM becomes “an effective tool for
discovering surprising consequences of simple assumptions” (Axelrod, 2005, p. 1). In the pilots conducted
for this study many surprising insights were revealed by interviewees about inter-relationships between the
roles of university actors in the adoption of e-learning innovations. These revelations occurred throughout
the modelling process which provided a dynamic “helicopter” view of the university system as “a whole
which is more than the sum of its parts" (Tubaro & Casilli, 2010, p. 61), a popular catch phrase, attributed
to Aristotle, in describing social complexity. During interviews for the pilot study, the consequences of
changing the relationships in the model could be seen immediately as new enabling and inhibiting
connections between the factors were applied and levels of influence were adjusted.
An interpretive complexity research design
The research design used in this study follows five phases in the interpretive interactionism methodology
proposed by Denzin (2001): deconstruction, capture, bracketing, construction and contextualisation. The
first four of these phases informs the development of the model and the final phase locates the context for
the model in the lived experiences of the interview participants. These phases also reflect the five guiding
questions in this study. This approach brings an interpretive complexity perspective to modelling and
investigating the phenomenon of sustainable diffusion of e-learning innovations. In the deconstruction
Page | 7
phase, prior conceptions of the phenomenon are examined in the research literature to identify the system
elements (the actors). The capture phase identifies the process elements (the factors) in the phenomenon
which are drawn from extant case studies. The bracketing phase reduces the system and process elements
(the actors and factors) to uncover “essential structures and features” (Denzin, 2001, p. 70). In the
construction phase the essential elements (the critical success factors and four system actors) are brought
together to build the model framework. The contextualisation phase relocates the “the phenomenon back
in the natural social world” (Denzin, 2001, p. 70). This last phase occurs during individual interviews with
participants who have a current or recent experience of the phenomenon in their own university and who
represent the roles of actors in the model.
The four steps in this contextualisation phase are
1. Obtaining and presenting personal experience stories and self-stories that embody, in full
detail, the essential features of the phenomenon as constituted in the bracketing and
construction phases
2. Presenting contrasting stories that will illuminate variations on the stages and forms of
the process
3. Indicating how lived experiences alter and shape the essential features of the process
4. Comparing and synthesizing the main themes of these stories so that their differences
may be brought together into a reformulated statement of the process
(Denzin, 2011, p. 79).
The interpretive complexity research design developed for this study challenges traditional research
methods found in previous studies that have adopted a linear view of scaling up e-learning innovations.
Conclusions
The outcomes of the pilot study reported in this paper, demonstrate that the development and interrogation
of a computer simulation that models the complexity of technology adoption in higher education teaching
practice can be used to reveal actionable insights for informing university change management and teaching
strategies. In seeking to elicit actionable insights through modelling the complexity of technology adoption,
this study applies a research design that looks beyond the purely linear and causal explanations found in
previous quantitative and qualitative studies. While agent-based modelling is a proven methodology that
has been used in other studies of the diffusion of innovations, its application as an interactive visual artefact
for eliciting data collection during interviews is a new approach in educational research. The data gathered
in this pilot study demonstrate the value of agent-based modelling in researching the sustainable diffusion
of e-learning innovations. As well as leading to actionable insights, this new approach also has the potential
for further applications in investigating complex problems and systems in other areas of educational and
social research.
This is the first study to apply an agent-based modelling methodology to examine technology adoption
processes within a university from a whole system perspective that presents a view that is more than the
sum of its parts. This is in contrast to previous studies that have drawn primarily on data from individual
case studies to identify lists of common success factors and barriers to the sustainable diffusion of e-learning
innovations. The application in this study of agent-based modelling through an interpretive interactive
process offers a new way of exploring and understanding the increasingly dynamic, complex and changing
demands faced by universities in adopting new technologies in teaching and learning. By applying this
method during interviews to examine the relationships between key stakeholders and critical success factors
in e-learning adoption from an interpretive complexity perspective, this study aims to contribute towards
furthering the application of agent-based modelling in organisational transformation. At a practical level,
the research methods applied in this study are also aimed at informing educational change management
policies, strategies and processes that can support the wider adoption of proven innovations in teaching and
learning with digital technologies and move university teaching practice into the 21st century.
References
Adams Becker, S., Cummins, M., Davis, A., Freeman, A., Hall Giesinger, C., & Ananthanarayanan, V.
(2017). NMC horizon report: 2017 higher education edition. Austin, Texas: The New Media
Consortium.
Page | 8
Alexander, S. (2006). Dissemination of innovations: A case study. In L. Markauskaite, P. Goodyear & P.
Reimann (Eds.), Who’s learning? Whose technology? Proceedings of ASCILITE Sydney 2006 (pp.
21-31). Retrieved from http://www.ascilite.org/conferences/sydney06/proceeding/pdf_papers/vol2.pdf
Axelrod, R. (2005). Advancing the art of simulation in the social sciences. In J.-P. Rennard (Ed.),
Handbook of Research on Nature Inspired Computing for Economy and Management (pp.1-13).
Hershey, PA: Idea Group.
Bates, A., & Sangrà, A. (2011). Managing technology in higher education: Strategies for transforming
and learning. San Francisco: Jossey-Bass/John Wiley & Co.
Birch, D. & Burnett, B. (2009). Bringing academics on board: Encouraging institution-wide diffusion of
elearning environments. Australasian Journal of Educational Technology, 25(1), 117-134.
Brennan, J., Broek, S., Durazzi, N., Kamphuis, B., Ranga, M., & Ryan, S. (2014). Study on innovation in
higher education: Final report. European Commission Directorate for Education and Training Study
on Innovation in Higher Education. Luxembourg: Publications Office of the European Union.
Bui, Q. (2015). A review of innovation diffusion theories and mechanisms. Proceedings of DIGIT 2015.
Retrieved from http://aisel.aisnet.org/digit2015/11/
Casanovas, I. (2010). Exploring the current theoretical background about adoption until
institutionalization of online education in universities: Needs for further research. Electronic Journal
of e-Learning, 8(2), 73-84.
Castellani, B. (2014). Modeling complex systems: A case based approach [PowerPoint slides]. Retrieved
from http://blogs.cim.warwick.ac.uk/complexity/wp-content/uploads/sites/11/2014/05/Castellani.pdf
Cooper, A. (2012). What is Analytics? Definition and Essential Characteristics. CETIS Analytics Series,
1(5). Retrieved from http://publications.cetis.org.uk/2012/521
Crilly, N., Blackwell, A. F., & Clarkson, P. J. (2006). Graphic elicitation: Using research diagrams as
interview stimuli. Qualitative Research, 6(3), 341-366.
Csete, J. & Evans, J. (2013). Strategies for impact: Enabling e-learning project initiatives. Campus-Wide
Information Systems, 30(3), 165-173.
Denzin, N. K. (2001). Interpretive interactionism. (2nd ed.). Thousand Oaks, CA: SAGE Publications.
Elgort, I. (2005). E-learning adoption: Bridging the chasm. In Balance, Fidelity, Mobility: Maintaining
the Momentum? Proceedings of ASCILITE Brisbane 2005 (pp. 181-185). Retrieved from
http://www.ascilite.org/conferences/brisbane05/blogs/proceedings/20_Elgort.pdf
Gunn, C. (2010). Sustainability factors for e-learning initiatives. ALT-J Research in Learning Technology,
18(2), 89-103.
Gunn, C. (2011). Sustaining elearning innovations. In Changing Demands, Changing Directions.
Proceedings of ASCILITE Hobart 2011 (pp.509-519). Retrieved from
http://www.ascilite.org/conferences/hobart11/downloads/papers/Gunn-full.pdf
Gunn, C., & Herrick, R. (2012). Sustaining eLearning innovations: A research study report. Auckland:
The University of Auckland Centre for Academic Development. Retrieved from
http://www.acode.edu.au/pluginfile.php/160/mod_resource/content/1/Sustaining_eLearning_Innovatio
ns_Report.pdf
Henderson, M. (2015). Wicked issues in educational technologies: What would happen if we stopped
pretending that technologies were solutions to teaching or learning? [PowerPoint slides]. Retrieved
from http://www.latrobe.edu.au/__data/assets/pdf_file/0007/644308/Michael-Henderson-Symposium-
Workshop-Presentation.pdf
Jacobson, M. J. (2015). Education as a complex system: Implications for educational research and policy.
In B. A. Furtado, P. A. M. Sakowski, & M. H. Tóvolli (Eds.), Modeling Complex Systems for Public
Policies (pp. 301-316). Brasilia, Brazil: IPEA.
Johnson, L., Adams Becker, S., Cummins, M., Estrada, V., Freeman, A., & Hall, C. (2016). NMC
Horizon Report: 2016 Higher Education Edition. Austin, Texas: The New Media Consortium.
Kiesling, E. Günther, M. Stummer, C., & Wakolbinger, L. M. (2012). Agent-based simulation of
innovation diffusion: A review. Central European Journal of Operations Research, 20(2), 183–230.
Laurillard, D., Oliver, M., Wasson, B., & Ulrich, H. (2009). Implementing technology-enhanced learning.
In N. Balacheff, S. Ludvigsen, T. de Jong, A. Lazonder, & S. Barnes (Eds.), Technology-enhanced
learning: Principles and products (pp. 289-306). Dordrecht: Springer Netherlands. doi: 10.1007/978-
1-4020-9827-7_17
Levin, J. A. (2015). Creating and evaluating multi-agent models of learning with qualitative and
quantitative data. San Diego: Department of Education Studies University of California, San Diego.
Retrieved from
https://docs.google.com/document/d/15mJloMh4sZ_CGnEYjMYHLjoTc6tXhbQyE75A4EqjlXc/edit?
pli=1
Page | 9
Levin, J. A., & Datnow, A. (2012). The principal role in data-driven decision making: Using case-study
data to develop multi-mediator models of educational reform. School Effectiveness and School
Improvement, 23(2), 179-201. doi:10.1080/09243453.2011.599
Levin, J. A., & Jacobson, M. J. (2017). Education as a complex system: Implications for educational
research and policy. Paper to be presented at the American Educational Research Association
Meeting. San Antonio, 27 April-1 May. Retrieved from
https://www.academia.edu/31085744/Education_as_a_Complex_System_Implications_for_Education
al_Research_and_Policy
Markus, M. L. (1987). Toward a critical mass theory of interactive media: Universal access,
interdependence and diffusion. Communications Research, 14(5), 491-511.
Nichols, M. (2008). Institutional perspectives: The challenges of e-learning diffusion. British Journal of
Educational Technology, 39(4), 598-609.
Nicholson, P. (2007). A history of e-learning: Echoes of the pioneers. In B. Fernandez-Manjon et al.
(Eds.), Computers and education: E-Learning from theory to practice (pp. 1-11). Dordrecht: Springer.
Nutley, S., Davies, H., & Walter, I. (2002). Learning from the diffusion of innovations. London: Research
Unit for Research Utilisation, University of St Andrews.
Pacansky-Brock, M. (2015, September 23). Straddling the chasm: Rethinking faculty support. Retrieved
from http://www.teachingwithoutwalls.com/2015/09/straddling-chasm-rethinking-faculty.html
Reid, P. (2014). Categories for barriers to adoption of instructional technologies. Education and
Information Technologies, 19, 383–407. doi:10.1007/s10639-012-9222-z
Roberston, I. (2008). Sustainable e-learning, activity theory and professional development. In Hello!
Where are you in the landscape of educational technology? Proceedings of ASCILITE Melbourne
2008 (pp. 819-826). Retrieved from
http://www.ascilite.org/conferences/melbourne08/procs/robertson.pdf
Rogers, E. M. (2003). Diffusion of innovations (5th ed.). New York: The Free Press.
Rogers, E. M., Medina, U. E., Rivera, M. A., & Wiley, C. J. (2005). Complex adaptive systems and the
diffusion of innovations. The Innovation Journal, 10(3).
Rossiter, D. E. (2006). Embedding e-Learning in universities: Analysis and conceptualisation of change
processes. PhD thesis, Queensland University of Technology. Retrieved from
http://eprints.qut.edu.au/16223/1/Darien_Rossiter_Thesis.pdf
Sabelli, N. H., & Harris, C. J. (2015). The role of innovation in scaling up educational innovations. In C.-
K. Looi & L. W. Teh (Eds.), Scaling educational innovations (pp. 13-30). Singapore: Springer.
Salmon, G., & Angood, R. (2013). Sleeping with the enemy. British Journal of Educational Technology,
44(6), 916-925. doi:10.1111/bjet.12097
Selwyn, N. (2013). Digital technologies in universities: Problems posing as solutions? Learning, Media
and Technology, 38(1), 1-3. doi:10.1080/17439884.2013.759965
Sharpe, R., Benfield, G. & Francis, R. (2006). Implementing a university e-learning strategy: Levers for
change within academic schools. The Association for Learning Technology Journal, 14(2), 135–152.
Singh, G., & Hardaker, G. (2014). Barriers and enablers to adoption and diffusion of eLearning: A
systematic review of the literature – a need for an integrative approach. Education and Training,
56(2/3), 105-121. Retrieved from http://dx.doi.org/10.1108/ET-11-2012-0123
Smigiel, H. M. (2013). Making Changes. In L. Morris (Ed.), Research in Educational Technology:
Development and Practice. Proceedings of ICICTE Crete 2013 (pp. 80-88). Retrieved from
http://www.icicte.org/Proceedings2013/Papers%202013/03-1-Smigiel.pdf
Snyder, S. (2013). The simple, the complicated and the complex: Educational reform through the lens of
complexity theory. OECD Education Working Papers, No 96, OECD Publishing.
Stepanyan, K., Littlejohn, A., & Margaryan, A. (2010). Sustainable e-learning in a changing landscape: A
scoping study (SELScope). Retrieved from
http://www.heacademy.ac.uk/assets/EvidenceNet/SELScope_Stepanyan_Littlejohn_Margaryan_FINA
L_2010.doc.
Stepanyan, K., Littlejohn, A., & Margaryan, A. (2013). Sustainable e-learning: Toward a coherent body
of knowledge. Educational Technology & Society, 16(2), 91–102.
Tubaro, P., & Casilli, A. A. (2010). An ethnographic seduction: How qualitative research and agent-based
models can benefit each other. Bulletin of Sociological Methodology, 106(1), 59-74.
U.S. Department of Education, Office of Educational Technology. (2017). Reimagining the role of
technology in higher education: A supplement to the national education technology plan. Washington,
D.C. Retrieved from http://tech.ed.gov
White, G. K. (2010). Diffusion of ICT in education and the role of collaboration: A study of EdNA. PhD
thesis, Curtin University. Retrieved from https://espace.curtin.edu.au/handle/20.500.11937/2559
