Content uploaded by Daan Kolkman
Author content
All content in this area was uploaded by Daan Kolkman on Feb 27, 2019
Content may be subject to copyright.
CHALLENGES TO DATA SCIENCE PROJECTS WITH SMES:AN
ANALYSIS AND DECISION SUPPORT TOOL
A PREPRINT
Daan A. Kolkman ∗
Jheronimus Academy of Data Science
Eindhoven University of Technology
’s-Hertogenbosch, the Netherlands
d.kolkman@tue.nl
Ruud Sneep
Jheronimus Academy of Data Science
Eindhoven University of Technology
’s-Hertogenbosch, the Netherlands
r.sneep@tue.nl
February 27, 2019
ABS TRAC T
Data science has quickly developed as an academic field and has sparked the imagination of public
and private sector alike. While considerable effort is devoted towards the technical development of
statistical approaches for analysing the wealth of available data, our understanding of data science in
practical contexts has lagged behind. In this paper we build on 40 applied data science projects that
were conducted with small to medium enterprises (SME’s) in the Netherlands to identify common
pitfalls and challenges. This analysis informs the development of a "data science project canvas"; a
tool that helps people with a non-technical background to define a data science project.
Keywords Entrepreneurship ·Data science ·Decision making support
1 Data Science in practice
The ongoing increase of computational power allows for the development of ever more sophisticated data analysis tech-
niques, models, and algorithms (Venturini, Jensen, & Latour, 2015). This broad collection of data-centric innovations
is encompassed by the field of ’data science’
(
Hey & Tolle, 2009). Data science has quickly proliferated beyond the
academic domain; industry and government have also taken an interest in the field
(
Cukier & Mayer-Schoenberger,
2013).
The perceived advantages of data science are numerous
(
Floridi & Taddeo, 2016) and the benefits of using quantified
information in general have been the subject of inquiry for decades
(
van Daalen & Janssen, 2002). Yet, the prevalence of
models and algorithms - which are the outcomes of data science - falls short of its potential
(
Inman, 2011). Explanations
for this "application gap" are numerous
(
Kolkman, Campo, Balke-Visser, & Gilbert, 2016). Diez and Mcintosh (2011)
suggests it is caused by a lack of understanding between those that develop algorithms and those that use them, Delden
(2011) argues that the technical capabilities of algorithms and the software they are embedded in are not flexible enough,
Happe and Ballman (2008) point out that algorithms need to fit within the day tot day routines of the intended user-base.
Despite decades of research, we know very little about why some quantifications are used and others are not
(
Syme,
Bennet, Macpherson, & Thomas, 2011).
This lack of understanding is particularly surprising considering the large investments in data science
(
Bulger, Taylor,
& Schroeder, 2014), the modest success rate of data science projects
(
Cavanillas, Curry, & Wahlster, 2016), and
several examples of analytics use gone wrong
(
Sluijs, 2002; Barocas & Selbst, 2016). Authors agree on the potentially
beneficial effects of data science, yet only some offer guidance on how these effects can be brought about in practice.
Such guidance is long overdue, particularly because small to medium sized businesses risk falling behind. Their
adoption of data science has been slow and progress is hampered by a lack of understanding. Typical questions asked
∗Corresponding author.
APREPRINT - FE BRUARY 27, 2019
by Small to Medium sized Enterprises (SMEs) include: "What data do I have?", "How can I use my data to create
value?", and ’Where do I start?" (van der Veen, van der Born, Smetsers, & Bosma, 2017).
This paper contributes to the debate on the application gap. It reports on our experience with running data science
projects in practice. We build on 40 cases in which data science was applied to assist an SME in solving some business
challenge or question. Through analysis of the project outcomes, our notes, and two focus groups we identified several
challenges and difficulties which can prevent implementation of data science in practice. We discuss these challenges
and present the Data project canvas; a decision making tool which helps SMEs (1) to identify a problem that can be
solved through data science and (2) design a data science project which outcome they will adopt and use.
2 The JADS SME Datalab and data science projects
The Jheronimus Academy of Data Science (JADS) SME Datalab was founded 2018 with the purpose of helping Small to
Medium Enterprises to become more data-literate. The Datalab connects postgraduate students to local SMEs to deliver
data projects. SMEs pay a modest fee to cover a stipend for the student and operating expenses for the Datalab. Each
project runs for six to ten weeks, with the students investing about 80 hours per project. The students are supervised by
experienced data scientists to safeguard analytic rigour, oversee communication to the client, and ensure timely delivery
of the project. SMEs go through one or two intake meetings, during which an experienced data scientist helps them to
identify and delineate a project which adds value to the business. This project is defined in a project proposal, which
lists the required data, deliverables, and goals of the client. The proposal is signed by the client and is used to evaluate
whether a project has been completed. Students fill out an online planning tool with information about the activities
and steps necessary to complete the project. They log their progress, which permits their supervisors to keep track of
the projects. The students can raise and discuss progress online, during face-to-face supervision meetings, or during
informal sessions in the SME Datalab.
Over the course over a year we completed a total of 44 projects with 30 businesses. The variety in terms of the
industrial sector of the companies was considerable. We worked with over-the-counter businesses such as a bakery
and a hairdresser, but also with sewer maintenance engineers and a lettuce-grower. In terms of the technical scope,
the projects were similarly diverse. The projects ranged from a customer segmentation for marketing purposes to the
design of a dimension reduction algorithm to benchmark the traffic-safety status of municipalities. The heterogeneity of
this sample provides a strong foundation for theorization
(
Patton, 2002), in our case to identify data science project
challenges that occur irrespective of industry or technical scope. However, the businesses were not selected on the
basis of some sampling scheme. The entrepreneurs we engaged with where of the enthusiastic sort. They were not a
representative sample of the population. Rather, they would probably classify as "innovators" or "front-runners".
3 Common issues
We collected data in the form of project outcomes, field notes, notes from two focus groups, and emails from students
or clients. We went over our material per project and for each project listed: the challenges we encountered, the way we
solved those challenges, whether or not the agreed upon deliverables were completed, and circumstances that were
conductive to the completion of the project. We then collaboratively open-coded
(
Straus & Corbin, 1998) this data and
selected those challenges which we could have mitigated before the start of the project if we would have known about
them. We then grouped similar challenges, the list below is the result of this process:
3.1 Infrastructure
Although both the name of the SME Datalab and that of the field of data science contain the word "data" we found that
many entrepreneurs did not share our notion of what data is. The challenges contained within the infrastructure group
pertain to the availability of data, the quality of this data, and the ways in which this data can be accessed. We found
that it is important to ask the entrepreneurs to share a sample of their data early on. This forces them to try and export it
from whatever system it is in and permits the data scientist to evaluate the quality of the data.
3.1.1 Data
Some entrepreneurs came to us with elevated expectations. One business was hoping to implement a machine learning
algorithm to develop a predictive maintenance system. This system would help them identify those machines that were
most likely to break down next. With this information, they could use their resources by developing more efficient
maintenance schedules. When asked about the available data, the client sent over a couple of Excel spreadsheets which
did not contain the raw inputs. Rather, the spreadsheets contained aggregated report data about the machines. When
2
APREPRINT - FE BRUARY 27, 2019
asked about the underlying data for these reports, they were not sure. We proceeded by adjusting the project scope by
defining a project which would explore and identify the available data-sources.
3.1.2 Software
Even if the business has a large collection of data, this is no guarantee that this data can be accessed. In several projects,
we found that the data was stored within a proprietary software. It could not be exported without assistance from
the software’s developers. In most projects we were able to liaise with the software developer and get a data export.
However, software developers can be weary to allow third parties access to their databases, as they perceive a risk
that their software will be replaced. Although the data portability principle of the General Data Protection Regulation
(GDPR) states that data should be transferable from one system to the next, some software developers try to retain
clients by policing third party access to the data.
3.1.3 Expertise
The businesses we worked with had very different Information Technology (IT) competence levels. Some businesses
already had some experience with Business Intelligence applications such as dashboard and visualisations, whereas
others had just digitized their financial administration. It is important to get an idea of the data maturity of the client
before the project. This is instrumental towards designing a realistic project planning. In addition, if an organisation
data maturity is higher a project can be more technically advanced. Organisations that have a higher data maturity are
typically more proficient in integrating the outcomes of a data science project within their existing infrastructure.
3.1.4 Partners
As mentioned, the use of proprietary software can be a challenge towards the timely completion of a data science
project. A similar challenge is introduced by business that work with one or more partners for their data management or
data collection. The more external stakeholders that are involved, the higher the complexity of the data science project.
In projects were many stakeholders were involved, we lost much time to project management tasks.
3.2 Preconditions
Data science is by nature a quite technical field, the algorithms and models that can be developed as part of a data
science project can be hard to comprehend for experts, let alone for those not trained in their use. The challenges
contained in the preconditions group mostly pertain to the human an regulatory side of data science. It is important to
keep in mind that data science projects should contribute to making someone’s job easier.
3.2.1 Commitment
For some of the projects we completed, the business owner was looking to demonstrate the potential of data science
to others in his or her organisation. In these projects, it was clear to the client that project would not immediately
effectuate a change in the way the organisation works. In other projects, the business owners were looking to implement
a project where the owner himself or herself was not the intended user of the project. In such cases, we found it was
paramount to involve the users at an early stage to ensure the deliverables aligned with their routines.
3.2.2 Culture
It is not useful for a business to engage in a data science project if the results do not align well with the current
organizational culture or routines. In one of the projects we proposed to develop a model that would predict the product
demand and turnover for a bakery. In our initial proposal we aimed to implement the model in Python. After further
discussions with the business owner, we found that he was used to - and comfortable with - Microsoft Excel only. As
such we had to adapt our approach to make it fit within his weekly routine of forecasting demand and making personnel
schedules.
3.2.3 Regulations
The recent effectuation of the GDPR has put privacy and data ethics higher on the agenda. The business owners we
spoke with were typically aware of the new legislation, but did not feel confident about their level of understanding.
More generally, business owners seem to be have limited knowledge about what is or is not allowed in relaation to data
collectin. One business owner who was in the human resources business asked us if we could automate data-collection
from LinkedIn. We had to point out that it was not allowed to retrieve that information. In one project, the data we
3
APREPRINT - FE BRUARY 27, 2019
needed was accessible only through interaction with a partner of our client business. In this particular case, we lost a lot
of time in negotiating access and setting up a system for remote access to the data.
3.2.4 Budget
The SMEs we work with often have a limited budget, but great ambitions. We try to identify a project that adds value or
helps them to reduce costs.
3.3 Expectations
3.3.1 Challenges
Often, businesses approach us and have no notion of where to start. We ask them to describe their business and invite
them to tell us about what challenges they are currently facing.
3.3.2 Results
Some entrepreneurs struggled to incorporate the findings of the projects into their business. In one case we identified
postal codes with demographics that matched those of the current customer base of the business. The entrepreneur was
happy for us to have mapped his current client base. However, he was unsure what to do with the information on where
his potential clients live.
4 Canvas
The challenges outlined in the previous section formed the basis of a Data science project canvas. The canvas presented
here is still under development and is by no means intended as an exhaustive list of all factors which contribute to
successful implementation of data science in SMEs. Nonetheless, we believe that by asking the right questions before
the project start can contribute to data science projects that are used and ultimately add value to the business.
5 Discussion
This paper identified twelve challenges that can impair the progress of a data science project and can ultimately result in
its failure. We described the twelve challenges and presented a decision support tool which allows businesses to design
data science projects that overcome the challenges.
We conclude that by addressing these challenges before the projects start, their success rate will be higher. Future
research could consider if these challenges occur in other data science contexts as well. Several of the challenges we
identified align with previous research on the application gap.
Acknowledgements
We would like to thank the businesses that worked with SME Datalab and the partners that helped us to connect with
the SME community. We are grateful to Matthijs Bookelmann, Arjan van den Born and Bas Bosma for their comments.
Appendix
4
APREPRINT - FE BRUARY 27, 2019
Figure 1: The original Dutch version of the Data project canvas.
References
Barocas, S., & Selbst, A. (2016). Big data’s disparate impact. California Law Review.
Bulger, M., Taylor, G., & Schroeder, R. (2014). Data-driven business models: Challenges and opportunities of big data.
Report by the Oxford Internet Institute.
Cavanillas, J. M., Curry, E., & Wahlster, W. (2016). New horizons for a data-driven economy. Springer, New York.
Cukier, K., & Mayer-Schoenberger, V. (2013). The rise of big data: How it’s changing the way we think about the
world. Foreign Affairs 92 (28).
Delden, H. v. (2011). A methodology for the design and development of integrated models for policy support.
Environmental Modelling Software 26(3).
Diez, E., & Mcintosh, B. (2011). Organisational drivers for, constraints on and impacts of decision and information
support tool use in desertification policy and management. Environmental Modelling Software 26(3).
Floridi, L., & Taddeo, M. (2016). What is data ethics? Philosophical Transactions of the Royal Society A: Mathematical,
Physical and Engineering Sciences 374(1).
Happe, K., & Ballman, A. (2008). Doing policy in the lab! options for the future use of modelbased policy analysis for
complex decision-making. Proceedings of the 107th EAAE Seminar.
Hey, T. S., T., & Tolle, K. (2009). The fourth paradigm: Data-intensive scientific discovery. Microsoft Research Paper.
Inman, D. (2011). Perceived effectiveness of environmental decision support systems in participatory planning:
Evidence from small groups of end-users. Environmental Modelling Software 26(1).
Kolkman, D. A., Campo, P., Balke-Visser, T., & Gilbert, N. (2016). How to build models for government: criteria
driving model acceptance in policymaking. Policy Sciences 49(4).
Patton, M. (2002). Qualitative evaluation and research methods. qualitative inquiry.
Sluijs, J. v. d. (2002). (2002). a way out of the credibility crisis of models used in integrated assessment. Futures 34(1).
Straus, A., & Corbin, J. (1998). Techniques and procedures for developing grounded theory.
Syme, G., Bennet, D., Macpherson, D., & Thomas, J. (2011). Guidelines for policy modellers–30 years on: New tricks
or old dogs? Proceedings of the International Congress on Modelling and Simulation.
van Daalen, L., C.and Dresen, & Janssen, M. A. (2002). The roles of computer models in the environmental policy life
cycle. Environmental Science Policy, 238(1).
van der Veen, M., van der Born, J., Smetsers, D., & Bosma, B. (2017). Ondernemen met (big) data door het mkb.
Venturini, T., Jensen, P., & Latour, B. (2015). Fill in the gap. a new alliance for social and natural sciences. Journal of
Artificial Societies and Social Simulation 18 (2).
5
