ChapterPDF Available

Abstract and Figures

This chapter provides an overview of big data storage technologies. It is the result of a survey of the current state of the art in data storage technologies in order to create a cross-sectorial technology roadmap. This chapter provides a concise overview of big data storage systems that are capable of dealing with high velocity, high volumes, and high varieties of data. It describes distributed file systems, NoSQL databases, graph databases, and NewSQL databases. The chapter investigates the challenge of storing data in a secure and privacy-preserving way. The social and economic impact of big data storage technologies is described, open research challenges highlighted, and three selected case studies are provided from the health, finance, and energy sector. Some of the key insights on big data storage are (1) in-memory databases and columnar databases typically outperform traditional relational database systems, (2) the major technical barrier to widespread up-take of big data storage solutions are missing standards, and (3) there is a need to address open research challenges related to the scalability and performance of graph databases.
Content may be subject to copyright.
Chapter 7
Big Data Storage
Martin Strohbach, J
org Daubert, Herman Ravkin, and Mario Lischka
7.1 Introduction
This chapter provides an overview of big data storage technologies which served as
an input towards the creation of a cross-sectorial roadmap for the development of
big data technologies in a range of high-impact application domains. Rather than
elaborating on concrete individual technologies, this chapter provides a broad
overview of data storage technologies so that the reader may get a high level
understanding about the capabilities of individual technologies and areas that
require further research. Consequently, the social and economic impacts are
described, and selected case studies illustrating the use of big data storage technol-
ogies are provided. The full results of the analysis on big data storage can be found
in Curry et al. (2014).
The position of big data storage within the overall big data value chain can be
seen in Fig. 7.1. Big data storage is concerned with storing and managing data in a
scalable way, satisfying the needs of applications that require access to the data.
The ideal big data storage system would allow storage of a virtually unlimited
amount of data, cope both with high rates of random write and read access, flexibly
and efficiently deal with a range of different data models, support both structured
and unstructured data, and for privacy reasons, only work on encr ypted data.
M. Strohbach (*) J. Daubert
AGT International, Hilpertstr, 35, 64295 Darmstadt, Germany
H. Ravkin
Department of Industrial Engineering, Tel-Aviv University, Ramat-Aviv, Tel-Aviv 69978,
M. Lischka
acentrix GmbH, Erika-Mann-Strasse 69, 80636 Munich, Germany
© The Author(s) 2016
J.M. Cavanillas et al. (eds.), New Horizons for a Data-Driven Economy ,
DOI 10.1007/978-3-319-21569-3_7
Obviously, all these needs cannot be fully satisfied. But over recent years many new
storage systems have emerged that at least partly address these challenges.
This chapter provides an overview of big data storage technologies and identifies
some areas where further research is required. Big data storage technologies are
referred to as storage technologies that in some way specifically address the
volume, velocity, or variety challenge and do not fall in the category of relational
database systems. This does not mean that relational database systems do not
address these challenges, but alternative storage technologies such as columnar
stores and clever combinations of different storage systems, e.g. using the Hadoop
Distributed File System (HDFS), are often more efficient and less expensive (Marz
and Warren 2014).
Big data storage systems typically address the volume challenge by making use
of distributed, shared nothing architectures. This allows addressing increased stor-
age requirements by scaling out to new nodes providing computatio nal power and
storage. New machines can seamlessly be added to a storage cluster and the storage
system takes care of distributing the data between individual nodes transparently.
Storage solutions also need to cope with the velocity and variety of data.
Velocity is important in the sense of query latencies, i.e. how long does it take to
get a reply for a query? This is particularly important in the face of high rates of
incoming data. For instance, random write access to a database can slow down
query performance considerably if it needs to provide transactional guarante es. In
contrast, variety relates to the level of effort that is required to integrate and work
with data that originates from a large number of different sources. For instance,
graph databases are suitable storage systems to address these challenges.
Structured data
Data streams
Stream mining
Linked Data
Data discovery
‘Whole world’
Community data
data analysis
Data Quality
Trust / Provenance
Data validation
Top-do w n / B o t t o m-
Community /
Curation at scale
In-Memory DBs
Cloud storage
Query Interfaces
Scalability and
Data Models
Security and
Decision support
In-use analytics
Big Data Value Chain
Fig. 7.1 Data storage in the big data value chain
See for instance the map of 451 Research available at
120 M. Strohbach et al.
Section 7.2 summarizes key insights and Sect. 7.3 illustrates the social and
economic impact of data storage. Section 7.4 presents the current state-of-the-art
including storage technologies and solutions for security and privacy. Section 7.5
includes future requirements and emerging trends for data storage that will play an
important role for unlocking the value hidden in large datasets. Section 7.6 presents
three selected case studies, and the chapter is concluded in Sect. 7.7.
7.2 Key Insights for Big Data Storage
As a result of the analysis of current and future data storage technologies, a number
of insights were gained relating to data storage technologies. It became apparent
that big data storage has become a commodity business and that scalable storage
technologies have reached an enterpr ise-grade level that can manage virtually
unbounded volumes of data. Evidence is provided by the widespread use of
Hadoop-based solutions offered by vendors such as Cloudera (2014a),
Hortonworks (2014), and MapR (2014) as well as various NoSQL
vendors, in particular those that use in-memory and columnar storage technologies.
Compared to traditional relational database management systems that rely on
row-based storage and expensive caching strategies, these novel big data storage
technologies offer better scalability at lower operational complexity and costs.
Despite these advances that improve the performance, scalability, and usability
of storage technologies, there is still significant untapped potential for big data
storage technologies, both for using and further developing the technologies:
Potential to Transform Society and Businesses across Sectors: Big data
storage technologies are a key enabler for advanced analytics that have the
potential to transform society and the way key busi ness decisions are made.
This is of particular importance in traditionally non-IT-based sectors such as
energy. While these sectors face non-technical issues such as the lack of skilled
big data experts and regulatory barriers, novel data storage technologies have the
potential to enable new value-generating analytics in and across various indus-
trial sectors.
Lack of Standards Is a Major Barrier: The history of NoSQL is based on
solving specific technological challenges which lead to a range of different
storage technologies. The large range of choices coupled with the lack of
standards for querying the data makes it harder to exchange data stores as it
may tie application specific code to a certain storage solution.
Open Scalability Challenges in Graph-Based Data Stores: Processing data
based on graph data structures is beneficial in an increasing amount of applica-
tions. It allows better capture of semantics and complex relationships with other
NoSQL is typically referred to as “Not only SQL”.
7 Big Data Storage 121
pieces of information coming from a large variety of different data sources, and
has the potential to improve the overall value that can be generated by analysing
the data. While graph databases are increasingly used for this purpose, it remains
hard to efficiently distribute graph-based data structure across computing nodes.
Privacy and Security Is Lagging Behind: Although there are several projects
and solutions that address privacy and security, the protection of individual s and
securing their data lags behind the technological advances of data storage
systems. Considerable research is required to better understand how data can
be misused, how it needs to be protected and integrated in big data storage
7.3 Social and Economic Impact of Big Data Storage
As emerging big data technologies and their use in different sectors show, the
capability to store, manage, and analyse large amounts of heterogeneous data hints
towards the emergence of a data-driven society and economy with huge transfor-
mational potential (Manyika et al. 2011). Enterprises can now store and analyse
more data at a lower cost while at the same time enhancing their analytical
capabilities. While companies such as Google, Twitter, and Facebook are
established play ers for which data constitutes the key asset, other sectors also
tend to become more data driven. For instance, the health sector is an excellent
example that illustrates how society can expec t better health services by better
integration and analysis of health-related data (iQuartic 2014).
Many other sectors are heavily impacted by the maturity and cost-effectiveness
of technologies that are able to handle big datasets. For instance, in the media sector
the analysis of social media has the potential to transform journalism by summa-
rizing news created by a large amount of indi viduals. In the transport sector, the
consolidated data management integration of transport systems has the potential to
enable personalized multimodal transportation, increasing the experience of trav-
ellers within a city and at the same time helping decision-makers to better manage
urban traffic. In all of these areas, NoSQL storage technologies prove a key enabler
to efficiently analyse large amounts of data and create additional business value.
On a cross-sectorial level, the move towards a data-dr iven economy can be seen
by the emergence of data platforms such as (Gislason 2013),, and open data initiatives of the European Union such as open- and other national portals (e.g. ,,
(Ahmadi Zeleti et al. 2014). Technology vendors are supporting the move towards a
data-driven economy as can be seen by the positioning of their products and
services. For instance, Cloudera is offering a product called the enterprise data
hub (Cloudera 2014b), an extended Hadoop ecosystem that is positioned as a data
management and analysis integration point for the whole company.
Further to the benefits described above, there are also threats to big data storage
technologies that must be addressed to avoid any negative impact. This relates for
122 M. Strohbach et al.
instance to the challenge of protecting the data of individuals and reducing the
energy consumption of data centres (Koomey 2008).
7.4 Big Data Storage State-of-the-Art
This section provides an overview of the current state-of-the-art in big data storage
technologies. Section 7.4.1 describes the storage technologies, and Sect. 7.4.2 pre-
sents technologies related to secure and privacy-preserving data storage.
7.4.1 Data Storage Technologies
During the last decade, the need to deal with the data explo sion (Turner et al. 2014)
and the hardware shift from scale-up to scale-out approaches led to an explosion of
new big data storage systems that shifted away from traditional relational database
models. These approaches typically sacrifice properties such as data consistency in
order to maintain fast query responses with increasing amounts of data. Big data
stores are used in similar ways as traditional relational database management
systems, e.g. for online transactional processing (OLTP) solutions and data ware-
houses over structured or semi-structured data. Particular strengths are in handling
unstructured and semi-structured data at large scale.
This section assesses the current state-of-the-art in data store technologies that
are capable of handling large amo unts of data, and identifies data store related
trends. Following are differing types of storage systems:
Distributed File Systems: File systems such as the Hadoop File System (HDFS)
(Shvachko et al. 2010) offer the capability to store large amounts of unstructured
data in a reliable way on commodity hardware. Although there are file systems
with better performance, HDFS is an integral part of the Hadoop framework
(White 2012) and has already reached the level of a de-facto standard. It has
been designed for large data files and is well suited for quickly ingesting data and
bulk processing.
NoSQL Databases: Probably the most important family of big data storage
technologies are NoSQL database management system s. NoSQL databases use
data models from outside the relational world that do not necessarily adhere to
the transactional properties of atomicity, consistency, isolation, and durability
NewSQL Databases: A modern form of relational databases that aim for
comparable scalability as NoSQL databases while maintaining the transactional
guarantees made by traditional database systems.
Big Data Querying Platforms: Technologies that provide query facades in
front of big data stores such as distributed file systems or NoSQL databases. The
7 Big Data Storage 123
main concern is providing a high-level interface, e.g. via SQL
like query
languages and achieving low query latencies. NoSQL Databases
NoSQL databases are designed for scalability, often by sacrificing consistency.
Compared to relational databases, they often use low-level, non-standardized query
interfaces, which make them more difficult to integrate in existing applications that
expect an SQL interface. The lack of standard interfaces makes it harder to switch
vendors. NoSQL databases can be distinguished by the data models they use.
Key-Value Stores: Key-value stores allow storage of data in a schema-less way.
Data objects can be completely unstructured or structure d and are accessed by a
single key. As no schema is used, it is not even necessary that data objects share
the same structure.
Columnar Stores: According to Wikipedia “A column-oriented DBMS is a
database management system (DBMS) that stores data tables as sections of
columns of data rather than as rows of data, like most relational DBMSs”
(Wikipedia 2013). Such databases are typically sparse, distributed, and persis-
tent multi-dimensional sorted maps in which data is indexed by a triple of a row
key, column key, and a timestamp. The value is represented as an uninterrupted
string data type. Data is accessed by column families, i.e. a set of related column
keys that effectively compress the sparse data in the columns. Colum n families
are created before data can be stored and their number is expected to be small. In
contrast, the number of columns is unlimited. In principle columnar stores are
less suitable when all columns need to be accessed. However in practice this is
rarely the case, leading to superior performance of columnar stores.
Document Databases: In contrast to the values in a key-value store, documents
are structured. However, there is no requirement for a common schema that all
documents must adhere to as in the case for records in relational databases. Thus
document databases are referred to as storing semi-structured data. Similar to
key-value stores, docum ents can be queried using a unique key. However, it is
possible to access documents by querying their internal structure, such as
requesting all documents that contain a field with a specified value. The capa-
bility of the query interface is typically dependent on the encoding format used
by the databases. Common encodings include XML or JSON.
Graph Databases: Graph databases, such as Neo4J (2015), store data in graph
structures making them suitable for storing highly associative data such as social
network graphs. A particular flavour of graph databases are triple stores such as
AllegroGraph (Franz 2015) and Virtuoso (Erling 2009) that are specifically
Here and throughout this chapter SQL refers to the Standard Query Language as defined in the
ISO/IEC Standard 9075-1:2011.
124 M. Strohbach et al.
designed to store RDF triples. However, existing triple store technologies are not
yet suitable for storing truly large datasets efficiently.
While in general NoSQL data stores scal e better than relational databases,
scalability decreases with increased complexity of the data model used by the
data store. This particularly applies to graph databases that support applications
that are both write and read intensive. One approach to optimize read access is to
partition the graph into sub-graph s that are minimally connected between each
other and to distribute these sub-graphs between computational nodes. However, as
new edges are added to a graph the connectivity between sub-graphs may increase
considerably. This may lead to higher query latencies due to increased networks
traffic and non-local computations. Efficient sharding schemes must therefore
carefully consider the overhead required for dynamically re-distributing graph data. NewSQL Databases
NewSQL databases are a modern form of relational data bases that aim for compa-
rable scalability with NoSQL databases while maintaining the transactional guar-
antees made by traditional database systems. According to Venkatesh and Nirmala
(2012) they have the following characteristics:
SQL is the primary mechanism for application interaction
ACID support for transactions
A non-locking concurrency control mechanism
An architecture providing much higher per-node performance
A scale-out, shared-nothing architecture, capable of running on a large number
of nodes without suffering bottlenecks
The expectation is that NewSQL systems are about 50 times faster than tradi-
tional OLTP RDBMS. For example, VoltDB (2014) scales linearly in the case of
non-complex (single-partition) queries and provides ACID support. It scales for
dozens of nodes where each node is restricted to the size of the main memory. Big Data Query Platforms
Big data query platforms provide query facades on top of underlying big data stores
that simplify querying the underlying data stores. They typically offer an SQL-like
query interface for accessing the data, but differ in their approach and performance.
Hive (Thusoo et al. 2009) provides an abstraction on top of the Hadoop Distrib-
uted File System (HDFS) that allows structured files to be queried by an SQL-like
query language. Hive executes the queries by translating queries in MapReduce
jobs. As a consequence, Hive queries have a high latency even for small datasets.
Benefits of Hive include the SQL- like query interface and the flexibility to evolve
schemas easily. This is possibl e as the schema is stored independently from the data
7 Big Data Storage 125
and the data is only validated at query time. This appro ach is referred to as schema-
on-read compared to the schema-on-wri te approach of SQL databases. Changing
the schema is therefore a comparatively cheap operation. The Hadoop columnar
store HBase is also supported by Hive.
In contrast to Hive, Impala (Russel 2013) is designed for executing queries with
low latencies. It re-uses the same metadata and SQL- like user interface as Hive but
uses its own distributed query engine that can achieve lower latencies. It also
supports HDFS and HBase as underlying data stores.
Spark SQL (Shenker et al. 2013) is another low latency query fac¸ade that
supports the Hive interface. The project claims that “it can execute Hive QL queries
up to 100 times faster than Hive without any modification to the existing data or
queries” (Shenker et al. 2013). This is achieved by executing the queries using the
Spark framework (Zaharia et al. 2010) rather than Hadoops MapReduce
Finally, Drill is an open source implementation of Googles Dremel (Melnik
et al. 2002) that similar to Impala is designed as a scalable, interactive ad-hoc query
system for nested data. Drill provides its own SQL-like query language DrQL that
is compatible with Drem el, but is designed to support other query languages such as
the Mongo Query Language. In contrast to Hive and Impala, it supports a range of
schema-less data sources, such as HDFS, HBase, Cassandra, MongoDB, and SQL
databases. Cloud Storage
As cloud computi ng grows in popularity, its influence on big data grows as well.
While Amazon, Microsoft, and Google build on their own cloud platforms, other
companies including IBM, HP, Dell, Cisco, Rackspace, etc., build their proposal
around OpenStack, an open source platform for building cloud systems (OpenStack
According to IDC (Grady 2013), by 2020 40 % of the digital universe “will be
touched by cloud computing”, and “perhaps as much as 15 % will be maintained
in a cloud”.
Cloud in g eneral, and particularly cloud storage, can be used by both enterprises
and end users. For end users, storing their data in the cloud enables access from
everywhere and from every device in a reliable way. In addition, end users can use
cloud storage as a simple solution for online backup of their desktop data. Similarly
for enterprises, cloud storage provides flexible access from multiple locations and
quick and easy scale capacity (Grady 2013) as well as cheaper storage prices and
better support based on economies of scale (CloudDrive 2013) with cost effective-
ness especially high in an environment where enterprise storage needs are changing
over time up and down.
Technically cloud storage solutions can be distinguished between object and
block storage. Object storage “is a generic term that describes an approach to
addressing and manipulating discrete units of storage called objects” (Margaret
126 M. Strohbach et al.
Rouse 2014a). In contrast, block storage data is stored in volumes also referred to as
blocks. According to Margaret Rouse (2014b), “each block acts as an individual
hard drive” and enables random access to bits and pieces of data thus working well
with applications such as databases.
In addition to object and block storage, major platforms provide support for
relational and non-relational database-based storage as well as in-memory storage
and queue storage. In cloud storage, there are significant differences that need to be
taken into account in the application-planning phase:
As cloud storage is a service, applica tions using this storage have less control
and may experience decreased performance as a result of networking. These
performance differences need to be taken into account during design and imple-
mentation stages.
Security is one of the main concerns related to public clouds. As a result the
Amazon CTO predicts that in five years all data in the cloud will be encrypted by
default (Vogels 2013).
Feature rich clouds like AWS supports calibration of latency, redundancy, and
throughput levels for data access, thus allowing users to find the right trade-off
between cost and quality.
Another important issue when considering clou d storage is the supported con-
sistency mode l (and associated scalability, availability, partition tolerance, and
latency). While Amazons Simple Storage Service (S3) supports eventual consis-
tency, Microsoft Azure blob storage supports strong consistency and at the same
time high availability and partition tolerance. Microsoft uses two layers: (1) a
stream layer “which provides high availability in the face of network partitioning
and other failures”, and (2) a partition layer which “provides strong consistency
guarantees” (Calder et al. 2011).
7.4.2 Privacy and Security
Privacy and secur ity are well-recognized challenges in big data. The CSA Big Data
Working Group published a list of Top 10 Big Data Security and Privacy Chal-
lenges (Mora et al. 2012). The following are five of those challenges that are vitally
important for big data storage. Security Best Practices for Non-relational Data Stores
The security threats for NoSQL databases are similar to traditional RDBMS and
therefore the same best practices should be applied (Winder 2012). However, many
security measures that are implemented by default within traditional RDBMS are
missing in NoSQL databases (Okman et al. 2011). Such measures would include
7 Big Data Storage 127
encryption of sensitive data, sandboxing of processes, input validation, and strong
user authentication.
Some NoSQL suppliers recommend the use of databases in a trusted environ-
ment with no additional security or authentication measures in place. However, this
approach is hardly reasonable when moving big data storage to the cloud.
Security of NoSQL databases is getting more attention by security researchers
and hackers, and security will further improve as the market matures. For example,
there are initiativ es to provide access control capabilities for NoSQL databases
based on Kerberos authentication modules (Winder 2012). Secure Data Storage and Transaction Logs
Particular security challenges for data storage arise due to the distribution of data.
With auto-tiering, operators give away control of data storage to algorithms in order
to reduce costs. Data whereabouts, tier movements, and changes have to be
accounted for by transaction log.
Auto-tiering strategies have to be carefully designed to prevent sensitive data
being moved to less secure and thus cheaper tiers; monitoring and logging mech-
anisms should be in place in order to have a clear view on data storage and data
movement in auto-tiering solutions (Mora et al. 2012).
Proxy re-encryption schemes (Blaze et al. 2006) can be applied to multi-tier
storage and data sharing in order to ensure seamless confidentiality and authenticity
(Shucheng et al. 2010). However, performance has to be improved for big data
applications. Transaction logs for multi-tier operations systems are still missing. Cryptographically Enforced Access Control and Secure
Today, data is often stored unencrypted, and access control solely depends on a
gate-like enforcement. However, data should only be accessible by authorized
entities by the guarantees of cryptography—likewise in storage as well as in
transmission. For these purposes, new cryptographic mechanisms are require d
that provide the required functionalities in an efficient and scalable way.
While cloud storage providers are starting to offer encryption, cryptographic key
material should be generated and stored at the client and never handed over to the
cloud provider. Some products add this functionality to the application layer of big
data storage, e.g., zNcrypt, Protegrity Big Data Protection for Hadoop, and the Intel
Distribution for Apache Hadoop (now part of Cloudera).
Attribute-based encryption (Goyal et al. 2006) is a promising technology to
integrate cryptography with access control for big data storage (Kamara and Lauter
2010; Lee et al. 2013; Li et al. 2013).
128 M. Strohbach et al. Security and Privacy Challenges for Granular Access Control
Diversity of data is a major challenge due to equally diverse security requirements,
e.g., legal restrictions, privacy policies, and other corporate policies. Fine-grained
access control mechanisms are needed to assure compliance with these requirements.
Major big data components use Kerberos (Miller et al. 1987) in conjunction
with token-based authentication, and Access Control Lists (ACL) based upon users
and jobs. However, more fine-grained mechanism, for instance Attribu te-Based
Access Control (ABAC) and eXtensible Access Control Markup Language
(XACLM), are required to model the vast diversity of data origins and analytical
usages. Data Provenance
Integrity and history of data objects within value chains is crucial. Traditional
provenance governs mostly ownership and usage. With big data however, the
complexity of provenance metadata will increase (Glav ic 2014).
Initial efforts have been made to integrate provenance into the big data ecosys-
tem (Ikeda et al. 2011; Sherif et al. 2013); however, secure provenance requires
guarantees of integrity and confidentiality of provenance data in all forms of big
data storage and remains an open challenge. Furthermor e, the analysis of very large
provenance graphs is computationally intensive and requires fast algorithms. Privacy Challenges in Big Data Storage
Researchers have show n (Acquisti and Gross 2009) that big data analysis of
publicly available information can be exploited to guess the social security number
of a person. Some products selective ly encrypt data fields to create reversible
anonymity, depending on the access privileges.
Anonymizing and de-identifying data may be insufficient as the huge amount of
data may allow for re-identification. A roundtable disc ussion (Bollier and Firestone
2010) advocated transparency on the handling of data and algorithms as well as a
new deal on big data (Wu and Guo 2013) to empower the end user as the owner of
the data. Both options not only involve organization transparency, but also techni-
cal tooling such as Security & Privacy by Design and the results of the EEXCESS
EU FP7 project (Hasan et al. 2013).
7 Big Data Storage 129
7.5 Future Requirements and Emerging Paradigms for Big
Data Storage
This section provides an overview of future requirements and emerging trends.
7.5.1 Future Requirements for Big Data Storage
Three key areas have been identified that can be expected to govern future big data
storage technologies. This includes standardization of query interfaces, increasing
support for data security, protection of users privacy, and the support of semantic
data models. Standardized Query Interfaces
In the medium to long-term NoSQL databases would greatly benefit from standard-
ized query interfaces, similar to SQL for relational systems. Currently no standards
exist for the individual NoSQL storage types beyond de-facto standard APIs for
graph databases (Blueprints 2014) and the SPARQL data manipulation language
(Aranda et al. 2013) supported by triplestores vendors. Other NoSQL databases
usually provide their own declarative language or API, and standardization for
these declarative languages is missing.
While for some database categories (key/value, document, etc.) declarative
language standardization is still missing, there are efforts discussing standardiza-
tion needs. For instance the ISO/IEC JTC Study Group on big data has recently
recommended that existing ISO/IEC standards committee should further investi-
gate the “definition of standard interfaces to support non-relational data stores”
(Lee et al. 2014).
The definition of standardized interfaces would enable the creation of a data
virtualization layer that would provide an abstraction of heterogeneous data storage
systems as they are commonly used in big data use cases. Some requirements of a
data virtualization layer have been discussed online in an Infowor ld blog article
(Kobielus 2013). Security and Privacy
Interviews were conducted with consultants and end users of big data storage who
have responsibility for security and privacy, to gain their personal views and
insights. Based upon these interviews and the gaps identified in Sect. 7.4.2, several
future requirements for security and privacy in big data storage were identified.
130 M. Strohbach et al.
Data Commons and Social Norms Data stored in large quantities will be subject
to sharing as well as derivative work in order to maximize big data benefits. Today,
users are not aware how big data processes their data (transparency), and it is not
clear how big data users can share and obtain data efficiently. Furthermore, legal
constraints with respect to privacy and copyright in big data are currently not
completely clear within the EU. For instance, big data allows novel analytics
based upon aggregated data from manifold sources. How does this approach affect
private information? How can rules and regulations for remixing and derivative
works be applied to big data? Such uncertainty may lead to a disadvantage of the
EU compared to the USA.
Data Privacy Big data storage must comply with EU privacy regulations such as
Directive 95/46/EC when personal information is being stored. Today, heteroge-
neous implementations of this directive render the storage of personal information
in big data difficult. The General Data Protection Regulation (GDRP)—first pro-
posed in 2012—is an on-going effort to harmonize data protection among EU
member states. The GDRP is expected to influence future requirements for big
data storage. As of 2014, the GDRP is subject to negotiations that make it difficult
to estimate the final rules and start of enforcement. For instance, the 2013 draft
version allows data subjects (persons) to request data controllers to delete personal
data, which is often not suffic iently considered by big data storage solutions.
Data Tracing and Provenance Tracing and provenance of data is becoming
more and more important in big data storage for two reasons: (1) users want to
understand where data comes from, if the data is correct and trustworthy, and what
happens to their results and (2) big data storage will become subject to compliance
rules as big data enters critical business processes and value chains. Therefore, big
data storage h as to maintain provenance metadata, provide provenance along the
data processing chain, and offer user-friendly ways to understand and trace the
usage of data.
Sandboxing and Virtualization Sandboxing and virtualization of big data ana-
lytics becomes more important in addition to access control. According to econo-
mies of scale, big data analytics benefit from resource sharing. However, security
breaches of shared analytical components lead to compromised cryptographic
access keys and full storage access. Thus, jobs in big data analytics must be
sandboxed to prevent an escalation of security breaches and therefore unauthorized
access to storage. Semantic Data Models
The multitude of heterogeneous data sources increases development costs, as
applications require knowledge about individual data formats of each individual
source. An emerging trend is the semantic web and in particular the semant ic sensor
web that tries to address this challenge. A multitude of research projects are
7 Big Data Storage 131
concerned with all levels of semantic modelling and computation. As detailed in
this book, the need for semantic annotations has for instance been identified for the
health sector. The requirement for data storage is therefore to support the large-
scale storage and management of semantic data models. In particular trade-offs
between expressivity and efficient storage and querying need to be further explored.
7.5.2 Emerging Paradigms for Big Data Stora ge
There are several new paradigms emerging for the storage of large and complex
datasets. These new paradigms include, among others, the increased use of NoSQL
databases, convergence with analytics frameworks, and managing data in a central
data hub. Increased Use of NoSQL Databases
NoSQL databases, most notably graph databases and columnar stores, are increas-
ingly used as a replacement or complement to existing relational systems.
For instance, the requirement of using semantic data models and cross linking
data with many different data and information sources strongly drives the need to be
able to store and analyse large amounts of data using graph-based models. How-
ever, this requires overcoming the limitation of current graph-based systems as
described above. For instance, Jim Webber states “Graph technologies are going to
be incredibly important” (Webber 2013). In another interview, Ricardo Baeza-
Yates, VP of Research for Europe and Latin America at Yahoo!, also states the
importance of handling large-scale graph data (Baeza-Yates 2013). The Microsoft
research project Trinity achieved a significant breakthrough in this area. Trinity is
an in-memory data storage and distributed processing platform. By building on its
very fast graph traversal capabilities, Microsoft researchers int roduced a new
approach to cope with graph queries. Other projects include Goog le s knowledge
graph and Facebooks graph search that demonstrate the increasing relevance and
growing maturity of graph technologies. In-Memory and Column-Oriented Designs
Many modern high-performance NoSQL databases are based on columnar designs.
The main advantage is that in most practical applications only a few columns are
needed to access the data. Conseque ntly storing data in columns allows faster
access. In addition, column-oriented databases often do not support the expensive
join operations from the relational world. Instead , a common approach is to use a
single wide column table that stores the data based on a fully denormalized schema.
132 M. Strohbach et al.
According to Michael Stonebraker “SQL vendors will all move to column stores,
because they are wildly faster than row stores” (Stonebraker 2012a).
High-performance in-memory databases such as SAP HANA typically combine
in-memory techniques with column-based designs. In contrast to relational systems
that cache data in-memory, in-memory databases can use techniques such as anti-
caching (DeBrabant et al. 2013). Harizopoulos et al. have shown that the most time
for executing a query is spent on administrative tasks such as buffer management
and locking (Harizopoulos et al. 2008). Convergence with Analytics Frameworks
During the course of the project many scenarios have been identified that call for
better analysis of available data to improve operations in various sectors. Techni-
cally, this means an increased need for complex analytics that goes beyond simple
aggregations and statistics. Stonebraker points out that the need for complex
analytics will strongly impact existing data storage solutions (Stonebraker 2012b).
As use case specific analytics are one of the most crucial components that are
creating actual business value, it becomes increasingly important to scale up these
analytics satisfying performance requirements, but also to reduce the overall devel-
opment complexity and cost. Figure 7.2 shows some differences between using
separate systems for data management and analytics versus integrated analytical
databases. The Data Hub
A central data hub that integrates all data in an enterprise is a paradigm that
considers managing all company data as a whole, rather than in different, isolated
databases managed by different organizational units. The benefit of a central data
hub is that data can be analysed as a whole, linking various datasets owned by the
company thus leading to deeper insights.
Typical technic al implementations are based on a Hadoop-based system that
may use HDFS or HBase (Apache 2014) to store an integrated master dataset. On
one hand, this master dataset can be used as ground truth and backup for existing
data management systems, but it also provides the basis for advanced analytics that
combine previously isolated datasets.
Companies such as Cloudera use this paradigm to market their Hadoop distri-
bution (Cloudera 2014b ). Many use cases of enterprise data hub exis t already. A
case study in the financial sector is described in the next section.
7 Big Data Storage 133
7.6 Sector Case Studies for Big Data Storage
In this section three selected use cases are described that illustrate the potential and
need for future storage technologies. The health use case illustrates how social
media based analytics is enabled by NoSQL storage technologies. The second use
case from the financial sector illustrates the emerging paradigm of a centralized
data hub. The last use case from the energy sector illustrates the benefits of
managing fine-grained Internet of Things (IoT) data for advanced analytics. An
overview of the key characteristics of the use case can be found in Table 7.1. More
case studies are presented in Curry et al. (2014).
Fig. 7.2 Paradigm shift from pure data storage systems to integrated analytical databases
134 M. Strohbach et al.
7.6.1 Health Sector: Social Media-Based Medication
Treato is an Israeli company that specializes in mining user-generated content from
blogs and foru ms in order to provide brand intelligence services to pharmaceutical
companies. As Treato is analysing the social web, it falls into the “classical”
category of analysing large amounts of unstructured data, an application area that
often asks for big data storage solutions. Treatos service as a use case demonstrates
the value of using big data storage technologies. The information is based on a case
study published by Cloudera (2012), the company that provided the Hadoop
distribution Treato has been using.
While building its prototype, Treato discovered “that side effects could be
identified through social media long before pharmaceutical companies or the
Food & Drug Administration (FDA) issued warnings about them. For example,
when looking at discussions about Singulair, an asthma medication, Treato found
that almost half of UGC discussed mental disorders; the side effect would have
been identifiable four years before the official warning came out.” (Cloudera 2012).
Treato initially faced two major chal lenges: First, it needed to develop the
analytical capabilities to analyse patients colloquial language and map that into a
medical terminology suitable for delivering insights to its customers. Second, it was
necessary to analyse large amounts of data sources as fast as possible in order to
provide accurate information in real time.
The first challenge, developing the analytics, has been addressed initially with a
non-Hadoop system based on a relational database. With that system Treato was
facing the limitation that it could only handle “data collection from dozens of
websites and could only process a couple of million posts per day” (Cloudera
2012). Thus, Treato was looking for a cost-efficient analytics platform that could
fulfil the following key requirements:
1. Reliable and scalable storage
2. Reliable and scalable processing infrastructure
3. Search engine capabilities for retrieving post s with high availability
4. Scalable real-time store for retrieving statistics with high availability
Table 7.1 Key characteristics of selected big data storage case studies
Case study Sector Volume
technologies Key requirements
Treato: Social
media based medi-
cation intelligence
Health >150 TB HBase Cost-efficiency, scalability
limitations of relational DBs
Centralized data
Finance Between several
petabytes and
over 150 PB
Building more accurate
models, scale of data, suit-
ability for unstructured data
Smart grid Energy Tens of TB per
Hadoop Data volume, operational
7 Big Data Storage 135
As a result Treato decided on a Hadoop-based system that uses HBase to store
the list of URLs to be fetched. The posts available at these URLs are analysed by
using natural language processing in conjunction with their proprietary ontology. In
addition “each individual post is indexed, statist ics are calculated, and HBase tables
are updated” (Cloudera 2012).
According to the case study report, the Hadoop-based solution stores more than
150 TB of data including 1.1 billion online posts from thousands of websites
including about more than 11,000 medications and more than 13,000 conditions.
Treato is able to process 150–200 million user posts per day.
For Treato, the impact of the Hadoop-based storage and processing infrastruc-
ture is that they obtain a scalabl e, reliable, and cost-effective system that may even
create insight s that would not have been possible wi thout this infrastructure. The
case study claims that with Hadoop, Treato improved execution time at least by a
factor of six. This allowed Treato to respond to a customer request about a new
medication within one day.
7.6.2 Finance Sector: Centralized Data Hub
As mapped out in the description of the sectorial roadmaps (Lobillo et al. 2013), the
financial sector is facing challenges with respect to increas ing data volumes and a
variety of new data sources such as social media. Here use cases are describ ed for
the financial sector based on a Cloudera solution brief (Cloudera 2013).
Financial products are increasingly digitalized including online banking and
trading. As online and mobil e access simplifies access to financial products, there
is an increased level of activity leading to even more data. The potential of big data
in this scenario is to use all available data for building accurate models that can help
the financial sector to better manage financial risks. According to the solution brief,
companies have access to several petabytes of data. According to Larry Feinsmith,
managing director of JPMorgan Chase, his company is storing over 150 petabytes
online and use Hadoop for fraud detection (Cloudera 2013).
Secondly, new data sources add to both the volume and variety of available data.
In particular, unstructured data from weblogs, social media, blogs, and other news
feeds can help in customer relationship management, risk management, and maybe
even algorithmic trading (Lobillo et al. 2013). Pulling all the data together in a
centralized data hub enables more detailed analytics that can provide a competitive
edge. However traditional systems cannot keep up with the scale, costs, and
cumbersome integration of traditional extract, transform, load (ETL) processes
using fixed data schemes, nor are they able to handle unstructured data. Big data
storage systems however scale extremely well and can process both structured and
unstructured data.
136 M. Strohbach et al.
7.6.3 Energy: Device Level Metering
In the energy sector, smart grid and smart meter management is an area that
promises both high economic and environmental benefits. As depicted in Fig. 7.3,
the introduction of renewable energies such as photovoltaic systems deployed on
houses can cause grid instabilities. Currently grid operators have little knowledge
about the last mile to energy consumers. Thus they are not able to appropriately
react to instabilities caused at the very edges of the grid network. By analysing
smart meter data sampled at second intervals, short-term forecasting of energy
demands and managing the demand of devices such as heating and electrical cars
becomes possible, thus stabilizing the grid. If deployed in millions of households
the data volumes can reach petabyte scale, thus greatly benefiting from new storage
technologies. Table 7.2 shows the data volume only for the raw data collected for
one day.
The Peer Energy Cloud (PE C) project (2014) is a public funded project that has
demonstrated how smart meter data can be analysed and used for trading energy in
the local neighbourhood, thus increasing the overall stability of the power grid.
Moreover, it has successfully shown that by collecting more fine granular data,
i.e. monitoring energy consumption of individual devices in the household, the
accuracy of predicting the energy consumption of households can be significantly
improved (Ziekow et al. 2013). As the data volumes increase it becomes
+ Control
Fig. 7.3 Introduction of renewable energy at consumer sites changes the topology of the energy
grid and requires new measurement points at the leaves of the grid
7 Big Data Storage 137
increasingly difficult to handle the data with legacy relational databases (Strohbach
et al. 2011).
7.7 Conclusions
The chapter contains an overview of current big data storage technologies as well as
emerging paradigms and future requirements. The overview specifically included
technologies and approaches related to privacy and security. Rather than focusing
on detailed descriptions of individual technologies a broad overview was provided,
and technical aspects that have an impact on creating value from large amounts of
data highlighted. The social and economic impact of big data storage technologies
was described, and three selected case studies in three different sectors were
detailed, which illustrate the need for easy to use scalable technologies.
It can be concluded that there is already a huge offering of big data storage
technologies. They have reached a maturity level that is high enough that early
adopters in various sectors already use or plan to use them. Big data storage often
has the advantage of better scalability at a lower price tag and operational com-
plexity. The current state of the art reflects that the efficient management of almost
any size of data is not a challenge per se. Thus it has huge potential to transform
business and society in many areas.
It can also be concluded that there is a strong need to increase the maturity of
storage technologies so that they fulfil future requireme nts and lead to a wider
adoption, in particular in non-IT-based companies. The required technical improve-
ments include the scalability of graph databases that will enable better handling of
complex relationships, as well as further minimizing query latencies to big datasets,
e.g. by usin g in-memory databases. Another major roadblock is the lack of stan-
dardized interfaces to NoSQL database systems. The lack of standardization
reduces flexibility and slows down adoption. Finally, considerable improvements
for security and privacy are required. Secure storage technologies need to be further
developed to protect the privacy of users.
More details about big data storage technologies can be found in Curry
et al. (2014). This report, in conjunction with the analysis of the public and
10 industrial sectors (Zillner et al. 2014), has been used as a basis to develop the
cross-sectorial roadmap described in this book.
Table 7.2 Calculation of the amount of data sampled by smart meters
Sampling rate 1 Hz
Record size 50 Bytes
Raw data per day and household 4.1 MB
Raw data per day for 10 Mio customers ~39 TB
138 M. Strohbach et al.
Open Access This chapter is distributed under the terms of the Creative Commons Attribution-
Noncommercial 2.5 License ( which permits any
noncommercial use, distribution, and reproduction in any medium, provided the original author(s)
and source are credited.
The images or other third party material in this book are included in the works Creative
Commons license, unless indicated otherwise in the credit line; if such material is not included in
the works Creative Commons license and the respective action is not permitted by statutory
regulation, users will need to obtain permission from the license holder to duplicate, adapt, or
reproduce the material.
Acquisti, A., & Gross, R. (2009). Predicting social security numbers from public data. Proceedings
of the National Academy of Sciences, 106(27), 10975–10980.
Ahmadi Zeleti, F., Ojo, A., & Curry, E. (2014). Business models for the open data industry:
Characterization and analysis of emerging models. In 15th Annual International Conference on
Digital Government Research (dg.o 2014) (pp. 215–226). ACM.
Apache. (2014). Apache HBase Project Website. Accessed Nov 21, 2014.
Aranda, C. B., Corby, O., Das, S., Feigenbaum, L., Gearon, P., Glimm, B., et al. (2013). SPARQL
1.1 overview. In W3C Recommendation.
Baeza-Yates, R. (2013). BIG Interview by John Dominque.
Blaze, M., Bleumer, G., & Strauss, M. (2006). Divertible protocols and atomic proxy cryptogra-
phy. In Proceedings of Eurocrypt (pp. 127–144).
Blueprints. (2014). Blueprints Project Homepage.
Accessed Feb 4, 2015.
Bollier, D., & Firestone, C. M. (2010). The promise and peril of big data. Washington, DC: Aspen
Institute, Communications and Society Program.
Calder, B., Wang, J., Ogus, A., Nilakantan, N., Skjolvsvold, A., McKelvie, S. et al. (2011).
Windows azure storage: a highly available cloud storage service with strong consistency. In:
Proceedings of the Twenty-Third ACM Symposium on Operating Systems Principles
(143–157). New York, NY: ACM.
CloudDrive. (2013). Advantages of cloud data storage. http://www.clouddriv Accessed Nov 20, 2013.
Cloudera. (2012). Treato Customer Case Study.
Cloudera. (2013). Identifying fraud, managing risk and improving compliance in financial
Cloudera. (2014a). Cloudera Company Web page. Accessed May 6, 2015.
Cloudera. (2014b). Rethink data. Accessed
Feb 4, 2014.
Curry, E., Ngonga, A., Domingue, J., Freitas, A., Strohbach, M., Becker, T., et al. (2014). D2.2.2.
Final version of the technical white paper. Public deliverable of the EU-Project BIG (318062;
DeBrabant, J., Pavlo, A., Tu, S., Stonebraker, M., & Zdonik, S. (2013). Anti-caching: a new
approach to database management system architecture. In Proceedings of the VLDB Endow-
ment (pp 1942–1953).
Erling, O. (2009). Virtuoso, a Hybrid RDBMS/Graph Column Store. In: Virtuoso
Open-Source Wiki.
VOSArticleVirtuosoAHybridRDBMSGraphColumnStore. Accessed Feb 6, 2015.
Franz. (2015). Allegrograph product web page. Accessed
Feb 6, 2015.
7 Big Data Storage 139
Gislason, H. (2013). BIG Interview by John Dominque.
Glavic, B. (2014). Big data provenance: Challenges and implications for benchmarking. In
T. Rabl, M. Poess, C. Baru, & H.-A. Jacobsen (Eds.), Specifying big data benchmarks
(pp. 72–80). Berlin: Springer.
Goyal, V., Pandey, O., Sahai, A., & Waters, B. (2006). Attribute-based encryption for fine-grained
access control of encrypted data. In Proceedings of the 13th ACM Conference on Computer
and Communications Security (pp. 89–98).
Grady, J. (2013). Is enterprise cloud storage a good fit for your business? http://www.1cloudroad.
Harizopoulos, S., Abadi, D. J., Madden, S., & Stonebraker, M. (2008). OLTP through the looking
glass, and what we found there. In Proceedings of the 2008 ACM SIGMOD International
Conference on Management data (SIGMOD 08).
Hasan, O., Habegger, B., Brunie, L., Bennani, N., & Damiani, E. (2013). A discussion of privacy
challenges in user profiling with big data techniques: The EEXCESS use case. IEEE Interna-
tional Congress on Big Data (pp. 25–30).
Hortonworks. (2014). Hortonworks Company Web Page. . Accessed Feb
6, 2015.
Ikeda, R., Park, H., & Widom, J. (2011). Provenance for generalized map and reduce workflows.
In Biennial Conference on Innovative. Data Systems Research (pp. 273–283).
iQuartic. (2014). iQuartic pioneers use of Hadoop with Electronic Health Records (EHRs/EMRs).
In: iQuartic Blog. Accessed Feb 7, 2014.
Kamara, S., & Lauter, K. (2010). Cryptographic cloud storage. In Financial Cryptography and
Data Security (pp. 136–149).
Kobielus, J. (2013) Big Data needs data virtualization. In: InfoWorld Webpage. http://www. .html. Accessed
Nov 18, 2014.
Koomey, J. G. (2008). Worldwide electricity used in data centers. Environmental Research
Letters, 3, 034008. doi:10.1088/1748-9326/3/3/034008.
Lee, C.-C., Chung, P.-S., & Hwang, M.-S. (2013). A survey on attribute-based encryption schemes
of access. International Journal of Network Security, 15, 231–240.
Lee, K., Manning, S., Melton, J., Boyd, D., Grady, N., & Levin, O. (2014). Final SGBD report to
JTC1, ISO/IEC JTC1 SGBD document no. N0095.
Li, M., Yu, S., Zheng, Y., Ren, K., Lou, W., et al. (2013). Scalable and secure sharing of personal
health records in cloud computing using attribute-based encryption. IEEE Transactions on
Parallel and Distributed Systems, 24, 131–143.
Lobillo, F., Puente, M. A., Lippell, H., Zillner, S., Bretschneider, C., Oberkampf, H., et al. (2013).
BIG deliverable D2.4.1 First draft of sector’s roadmaps.
Manyika, J., Chui, M., Brown, B., Bughin, J., Dobbs, R., Roxburgh, C., et al. (2011). Big data: The
next frontier for innovation, competition, and productivity. Sydney: McKinsey Global
MapR. (2014). MapR Company Website. Accessed Feb 6, 2014.
Margaret Rouse. (2014a). Object Storage.
storage. Accessed Nov 20, 2014.
Margaret Rouse. (2014b). Block Storage.
age. Accessed Feb 4, 2014.
Marz, N., & Warren, J. (2014). A new paradigm for Big Data. In N. Marz & J. Warren (Eds.), Big
Data: Principles and best practices of scalable real-time data systems. Shelter Island, NY:
Manning Publications.
Melnik, S., Garcia-Molina, H., & Rahm, E. (2002). Similarity flooding: A versatile graph
matching algorithm and its application to schema matching. In Proceedings of the 18th
International Conference Data Engineering. IEEE Computer Society (pp. 117–128).
Miller, S. P., Neuman, B. C., Schiller, J. I., & Saltzer, J. H. (1987, December 21). Section E.2.1:
Kerberos authentication and authorization system. MIT Project Athena, Cambridge, MA.
Mora, A. C., Chen, Y., Fuchs, A., Lane, A., Lu, R., Manadhata, P. et al. (2012). Top Ten Big Data
Security and Privacy Challenges. Cloud Security Alliance (CSA).
140 M. Strohbach et al.
Neo4j. (2015). Neo4j Company Website. Accessed Feb 6, 2015.
Okman, L., Gal-Oz, N., Gonen, Y., et al. (2011). Security issues in nosql databases. In 2011 I.E.
10th International Conference on Trust, Security and Privacy in Computing and Communica-
tions (TrustCom) (pp. 541–547).
OpenStack. (2014). OpenStack Website. Accessed Feb 3, 2014.
PEC. (2014). Peer Energy Cloud Project Website. Accessed Feb
6, 2015.
Russel, J. (2013). Cloudera Impala. O#Reilly Media
Shenker, S., Stoica, I., Zaharia, M., & Xin, R. (2013). Shark: SQL and rich analytics at scale. In
Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data
(pp. 13–24).
Sherif, A., Sohan, R., & Hopper, A. (2013). HadoopProv: Towards provenance as a first class
citizen in MapReduce. In Proceedings of the 5th USENIX Workshop on the Theory and
Practice of Provenance.
Shucheng, Y., Wang, C., Ren, K., & Lou, W. (2010). Achieving secure, scalable, and fine-grained
data access control in cloud computing. INFOCOM (pp. 1–9).
Shvachko, K. H. K., Radia, S., & Chansler, R. (2010). The Hadoop distributed file system. In IEEE
26th Symposium on Mass Storage Systems and Technologies (pp. 1–10).
Stonebraker, M. (2012a) What Does “Big Data” Mean? In: BLOG@ACM.
blogs/blog-cacm/155468-what-does-big-data-mean/fulltext. Accessed Feb 5, 2015.
Stonebraker, M. (2012b). What Does “Big Data” Mean? (Part 2). In: BLOG@ACM. http://cacm. Accessed Apr
25, 2013.
Strohbach, M., Ziekow, H., Gazis, V., & Akiva, N. (2011). Towards a big data analytics
framework for IoT and smart city applications. In F. Xhafa & P. Papajorgji (Eds.), Modeling
and processing for next generation big data technologies with applications (pp. 257–282).
Berlin: Springer.
Thusoo, A., Sarma, J., Jain, N., Shao, Z., Chakka, P., Anthony, S., et al. (2009). Hive a
warehousing solution over a map-reduce framework. Statistics and Operations Research
Transactions, 2, 1626–1629.
Turner, V., Gantz, J. F., Reinsel, D., & Minton, S. (2014). The digital universe of opportunities:
Rich data and the increasing value of the internet of things. Framingham, MA: International
Data Corporation (IDC). Accessed Aug 19, 2015.
Venkatesh, P., & Nirmala, S. (2012). NewSQL The new way to handle big data. In: Blog. http:// Accessed Nov 18, 2014.
Vogels, W. (2013). Day 2 Keynote. AWS reInvent.
VoltDB. (2014). VoltDB Company Website. Accessed Nov 21, 2014.
Webber, J. (2013). BIG Interview by John Dominque.
White, T. (2012). Hadoop: The Definitive Guide.OReilly.
Wikipedia. (2013). Column-oriented DBMS.
DBMS. Accessed Apr 25, 2013.
Winder, D. (2012). Securing NoSQL applications: Best practises for big data security. Computer
Wu, C., & Guo, Y. (2013). Enhanced user data privacy with pay-by-data model. In Proceeding of
2013 I.E. International Conference on Big Data (pp. 53–57).
Zaharia, M., Chowdhury, M., Franklin, M. J., Shenker, S., & Stoica, I. (2010). Spark: Cluster
computing with working sets. In HotCloud10 Proceedings of the 2nd USENIX Conference on
Hot Topics in Cloud Computing (p. 10).
Ziekow, H., Goebel, C., Struker, J., & Jacobsen, H.-A. (2013). The potential of smart home sensors
in forecasting household electricity demand. In Smart Grid Communications
(SmartGridComm), 2013 I.E. International Conferences (pp. 229–234).
Zillner, S., Neururer, S., Munne, R., Prieto, E., Strohbach, M., van Kasteren, T., et al. (2014).
D2.3.2. Final Version of the Sectorial Requisites. Public Deliverable of the EU-Project BIG
(318062; ICT-2011.4.4).
7 Big Data Storage 141
... Some data storage technologies used for handling and managing huge amount of data are: Distributed File Systems, NoSQL databases, NewSQL databases and Big Data Querying Platforms [14]. NoSQL databases in big data storage can be compared according some parameters as storage, flexibility, performances, scalability, format and, complexity [15] etc. Business organizations must take decisions about storing the data physically in Cloud Storage or On-Premises storage. ...
Conference Paper
Data Management can be defined as the process of extracting, storing, organizing, and maintaining the data created and collected in organizations. Today's organizations invest in data management solutions that provide an efficient way to manage data in a unified structure. The enormous growth of data in the last decades has created a necessity for the fast extracting, accessing, and processing of data. Optimization has been a key component in improving the system's performance, searching, and accessing data in different data management solutions. Optimization is a mathematical discipline that formulates mathematical models and finds the best solution among a set of feasible solutions. This paper aims to give a general overview of applications of optimization techniques and algorithms in different areas of data management in the last decades. Data management includes a large group of functionalities, but we will focus on studying and reviewing the recent development of optimization algorithms used in databases, data warehouses, big data, and machine learning. Furthermore, this paper will identify applications of optimization in data management, reviews the current solutions proposed, and emphasize future topics where there is a lack of studies in data management.
... After generating data, it must be stored to be processed at later stages by analytics tools. Storage technologies must address the volume, velocity, and variety challenges of Big Data in a flexible and efficient manner [93]. NoSQL database and distributed file system technologies have become widespread and are often more efficient and more powerful than traditional relational database systems in data with high volume [77,94,95]. ...
Smart grid is a new paradigm that integrates traditional electricity grid and communication networks. Reliability is a critical challenge associated with smart grid and needs to be addressed. Based on comprehensive literature review and experts’ judgments, we develop a model to identify the most important criteria that have an impact on smart grid reliability from the perspective of users. The model takes into account three main criteria: “Big Data Management,” “Communication System,” and “System Functionality.” The fuzzy analytic hierarchy process is applied to analyze and prioritize these criteria based on the triangular fuzzy numbers and triangular membership function.
Purpose Smart grid is an integration between traditional electricity grid and communication systems and networks. Providing reliable services and functions is a critical challenge for the success and diffusion of smart grids that needs to be addressed. The purpose of this study is to determine the critical criteria that affect smart grid reliability from the perspective of users and investigate the role big data plays in smart grid reliability. Design/methodology/approach This study presents a model to investigate and identify criteria that influence smart grid reliability from the perspective of users. The model consists of 12 sub-criteria covering big data management, communication system and system characteristics aspects. Multi-criteria decision-making approach is applied to analyze data and prioritize the criteria using the fuzzy analytic hierarchy process based on the triangular fuzzy numbers. Data was collected from 16 experts in the fields of smart grid and Internet of things. Findings The results show that the “Big Data Management” criterion has a significant impact on smart grid reliability followed by the “System Characteristics” criterion. The “Data Analytics” and the “Data Visualization” were ranked as the most influential sub-criteria on smart grid reliability. Moreover, sensitivity analysis has been applied to investigate the stability and robustness of results. The findings of this paper provide useful implications for academicians, engineers, policymakers and many other smart grid stakeholders. Originality/value The users are not expected to actively participate in smart grid and its services without understanding their perceptions on smart grid reliability. Very few works have studied smart grid reliability from the perspective of users. This study attempts to fill this considerable gap in literature by proposing a fuzzy model to prioritize smart grid reliability criteria.
Located in the center of contemporary information and technology society, big data causes evolutionary transformations in many areas. A potential competitive advantage is provided through big data analytics to revolutionize social, cultural, political and economic relations. Just like other industries, television has also been affected by this digital transformation. The integration of television into technology can be observed in areas such as content production and distribution occurring through big data processing in digital media platforms. The digital transformation process in television was covered through the usage areas of big data in digital platforms and within the scope of current applications in this study. The importance of big data for media industry, which is closely related to technology was presented through the innovations it provided to the new broadcasting ecosystem. With its theoretical approach, the study is aiming to examine the conceptualization of big data and the improvement and use of big data in digital media platforms.
Big data applications consist of i) data collection using big data sources, ii) storing and processing the data, and iii) analysing data to gain insights for creating organisational benefit. The influx of digital technologies and digitization in the construction process includes big data as one newly emerging digital technology adopted in the construction industry. Big data application is in a nascent stage in construction, and there is a need to understand the tangible benefit(s) that big data can offer the construction industry. This study explores the benefits of big data in the construction industry. Using a qualitative case study design, construction professionals in an Australian Construction firm were interviewed. The research highlights that the benefits of big data include reduction of litigation amongst projects stakeholders, enablement of near to real-time communication, and facilitation of effective subcontractor selection. By implication, on a broader scale, these benefits can improve contract management, procurement, and management of construction projects. This study contributes to an ongoing discourse on big data application, and more generally, digitization in the construction industry.
The digital economy has been defined in the economic literature as one with near zero marginal cost, unmonetized services but also an escalating data flow. After a careful review of the most recent economic papers, we offer an alternative theory on the cost of privacy and data protection regulations. We have observed that the characteristics of the regulation lead not only to the amplification of costs that have been traditionally assigned as variable costs by the literature, but also of costs that used to be fixed but have been outsourced in the digital economy, meaning that significant new variable costs might trigger diseconomies of scale. At the same time, privacy and data protection regulations have created incentives that are making the dominant firms insource, in what seems to be a way back to increased sunk fixed costs for these firms. Having all that in mind, we claim that the perception of deterrence and compliance costs has affected how firms might decide to incur higher risks to avoid costs. Although compliance costs are high, we claim that an efficient implementation of the regulation avoids much of these costs. Our claim is supported by evidence that a relevant share of the regulatory costs are now variable costs, leaving room for at least two efficient strategies that medium-sized firms might implement in order to avoid them. First, firms can lower the volumes of data that they use without significantly impairing the predictive functions of their algorithms. Second, firms can invest in security at a comparatively lower degree than dominant firms considering their lower exposure to strong regulatory action.
Conference Paper
In the recent years, the term big data has attracted a lot of attention. It refers to the processing of data that is characterized mainly by 4Vs, namely volume, velocity, variety and veracity. The need for collecting and analysing big data has increased manifolds these days as organizations want to derive meaningful information out of any data that is available and create value for the business. A challenge that comes with big data is inferior data quality due to which a lot of time is spent on data cleaning. One prerequisite for solving data quality issues is to understand the reasons for their occurrence. In this paper, we discuss various issues that cause reduced quality of the data during the acquisition and management. Furthermore, we extend the research to categorize the quality of data with respect to the identified issues.
Full-text available
Nowadays, Big Data management has become a key basis for innovation, productivity growth, and competition. The correlated exploitation of data of this magnitude remains primordial to discover valuable insights and support decision making for domains of major interest. Furthermore, despite the complex aspects of Big Data environments, users are usually looking for a unified and appropriate view of this huge and heterogeneous data, to support the extraction of reliable and consistent knowledge. Thus, Big Data integration mechanisms must be considered to provide a uniform query interface, to mediate across large datasets and provide data scientists with a consistent integrated view suitable for analytical exploitations. Thus, this paper presents a semantic-based Big Data integration framework that relies on large-scale ontology matching and probabilistic-logical based assessment strategies. This framework applies optimization mechanisms and leverages parallel-computing paradigms (Hadoop and MapReduce) using commodity computational resources, to efficiently address the Big Data challenges and aspects. Several experiments were conducted and have proven the efficiency of this framework in terms of accuracy, performance, and scalability.
Conference Paper
Full-text available
Abstract- Business Intelligence (BI) is an approach to optimize business decisions by utilizing existing information. BI has been supporting the organization to form effective strategic and tactical business strategies. For the past 2 decades, BI hasn't only had a great impact on large enterprises, but also it has aided in forming strong core strategies of small to mid-sized enterprises. This paper details the full BI solution's cycle; planning, design, and development with a focus on small enterprises. Fig. 1 ("System Development Cycle") summarizes the process followed to develop a BI solution in this paper. Fig. 1 System Development Cycle for Developing BI Solution [1] The study involves a case study of a fictitious company, created for the sole purpose of applying the suggested BI solution. This case study elaborates on all the phases required to develop a BI solution for each phase. The challenges and benefits of the BI solution are thoroughly considered in this study. To evaluate the functionality of the proposed BI solution, several testing iterations were performed and the results of these iterations are presented at the end of this case study. The paper concludes by highlighting the areas for future research in BI.
Full-text available
Data Provenance is information about the origin and creation process of data. Such information is useful for debugging data and transformations, auditing, evaluating the quality of and trust in data, modelling authenticity, and implementing access control for derived data. Provenance has been studied by the database, workflow, and distributed systems communities, but provenance for Big Data - which we refer to as Big Provenance - is a largely unexplored field. This paper reviews existing approaches for large-scale distributed provenance and discusses potential challenges for Big Data benchmarks that aim to incorporate provenance data/management. Furthermore, we will examine how Big Data benchmarking could benefit from different types of provenance information. We argue that provenance can be used for identifying and analyzing performance bottlenecks, to compute performance metrics, and to test a system’s ability to exploit commonalities in data and processing.
Full-text available
An increasing amount of valuable data sources, advances in Internet of Things and Big Data technologies as well as the availability of a wide range of machine learning algorithms offers new potential to deliver analytical ser-vices to citizens and urban decision makers. However there is still a gap in combining the current state-of-the art in an integrated framework that would help reducing development costs and enable new kind of services. In this chap-ter we show how such an integrated Big Data analytical framework for Internet of Things and Smart City application could look like. The contributions of this chapter are threefold: (1) we provide an overview of Big Data and Internet of Things technologies including a summary of their relationships, (2) we present a case study in the smart grid domain that illustrates the high level requirements towards such an analytical Big Data framework, and (3) we present an initial version of such a framework mainly addressing the volume and velocity chal-lenge. The findings presented in this chapter are extended results from the EU funded project BIG and the German funded project PEC.
An increasing amount of valuable data sources, advances in Internet of Things and Big Data technologies as well as the availability of a wide range of machine learning algorithms offers new potential to deliver analytical services to citizens and urban decision makers. However, there is still a gap in combining the current state of the art in an integrated framework that would help reducing development costs and enable new kind of services. In this chapter, we show how such an integrated Big Data analytical framework for Internet of Things and Smart City application could look like. The contributions of this chapter are threefold: (1) we provide an overview of Big Data and Internet of Things technologies including a summary of their relationships, (2) we present a case study in the smart grid domain that illustrates the high-level requirements towards such an analytical Big Data framework, and (3) we present an initial version of such a framework mainly addressing the volume and velocity challenge. The findings presented in this chapter are extended results from the EU funded project BIG and the German funded project PEC.
In Attribute-based Encryption (ABE) scheme, attributes play a very important role. Attributes have been ex- ploited to generate a public key for encrypting data and have been used as an access policy to control users' access. The access policy can be categorized as either key-policy or ciphertext-policy. The key-policy is the access struc-ture on the user's private key, and the ciphertext-policy is the access structure on the ciphertext. And the access structure can also be categorized as either monotonic or non-monotonic one. Using ABE schemes can have the advantages: (1) to reduce the communication overhead of the Internet, and (2) to provide ne fia-grained access control. In this paper, we survey a basic attribute-based encryption scheme, two various access policy attribute-based encryption schemes, and two various access struc-tures, which are analyzed for cloud environments. Finally, we list the comparisons of these schemes by some criteria for cloud environments.
Conference Paper
Business models for open data have emerged in response to the economic opportunities presented by the increasing availability of open data. However, scholarly efforts providing elaborations, rigorous analysis and comparison of open data models are very limited. This could be partly attributed to the fact that most discussions on open data business models are predominantly in the practice community. This shortcoming has resulted in a growing list of open data business models which, on closer examination, are not clearly delineated and lack clear value orientation. We address this problem by 1) consolidating reported open data business models in both academic and practice literature, 2) describe the models based on a business model framework, and 3) determine open data business models patterns. In addition, we identified the emerging core value disciplines for open data businesses. Our results help to streamline existing useful models, and link them to the overall business strategy through value disciplines.
Conference Paper
We introduce HadoopProv, a modified version of Hadoop that implements provenance capture and analysis in MapReduce jobs. It is designed to minimise provenance capture overheads by (i) treating provenance tracking in Map and Reduce phases separately, and (ii) deferring construction of the provenance graph to the query stage. Provenance graphs are later joined on matching intermediate keys of the Map and Reduce provenance files. In our prototype implementation, HadoopProv has an overhead below 10% on typical job runtime (<7% and <30% average temporal increase on Map and Reduce tasks respectively). Additionally, we demonstrate that provenance queries are serviceable in O (k log n), where n is the number of records per Map task and k is the set of Map tasks in which the key appears.