ArticlePDF Available

Building high throughput permissioned blockchain fabrics: challenges and opportunities

Authors:

Abstract

Since the introduction of Bitcoin---the first widespread application driven by blockchains---the interest in the design of blockchain-based applications has increased tremendously. At the core of these applications are consensus protocols that securely replicate client requests among all replicas, even if some replicas are Byzantine faulty. Unfortunately, these consensus protocols typically have low throughput, and this lack of performance is often cited as the reason for the slow wider adoption of blockchain technology. Consequently, many works focus on designing more efficient consensus protocols to increase throughput of consensus. We believe that this focus on consensus protocols only explains part of the story. To investigate this belief, we raise a simple question: Can a well-crafted system using a classical consensus protocol outperform systems using modern protocols? In this tutorial, we answer this question by diving deep into the design of blockchain systems. Further, we take an in-depth look at the theory behind consensus, which can help users select the protocol that best-fits their requirements. Finally, we share our vision of high-throughput blockchain systems that operate at large scales.
Building High Throughput Permissioned Blockchain
Fabrics: Challenges and Opportunities
Suyash Gupta, Jelle Hellings, Sajjad Rahnama, Mohammad Sadoghi
Exploratory Systems Lab, Department of Computer Science
University of California, Davis, CA 95616-8562, USA
ABSTRACT
Since the introduction of Bitcoin—the first widespread ap-
plication driven by blockchains—the interest in the design of
blockchain-based applications has increased tremendously.
At the core of these applications are consensus protocols
that securely replicate client requests among all replicas,
even if some replicas are Byzantine faulty. Unfortunately,
these consensus protocols typically have low throughput,
and this lack of performance is often cited as the reason
for the slow wider adoption of blockchain technology. Con-
sequently, many works focus on designing more efficient con-
sensus protocols to increase throughput of consensus.
We believe that this focus on consensus protocols only ex-
plains part of the story. To investigate this belief, we raise
a simple question: Can a well-crafted system using a clas-
sical consensus protocol outperform systems using modern
protocols? In this tutorial, we answer this question by div-
ing deep into the design of blockchain systems. Further, we
take an in-depth look at the theory behind consensus, which
can help users select the protocol that best-fits their re-
quirements. Finally, we share our vision of high-throughput
blockchain systems that operate at large scales.
PVLDB Reference Format:
Suyash Gupta, Jelle Hellings, Sajjad Rahnama and Mohammad
Sadoghi. Building High Throughput Permissioned Blockchain
Fabrics: Challenges and Opportunities. PVLDB, 13(12): 3441-
3444, 2020.
DOI: https://doi.org/10.14778/3415478.3415565
1. INTRODUCTION
Since the introduction of Bitcoin—the first wide-spread
application driven by blockchain—the interest in the de-
sign of blockchain-based applications has increased tremen-
dously. This interest has resulted in several blockchain-
inspired fabrics and database systems [2, 3, 17, 27, 39, 40].
Further, blockchain-based systems have been employed to
address challenges in various other fields such as food pro-
duction, managing land property rights, energy trading, and
managing identities.
This work is licensed under the Creative Commons Attribution-
NonCommercial-NoDerivatives 4.0 International License. To view a copy
of this license, visit http://creativecommons.org/licenses/by-nc-nd/4.0/. For
any use beyond those covered by this license, obtain permission by emailing
info@vldb.org. Copyright is held by the owner/author(s). Publication rights
licensed to the VLDB Endowment.
Proceedings of the VLDB Endowment, Vol. 13, No. 12
ISSN 2150-8097.
DOI: https://doi.org/10.14778/3415478.3415565
At the core of these blockchain applications are Byzantine-
Fault Tolerant (BFT) consensus protocols that ensures all
replicas of this blockchain application reach consensus on
the ordering of incoming client request, this even if some of
the replicas are Byzantine [8, 24, 27, 34, 45].
A decade after the introduction of blockchains in crypto-
currencies and after several prominent research projects, we
see that crypto-currencies are still the major known use-
cases of blockchains. This raises a key question: Why have
blockchain applications seen such a slow wider adoption?
The low throughput and high latency of BFT consensus
are cited as key reasons for this. Prior works have shown
that traditional distributed databases can achieve through-
puts of the order 100 K transactions per second [29, 30],
while initial permissionless blockchain applications such as
Bitcoin [38] and Ethereum [44] have throughputs of only
a few transactions per second. In these crypto-currency
applications, low throughputs are seen as acceptable, as
these costly techniques enable a fully decentralized currency
that is not controlled by a single government or corpora-
tion. Indeed, crypto-currency blockchains typically have
open-membership, as anyone can join these blockchains.
Although permissionless blockchains highlight the notions
of decentralization and resilience, their open-membership
is often unnecessary for fault-tolerant transaction process-
ing. This led to the design of industry-grade permissioned
blockchains, where only a select group of users, some of
which may be untrusted, can participate [3]. These permis-
sioned designs employ traditional BFT consensus to provide
throughputs of up-to 10 K transactions per second [2, 3],
which is still short of the performance expected of modern
systems. Several prior works [9, 11, 34, 45] blame the low
throughput and scalability of permissioned blockchains on
the underlying BFT consensus. Although these claims are
not false, they only explain part of the story. From our per-
spective, permissioned blockchain applications can achieve
wider adoption by improving in three vital directions.
First, it is well-known that an efficient protocol may not
always lead to a high-throughput implementation. The same
principle applies to consensus protocols used in existing per-
missioned blockchain fabrics. Although these fabrics provide
platforms to employ blockchains in various use-cases, these
fabrics fall short in their architectural details [2, 3, 6, 21].
We claim that the low throughput of these fabrics is due
to missed opportunities during their design and implemen-
tation. Indeed, a well-crafted blockchain fabric can have
an order-of-magnitude increase in its throughput, e.g., by
exploiting parallelization and pipelining opportunities.
3441
Second, the full replication employed by current block-
chain applications stands in their way of achieving ever-
higher performance. To make blockchains more usable, their
design needs to evolve to incorporate sharding and special-
ization. To support such designs, new resilient techniques
besides consensus need to be developed.
Finally, permissioned blockchain systems need tuning to
specific settings. E.g., blockchains can be used for feder-
ated data management, the collective management of a single
database among various stakeholders. Federated data man-
agement is in itself a major step towards dealing with data
quality issues arising from the non-federated interchange of
information between various stakeholders and, as such, can
reduce the huge negative economic impact of bad data [16,
33, 42]. In federated data management, resilience is less of a
priority, and the focus of such blockchain systems is on fast
data update and retrieval, efficient query processing, and
modular data analysis.
2. OUTLINE OF THE TUTORIAL
In this tutorial, we will provide a deep dive into consen-
sus protocols with a focus on data management. To do so,
we take an in-depth look at Byzantine fault-tolerant con-
sensus protocols, the main technique powering permissioned
blockchains. The tutorial is intended for an audience that
has prior knowledge of databases and will be of interest to
both theoreticians and practitioners who want to employ
blockchain concepts to their work.
This tutorial starts with a general-purpose introduction to
blockchains from the perspective of data management. Fol-
lowing the introduction, the tutorial will focus on three av-
enues. First, we look at the theoretical framework in which
permissioned blockchains operate. Then, we look at prac-
tical high-performance consensus protocols and at current
developments. This theoretical framework provides users of
a permissioned blockchain system with the right tools to se-
lect the consensus protocols that best-fits their requirements.
Second, we look at the architectural challenges in the de-
sign of high-performance permissioned blockchain systems,
in which we show how existing principles of thread paral-
lelization and task pipelining can be applied to blockchain
fabrics. Further, we illustrate ways to optimize data access
and query processing in permissioned blockchain applica-
tions. Finally, we look at the design challenges for high-
performance permissioned blockchain systems of the future
that can deal with huge amounts of data. We conclude by
presenting our vision on future developments. Next, we ex-
plain these three avenues in detail.
BFT Consensus Protocols. Blockchains are, at their ba-
sis, fully replicated distributed systems that aim to main-
tain data consistency. The well-known CAP Theorem puts
restrictions on the types of failures these blockchains can
deal with while guaranteeing continued services [7, 20]. The
CAP Theorem puts rather general limitations on the design
of blockchains, however. More specific limitations are also
known, as the Byzantine consensus problem and other re-
lated problems, such as the Byzantine agreement problem
and the interactive consistency problem, have received con-
siderable attention.
It is well-known that the Byzantine agreement problem
can only be solved when using synchronous communica-
tion [19, 37, 43]. In a synchronous environment with n
4 8 16 32
0
0.5
1
1.5
·105
Number of replicas (n)
Throughput (txn/s)
Pbft
Zyzzyva
Figure 1: Two permissioned applications employing
distinct BFT protocols.
replicas of which fare Byzantine (e.g., malicious), Byzan-
tine agreement requires that n>3f[12, 13]. When strong
cryptographic primitives are available, this can be improved
to n>f[15, 35, 41] (although practical systems will still re-
quire n>2f). Additionally, bounds on the amount of com-
munication and the quality of the network are known [10,
12, 13, 14, 15, 18].
Having provided a theoretical background, we take a step
towards detailing practical consensus protocols. We do so
by a full coverage of the Practical Byzantine Fault Tolerance
consensus protocol (Pbft) of Castro et al. [8]. Next, we also
look at the lineage of consensus protocols that refine and im-
prove Pbft. This detailed overview will cover many of the
practical consensus protocols currently in use and, simulta-
neously, also covers recent developments. Our coverage will
include protocols such as Pbft, HotStuff [45], Zyzzyva [4,
34], FaB [36], SynBFT [1], RBFT [5], PoE [23], and Multi-
BFT [24, 25]. All of these protocols make some tradeoffs
and, hence, achieve optimal throughput only under specific
conditions. In our tutorial, we will discuss these tradeoffs
and conditions in detail, as this enables one to select the
protocol that best-fits the requirements of their particular
blockchain application.
Architectural Paradigm. Although an efficient consensus
protocol can help increase the throughput of the associated
permissioned blockchain application, the design of the sys-
tem and its implementation matters equally. In our tutorial,
we will show that classical BFT protocols that are perceived
to be slow such as Pbft [8] can always outperform niche-
case optimized BFT protocols such as Zyzzyva [34] if im-
plemented in a well-designed and skillfully-optimized block-
chain fabric. We use Figure 1 to illustrate such a possibility.
In this figure, we measure the throughput of ResilientDB,
our permissioned blockchain system [27, 28], and intention-
ally make it employ the “slow” Pbft protocol. Next, we
compare the throughput of ResilientDB against a permis-
sioned blockchain system that adopts practices suggested
in BFTSmart [6], and employs the fast Zyzzyva protocol.
We observe that the system-centric design of ResilientDB
can use the costly three-phase Pbft protocol (of which two
phases require quadratic communication among replicas),
while still outperforming systems utilizing Zyzzyva (a single-
phase protocol with linear communication among replicas).
Decades of academic research [29, 30] have helped the
community in designing efficient distributed databases and
applications. In this tutorial, we study existing practices
to design an optimal permissioned blockchain fabric. These
practices include the use of speculation [34, 36], execute-
order or order-execute paradigms [3], component modular-
3442
ity [6], transaction batching or streaming [2, 11], and out-of-
order message processing [28]. Further, we will discuss how
data storage, data maintenance, data retrieval, and NoSQL
and relational database support is provided by our state-of-
the-art permissioned blockchain fabrics [28]. We will demon-
strate this functionality via a user interface that can ease
query processing and data analysis.
Challenges and our Vision. As outlined above, the key
component of any permissionless and permissioned block-
chain remains the underlying BFT consensus protocol that
provides reliable replication. Unfortunately, these protocols
are challenged by the scalability and performance required
by many modern big-data-driven applications. In specific,
we see that there is no obvious way to scale up BFT con-
sensus: adding more replicas will only increase the cost of
replication and decrease the throughput of the system, even
when using the most efficient consensus protocols.
We will close our tutorial by discussing recent steps to-
ward the design of new fault-tolerant architectures that step
away from the full-replicated nature of blockchains, this to
increase scalability and the ability to serve big-data-driven
applications. To put our vision in practice, we will first look
at two low-level techniques, cluster-sending [31] and delayed-
replication [32]. Next, we look at high-level designs enabled
by these techniques to provide high-performance parallelized
consensus [24, 25] and to provide high-performance consen-
sus in sharded and geo-scale aware architectures [27].
3. BIOGRAPHICAL SKETCHES
Suyash Gupta is a Ph.D. Candidate at the Computer
Science Department at University of California, Davis. At
UC Davis, he is a senior member of Exploratory Systems
Lab and works under the supervision of Prof. Sadoghi. He
also works as the Lead Architect at the blockchain company
Moka Blox and ResilientDB. He also holds a Master of Sci-
ence degree from Purdue University and a Master of Sci-
ence (Research) degree from Indian Institute of Technology
Madras. His current research focuses on attaining safe and
efficient fault-tolerant consensus in distributed and block-
chain systems. He also has published works that present
efficient compiler optimizations and designs for parallel and
distributed algorithms.
Jelle Hellings finished his graduate studies at the Eind-
hoven University of Technology, Netherlands in 2011, with
a final research project focused on external memory algo-
rithms for indexing trees and directed acyclic graphs. He
then moved to Hasselt University, Belgium, where he did
his doctoral research in the Databases and Theoretical Com-
puter Science research group. He finished his doctoral re-
search on the expressive power of query languages in 2018.
Since the summer of 2018, Jelle is a Postdoc Fellow at UC
Davis in the Exploratory Systems Lab led by Prof. Sadoghi.
His current research focus is on the theoretical bounds of
consensus protocols in malicious environments and, more
general, on exploring new directions for replicated systems
in malicious environment.
Sajjad Rahnama is a Ph.D. student at the Computer
Science Department of the University of California Davis
supervised by Prof. Sadoghi. He is also a member of Ex-
ploratory Systems Lab. He also works as the System De-
signer at the blockchain company called Moka Blox and is
the main developer for ResilientDB. His current research fo-
cuses on secure transaction processing and designing global-
scale fault-tolerant protocols, distributed systems, and their
applications in blockchain. He holds a B.Sc. in Computer
Science from Amirkabir University of Technology, Tehran,
Iran. Prior to starting his Ph.D., he worked in several tech
companies as a infrastructure engineer and developer.
Mohammad Sadoghi is an Assistant Professor in the
Computer Science Department at UC Davis. He leads the
ExpoLab research group with the aim to pioneer a dis-
tributed ledger that unifies secure transactional and real-
time analytical processing (L-Store), all centered around
a democratic and decentralized computational model (Re-
silientDB). He has co-founded a blockchain company called
Moka Blox LLC, a ResilientDB spin-off. He has over 80
publications in leading database conferences/journals and
34 filed U.S. patents. He has co-authored a book enti-
tled “Transaction Processing on Modern Hardware”, Mor-
gan & Claypool Synthesis Lectures on Data Management,
and currently co-authoring a book entitled “Fault-tolerant
Distributed Transactions on Blockchain”, Morgan & Clay-
pool series.
Acknowledgments
This tutorial is based on the outline of our upcoming book
on fault-tolerant transaction processing on blockchains [26].
Furthermore, this tutorial is an evolution of a tutorial pre-
sented at Middleware 2019 [22]. We have updated that tu-
torial with new and upcoming techniques and insights.
This work is partially supported by the U.S. Department
of Energy, Office of Science, Office of Small Business Inno-
vation Research, under Award Number DE-SC0020455.
4. REFERENCES
[1] I. Abraham, S. Devadas, D. Dolev, K. Nayak, and
L. Ren. Synchronous byzantine agreement with
expected O(1) rounds, expected O(n2)
communication, and optimal resilience, 2018.
[2] M. J. Amiri, D. Agrawal, and A. E. Abbadi. CAPER:
A cross-application permissioned blockchain. PVLDB,
12(11):1385–1398, 2019.
[3] E. Androulaki, A. Barger, V. Bortnikov, C. Cachin,
K. Christidis, A. De Caro, D. Enyeart, C. Ferris,
G. Laventman, Y. Manevich, S. Muralidharan,
C. Murthy, B. Nguyen, M. Sethi, G. Singh, K. Smith,
A. Sorniotti, C. Stathakopoulou, M. Vukoli´c, S. W.
Cocco, and J. Yellick. Hyperledger Fabric: A
distributed operating system for permissioned
blockchains. In Proceedings of the Thirteenth EuroSys
Conference, pages 30:1–30:15. ACM, 2018.
[4] P.-L. Aublin, R. Guerraoui, N. Kneˇzevi´c, V. Qu´ema,
and M. Vukoli´c. The next 700 bft protocols. ACM
Trans. Comput. Syst., 32(4):12:1–12:45, 2015.
[5] P.-L. Aublin, S. B. Mokhtar, and V. Qu´ema. RBFT:
Redundant byzantine fault tolerance. In IEEE 33rd
International Conference on Distributed Computing
Systems, pages 297–306. IEEE, 2013.
[6] A. Bessani, J. Sousa, and E. E. Alchieri. State
machine replication for the masses with BFT-SMART.
In 44th Annual IEEE/IFIP International Conference
on Dependable Systems and Networks, pages 355–362.
IEEE, 2014.
[7] E. Brewer. CAP twelve years later: How the “rules”
have changed. Computer, 45(2):23–29, 2012.
3443
[8] M. Castro and B. Liskov. Practical byzantine fault
tolerance and proactive recovery. ACM Trans.
Comput. Syst., 20(4):398–461, 2002.
[9] H. Dang, T. T. A. Dinh, D. Loghin, E.-C. Chang,
Q. Lin, and B. C. Ooi. Towards scaling blockchain
systems via sharding. In Proceedings of the 2019
International Conference on Management of Data,
pages 123–140. ACM, 2019.
[10] R. A. DeMillo, N. A. Lynch, and M. J. Merritt.
Cryptographic protocols. In Proceedings of the
Fourteenth Annual ACM Symposium on Theory of
Computing, pages 383–400. ACM, 1982.
[11] T. T. A. Dinh, J. Wang, G. Chen, R. Liu, B. C. Ooi,
and K.-L. Tan. BLOCKBENCH: A framework for
analyzing private blockchains. In Proceedings of the
2017 ACM International Conference on Management
of Data, pages 1085–1100. ACM, 2017.
[12] D. Dolev. Unanimity in an unknown and unreliable
environment. In 22nd Annual Symposium on
Foundations of Computer Science, pages 159–168.
IEEE, 1981.
[13] D. Dolev. The byzantine generals strike again. J.
Algorithms, 3(1):14–30, 1982.
[14] D. Dolev and R. Reischuk. Bounds on information
exchange for byzantine agreement. J. ACM,
32(1):191–204, 1985.
[15] D. Dolev and H. Strong. Authenticated algorithms for
byzantine agreement. SIAM J. Comput.,
12(4):656–666, 1983.
[16] W. W. Eckerson. Data quality and the bottom line:
Achieving business success through a commitment to
high quality data. Technical report, The Data
Warehousing Institute, 101communications LLC.,
2002.
[17] M. El-Hindi, C. Binnig, A. Arasu, D. Kossmann, and
R. Ramamurthy. BlockchainDB: A shared database on
blockchains. PVLDB, 12(11):1597–1609, 2019.
[18] M. J. Fischer and N. A. Lynch. A lower bound for the
time to assure interactive consistency. Inform.
Process. Lett., 14(4):183–186, 1982.
[19] M. J. Fischer, N. A. Lynch, and M. S. Paterson.
Impossibility of distributed consensus with one faulty
process. J. ACM, 32(2):374–382, 1985.
[20] S. Gilbert and N. Lynch. Brewer’s conjecture and the
feasibility of consistent, available, partition-tolerant
web services. ACM SIGACT News, 33(2):51–59, 2002.
[21] G. Greenspan. Multichain private blockchain, 2015.
[22] S. Gupta, J. Hellings, S. Rahnama, and M. Sadoghi.
An in-depth look of BFT consensus in blockchain:
Challenges and opportunities. In Proceedings of the
20th International Middleware Conference Tutorials,
pages 6–10. ACM, 2019.
[23] S. Gupta, J. Hellings, S. Rahnama, and M. Sadoghi.
Proof-of-execution: Reaching consensus through
fault-tolerant speculation, 2019.
[24] S. Gupta, J. Hellings, and M. Sadoghi. Brief
announcement: Revisiting consensus protocols
through wait-free parallelization. In 33rd International
Symposium on Distributed Computing (DISC 2019),
volume 146, pages 44:1–44:3. Schloss
Dagstuhl–Leibniz-Zentrum fuer Informatik, 2019.
[25] S. Gupta, J. Hellings, and M. Sadoghi. Scaling
blockchain databases through parallel resilient
consensus paradigm, 2019.
[26] S. Gupta, J. Hellings, and M. Sadoghi. Fault-Tolerant
Distributed Transactions on Blockchains. Synthesis
Lectures on Data Management. Morgan & Claypool
Publishers, 2020. (to appear).
[27] S. Gupta, S. Rahnama, J. Hellings, and M. Sadoghi.
ResilientDB: Global scale resilient blockchain fabric.
PVLDB, 13(6):868–883, 2020.
[28] S. Gupta, S. Rahnama, and M. Sadoghi. Permissioned
blockchain through the looking glass: Architectural
and implementation lessons learned. In 40th
International Conference on Distributed Computing
Systems. IEEE, 2020.
[29] R. Harding, D. Van Aken, A. Pavlo, and
M. Stonebraker. An evaluation of distributed
concurrency control. PVLDB, 10(5):553–564, 2017.
[30] S. Harizopoulos, D. J. Abadi, S. Madden, and
M. Stonebraker. OLTP through the looking glass, and
what we found there. In Proceedings of the 2008 ACM
SIGMOD International Conference on Management of
Data, pages 981–992. ACM, 2008.
[31] J. Hellings and M. Sadoghi. Brief announcement: The
fault-tolerant cluster-sending problem. In 33rd
International Symposium on Distributed Computing
(DISC 2019), volume 146, pages 45:1–45:3. Schloss
Dagstuhl–Leibniz-Zentrum fuer Informatik, 2019.
[32] J. Hellings and M. Sadoghi. Coordination-free
byzantine replication with minimal communication
costs. In 23rd International Conference on Database
Theory, volume 155, pages 17:1–17:20. Schloss
Dagstuhl–Leibniz-Zentrum fuer Informatik, 2020.
[33] T. N. Herzog, F. J. Scheuren, and W. E. Winkler.
Data Quality and Record Linkage Techniques. Springer
New York, 2007.
[34] R. Kotla, L. Alvisi, M. Dahlin, A. Clement, and
E. Wong. Zyzzyva: Speculative byzantine fault
tolerance. In Proceedings of Twenty-first ACM
SIGOPS Symposium on Operating Systems Principles,
pages 45–58. ACM, 2007.
[35] L. Lamport, R. Shostak, and M. Pease. The byzantine
generals problem. ACM Trans. Program. Lang. Syst.,
4(3):382–401, 1982.
[36] J.-P. Martin and L. Alvisi. Fast byzantine consensus.
IEEE Trans. Depend. Secure Comput., 3(3):202–215,
2006.
[37] S. Moran and Y. Wolfstahl. Extended impossibility
results for asynchronous complete networks. Inform.
Process. Lett., 26(3):145–151, 1987.
[38] S. Nakamoto. Bitcoin: A peer-to-peer electronic cash
system, 2009.
[39] S. Nathan, C. Govindarajan, A. Saraf, M. Sethi, and
P. Jayachandran. Blockchain meets database: Design
and implementation of a blockchain relational
database. PVLDB, 12(11):1539–1552, 2019.
[40] F. Nawab and M. Sadoghi. Blockplane: A global-scale
byzantizing middleware. In 35th International
Conference on Data Engineering (ICDE), pages
124–135. IEEE, 2019.
[41] M. Pease, R. Shostak, and L. Lamport. Reaching
agreement in the presence of faults. J. ACM,
27(2):228–234, 1980.
[42] T. C. Redman. The impact of poor data quality on the
typical enterprise. Commun. ACM, 41(2):79–82, 1998.
[43] G. Taubenfeld and S. Moran. Possibility and
impossibility results in a shared memory environment.
Acta Inform., 33(1):1–20, 1996.
[44] G. Wood. Ethereum: a secure decentralised
generalised transaction ledger, 2016. EIP-150 revision.
[45] M. Yin, D. Malkhi, M. K. Reiter, G. G. Gueta, and
I. Abraham. HotStuff: BFT consensus with linearity
and responsiveness. In Proceedings of the ACM
Symposium on Principles of Distributed Computing,
pages 347–356. ACM, 2019.
3444
Article
Federated Learning (FL) is a novel distributed, privacy-preserving machine learning paradigm. Conventional FL suffers from drawbacks such as single point of failure and client drift. Blockchain is a distributed computing architecture famous for decentralization, transparency, and traceability. Incorporating blockchain as the underlying basis for FL decentralizes the FL process and brings opportunities to resolve the drawbacks. However, there still remain challenges to fulfilling FL with blockchain, regarding effectiveness, efficiency, and security. In this paper, we propose a new blockchain system for FL, called FedChain. To mitigate client drift and accelerate training, we present a clustered semi-asynchronous method for model aggregation. To optimize the local training in FL, we introduce a knowledge transfer method using other clients on the peer-to-peer network of blockchain. Moreover, we implement an access control mechanism to store and transmit models safely and efficiently. Extensive experiments on various benchmark datasets show that FedChain achieves superior results in accuracy, convergence, throughput, and latency.
Preprint
Full-text available
In the realm of Machine Vision, the progress in cinematography and image processing has empowered computers to comprehend images and videos. Cameras and infrared (IR) technology have become nearly omnipresent, facilitating the acquisition of crucial information about individuals and their surroundings. They are applied across various sectors, including high-security access, biometric cards, and modern surveillance systems embedded in the Internet of Things (IoT) ecosystem. However, an issue arises with the surge in information and its dissemination, raising concerns regarding safety, confidentiality, and secure storage as these systems interact with others, potentially leading to problems. To tackle these challenges, experts in computer vision are exploring the integration of blockchain technology. This article presents an overview of blockchain applications in computer vision, encompassing biometrics, video surveillance, and the sharing of image and video data. The objective is to demonstrate how blockchain could augment the safety, privacy, and reliability of these applications. The paper also delves into the tools and methodologies contributing to these goals, shedding light on prospective developments. Essentially, this review seeks to furnish clear insights into the collaboration between blockchain and computer vision. It achieves this by examining the latest advancements, existing challenges, and opportunities for leveraging visual data in a blockchain-driven environment. Ultimately, the focus is on understanding how these technologies can harmonize to influence the future landscape of Machine Vision.
Article
Full-text available
This study aims to explore the construction of a personalized recommendation system (PRS) based on deep learning under the hybrid blockchain model to further improve the performance of the PRS. Blockchain technology is introduced and further improved to address security problems such as information leakage in PRS. A Delegated Proof of Stake-Byzantine Algorand-Directed Acyclic Graph consensus algorithm, namely PBDAG consensus algorithm, is designed for public chains. Finally, a personalized recommendation model based on the hybrid blockchain PBDAG consensus algorithm combined with an optimized back propagation algorithm is constructed. Through simulation, the performance of this model is compared with practical Byzantine Fault Tolerance, Byzantine Fault Tolerance, Hybrid Parallel Byzantine Fault Tolerance, Redundant Byzantine Fault Tolerance, and Delegated Byzantine Fault Tolerance. The results show that the model algorithm adopted here has a lower average delay time, a data message delivery rate that is stable at 80%, a data message leakage rate that is stable at about 10%, and a system classification prediction error that does not exceed 10%. Therefore, the constructed model not only ensures low delay performance but also has high network security performance, enabling more efficient and accurate interaction of information. This solution provides an experimental basis for the information security and development trend of different types of data PRSs in various fields.
Article
With the proliferation of pump-and-dump schemes (P&Ds) in the cryptocurrency market, it becomes imperative to detect such fraudulent activities in advance to alert potentially susceptible investors. In this paper, we focus on predicting the pump probability of all coins listed in the target exchange before a scheduled pump time, which we refer to as the target coin prediction task. Firstly, we conduct a comprehensive study of the latest 709 P&D events organized in Telegram from Jan. 2019 to Jan. 2022. Our empirical analysis reveals some interesting patterns of P&Ds, such as that pumped coins exhibit intra-channel homogeneity and inter-channel heterogeneity. Here channel refers a form of group in Telegram that is frequently used to coordinate P&D events. This observation inspires us to develop a novel sequence-based neural network, dubbed SNN, which encodes a channel's P&D event history into a sequence representation via the positional attention mechanism to enhance the prediction accuracy. Positional attention helps to extract useful information and alleviates noise, especially when the sequence length is long. Extensive experiments verify the effectiveness and generalizability of proposed methods. Additionally, we release the code and P&D dataset on GitHub https://github.com/Bayi-Hu/Pump-and-Dump-Detection-on-Cryptocurrency, and regularly update the dataset.
Article
Full-text available
Recent developments in blockchain technology have inspired innovative new designs in resilient distributed and database systems. At their core, these blockchain applications typically use Byzantine fault-tolerant consensus protocols to maintain a common state across all replicas, even if some replicas are faulty or malicious. Unfortunately, existing consensus protocols are not designed to deal with geo-scale deployments in which many replicas spread across a geographically large area participate in consensus. To address this, we present the Geo-Scale Byzantine Fault-Tolerant consensus protocol (GeoBFT). GeoBFT is designed for excellent scalability by using a topological-aware grouping of replicas in local clusters, by introducing parallelization of consensus at the local level, and by minimizing communication between clusters. To validate our vision of high-performance geo-scale resilient distributed systems, we implement GeoBFT in our efficient ResilientDB permissioned blockchain fabric. We show that GeoBFT is not only sound and provides great scalability, but also outperforms state-of-the-art consensus protocols by a factor of six in geo-scale deployments.
Conference Paper
Full-text available
Since the introduction of Bitcoin---the first wide-spread application driven by blockchains---the interest of the public and private sector in blockchains has skyrocketed. At the core of this interest are the ways in which blockchains can be used to improve data management, e.g., by enabling federated data management via decentralization, resilience against failure and malicious actors via replication and consensus, and strong data provenance via a secured immutable ledger. In practice, high-performance blockchains for data management are usually built in permissioned environments in which the participants are vetted and can be identified. In this setting, blockchains are typically powered by Byzantine fault-tolerant consensus protocols. These consensus protocols are used to provide full replication among all honest blockchain participants by enforcing an unique order of processing incoming requests among the participants. In this tutorial, we take an in-depth look at Byzantine fault-tolerant consensus. First, we take a look at the theory behind replicated computing and consensus. Then, we delve into how common consensus protocols operate. Finally, we take a look at current developments and briefly look at our vision moving forward.
Article
Full-text available
Despite recent intensive research, existing blockchain systems do not adequately address all the characteristics of distributed applications. In particular, distributed applications collaborate with each other following service level agreements (SLAs) to provide different services. While collaboration between applications, e.g., cross-application transactions, should be visible to all applications, the internal data of each application, e.g, internal transactions, might be confidential. In this paper, we introduce CAPER, a permissioned blockchain system to support both internal and cross-application transactions of collaborating distributed applications. In CAPER, the blockchain ledger is formed as a directed acyclic graph where each application accesses and maintains only its own view of the ledger including its internal and all cross-application transactions. CAPER also introduces three consensus protocols to globally order cross-application transactions between applications with different internal consensus protocols. The experimental results reveal the efficiency of CAPER in terms of performance and scalability.
Conference Paper
Full-text available
We present HotStuff, a leader-based Byzantine fault-tolerant replication protocol for the partially synchronous model. Once network communication becomes synchronous, HotStuff enables a correct leader to drive the protocol to consensus at the pace of actual (vs. maximum) network delay--a property called responsiveness---and with communication complexity that is linear in the number of replicas. To our knowledge, HotStuff is the first partially synchronous BFT replication protocol exhibiting these combined properties. Its simplicity enables it to be further pipelined and simplified into a practical, concise protocol for building large-scale replication services.
Conference Paper
Full-text available
Existing blockchain systems scale poorly because of their distributed consensus protocols. Current attempts at improving blockchain scalability are limited to cryptocurrency. Scaling blockchain systems under general workloads (i.e., non-cryptocurrency applications) remains an open question. This work takes a principled approach to apply sharding to blockchain systems in order to improve their transaction throughput at scale. This is challenging, however, due to the fundamental difference in failure models between databases and blockchain. To achieve our goal, we first enhance the performance of Byzantine consensus protocols, improving individual shards' throughput. Next, we design an efficient shard formation protocol that securely assigns nodes into shards. We rely on trusted hardware, namely Intel SGX, to achieve high performance for both consensus and shard formation protocol. Third, we design a general distributed transaction protocol that ensures safety and liveness even when transaction coordinators are malicious. Finally, we conduct an extensive evaluation of our design both on a local cluster and on Google Cloud Platform. The results show that our consensus and shard formation protocols outperform state-of-the-art solutions at scale. More importantly, our sharded blockchain reaches a high throughput that can handle Visa-level workloads, and is the largest ever reported in a realistic environment.
Conference Paper
Full-text available
The byzantine fault-tolerance model captures a wide-range of failures-common in real-world scenarios-such as ones due to malicious attacks and arbitrary software/hardware errors. We propose Blockplane, a middleware that enables making existing benign systems tolerate byzantine failures. This is done by making the existing system use Blockplane for durability and as a communication infrastructure. Blockplane proposes the following: (1) A middleware and communication infrastructure to make an entire benign protocol byzantine fault-tolerant, (2) A hierarchical locality-aware design to minimize the number of wide-area messages, (3) A separation of fault-tolerance concerns to enable designs with higher performance. I. INTRODUCTION A byzantine failure model [11] is a model of arbitrary failures that includes-in addition to crashes-unexpected behavior due to software and hardware malfunctions, malicious breaches, and violation of trust between participants. It is significantly more difficult to develop byzantine fault-tolerant protocols compared to benign (non-byzantine) protocols. This poses a challenge to organizations that want to adopt byzantine fault-tolerant software solutions. This challenge is exacerbated with the need of many applications to be globally distributed. With global distribution, the wide-area latency between participants amplifies the performance overhead of byzantine fault-tolerant protocols. To overcome the challenges of adopting byzantine fault-tolerant software solutions, we propose pushing down the byzantine fault-tolerance problem to the communication layer rather than the application/storage layer. Our proposal, Block-plane, is a communication infrastructure that handles the delivery of messages from one node to another. Blockplane exposes an interface of log-commit, send, and receive operations to be used by nodes to both persist their state and communicate with each other. Blockplane adopts a locality-aware hierarchical design due to our interest in supporting efficient byzantine fault-tolerance in global-scale environments. Hierarchical designs have recently been shown to perform well in global-scale settings [15]. Blockplane optimizes for the communication latency by performing as much computation as possible locally and only communicate across the wide-area link when necessary. In the paper, we distinguish between two types of failures. The first is independent byzantine failures that are akin to traditional byzantine failures which affect each node independently (the failure of one node does not correlate with the failure of another node). The second type of failures is benign
Chapter
We present new protocols for Byzantine agreement in the synchronous and authenticated setting, tolerating the optimal number of f faults among n=2f+1 parties. Our protocols achieve an expected O(1) round complexity and an expected O(n2)O(n^2) communication complexity. The exact round complexity in expectation is 10 for a static adversary and 16 for a strongly rushing adaptive adversary. For comparison, previous protocols in the same setting require expected 29 rounds.
Article
In this paper, we design and implement the first-ever decentralized replicated relational database with blockchain properties that we term blockchain relational database. We highlight several similarities between features provided by blockchain platforms and a replicated relational database, although they are conceptually different, primarily in their trust model. Motivated by this, we leverage the rich features, decades of research and optimization, and available tooling in relational databases to build a blockchain relational database. We consider a permissioned blockchain model of known, but mutually distrustful organizations each operating their own database instance that are replicas of one another. The replicas execute transactions independently and engage in decentralized consensus to determine the commit order for transactions. We design two approaches, the first where the commit order for transactions is agreed upon prior to executing them, and the second where transactions are executed without prior knowledge of the commit order while the ordering happens in parallel. We leverage serializable snapshot isolation (SSI) to guarantee that the replicas across nodes remain consistent and respect the ordering determined by consensus, and devise a new variant of SSI based on block height for the latter approach. We implement our system on PostgreSQL and present detailed performance experiments analyzing both approaches.
Article
In this paper we present BlockchainDB, which leverages blockchains as a storage layer and introduces a database layer on top that extends blockchains by classical data management techniques (e.g., sharding) as well as a standardized query interface to facilitate the adoption of blockchains for data sharing use cases. We show that by introducing the additional database layer, we are able to improve the performance and scalability when using blockchains for data sharing and also massively decrease the complexity for organizations intending to use blockchains for data sharing.