Abstract

ResilientDB is a distributed blockchain framework that is open-source, lightweight, modular, and highly performant. It is written in C++ with 25,000 LOC (not including the experimental features noted in our roadmap that totals to 50,000 LOC). It allows for easy integration with various Byzantine Fault-Tolerant (BFT) and Crash Fault-Tolerant (CFT) consensus protocols as its underlying ordering service.

Proposal

ResilientDB is an open-source, lightweight, modular, and highly performant distributed blockchain framework that provides the following desirable features:

  • Provides a scalable client-server architecture. Each developer can use the ResilientDB framework to deploy a replicated service acting as a service. The developer can choose the desired number of replicas and the number of clients its system should tolerate. 
  • Provides native integration with PBFT consensus protocol – arguably the most popular BFT consensus protocol. PBFT helps replicas reach an agreement for ordering the client's requests.
  • Provides a mechanism to simulate the failure of different replicas (including the leader).
  • Provides a correct implementation of the view-change protocol that replaces a faulty (or malicious) leader and moves all replicas to the new view.
  • Provides checkpoint and recovery protocols to facilitate garbage collection, recovery of failed replicas, and durably logging of the blockchain state.
  • Eases development and testing of newer and optimized BFT and CFT consensus protocols. 
  • Provides clients with support for three different application interfaces:
    • Key-Value Stores - where client transactions include key-value pairs.
    • Smart Contracts - where clients issue smart contracts in Solidity for processing.
    • UTXO - where clients issue unspent transactions similar to ones in Bitcoin.
  • Facilitates benchmarking system/protocol performance with the help of existing benchmarks, such as YCSB [SoCC’10] and Diablo [EuroSys’23].
  • Stores non-volatile ledger (blockchain) in memory and for further durability, provides APIs to store both client data and blockchain in LevelDB and RocksDB. 


ResilientDB Software Stack

The high-level architecture of ResilientDB consists of the following five core layers: SDK, Interface/API, Platform, Execution, and Chain Layers

  1. SDK Layer: Python SDK
  2. Interface Layer: Key-Value, Solidity Smart Contract, Unspent Transaction Output (UTXO) Model, ResilientDB Database Connectivity (RDBC) API
  3. Platform Layer: Consensus Manager Architecture (ordering, recovery, network, chain management)
  4. Execution Layer: Transaction Manager Design (Runtime)
  5. Chain Layer: Chain State & Storage Manager Design (Durability)
  6. Installing & Deploying ResilientDB

Background

The goal of a blockchain system is to enable a decentralized client-server architecture where the server is not controlled by a single party and the state is logged in an immutable log (blockchain). To realize this goal, each blockchain system implements the server as a multi-party replicated subsystem; each party has access to all the states and data, and all the parties are replicas of each other. Each blockchain system requires its replicas to (1) batch incoming client transactions in a block, (2) propose the block to all the replicas, and (3) reach a consensus on the order for processing these blocks of client transactions. As the participants do not trust each other, a blockchain system runs a BFT consensus protocol to order blocks of transactions and to determine which proposer’s block is appended to the blockchain. Thus, a blockchain is a linked list of blocks where each block includes the hash of the previous block in the chain. Each block stores a set of client transactions along with other metadata. 

The choice of BFT consensus protocol divides the blockchain systems into two groups: permissionless and permissioned. Permissionless blockchain systems (like Bitcoin) advocate pseudo-anonymity by identifying each replica through its public key; each replica can have multiple public-private key pairs. Permissioned blockchains, instead, verify the identity of each replica before it can participate in the consensus. 

Permitting pseudo-anonymity forces permissionless blockchains to prevent malicious replicas from assuming multiple identities. As a result, they need BFT consensus protocols that can assign some weight to each replica. A popular way to do so is to use the Proof-of-Work (PoW) algorithm, which forces replicas to compete with each other to get the opportunity to append a new block to the chain. This competition requires finding a value that can help to reach a system-defined hash. To increase the probability of only one winner, PoW forces replicas to expend their computational resources, which leads to massive energy wastage. Additionally, this results in extremely low throughput (1 block every 10 minutes), and there is always a probability of multiple replicas finding the correct value, which temporarily forks the blockchain. A fork in the blockchain is a precarious situation as two forks may have conflicting transactions, and replicas may be split on which fork is the correct chain. Thus, PoW-based systems wait for forks to resolve before marking a transaction as ordered. An alternative solution is to employ the Proof-of-Stake (PoS) protocol, which forces replicas to compete by depositing their monetary resources. Although PoS does not waste energy, it has a higher probability of blockchain forks as, unlike PoW, a replica has nothing to lose, and it can continue proposing blocks on all the forks.  

Permissioned blockchains have an opportunity to eliminate these challenges as they employ traditional BFT consensus protocols, which neither waste energy nor allow forks in the blockchain. These BFT protocols require replicas to vote on proposals, and if a proposal receives votes from a two-thirds majority of replicas, it is appended to the chain. 

Rationale

Despite the visible benefits of permissioned blockchain systems and extensive academic research, there is a lack of open-source permissioned blockchain systems that meet all the following desirable goals: (1) Employs a BFT consensus protocol that is safe under asynchrony and live under partial synchrony. (2) Supports out-of-order consensus of client transactions. (3) Supports simulating replica failures and replacing Byzantine replicas. (4) Provides easy integration with standard benchmarking frameworks. (5) Reduces the time needed by a researcher/developer to design/test a new consensus protocol. 

Several existing permissioned blockchain designs fall short of meeting these goals. For example, the Linux Foundation’s Hyperledger project employs the Raft consensus protocol, which is not designed to handle malicious attacks. Additionally, Hyperledger does not allow simulating replica failures. HotStuff provides a simple replicated state machine that implements HotStuff consensus protocol but does not meet goals (2) to (5). Ethereum Enterprise Alliance helps organizations set up permissioned blockchains, but its codebase is not open-source. 

ResilientDB aims to fill this gap by providing researchers/developers access to a framework that helps to design and test consensus protocols and blockchain applications. ResilientDB’s lightweight framework yields at least 10x higher throughput than existing permissioned blockchains. ResilientDB allows developers to insert and simulate crash or Byzantine failures and run recovery procedures that can bring replicas to the common state.

Roadmap

In addition to the features stated in the aforementioned proposal, we set out the following roadmap for ResilientDB. Our vision is to complete these tasks as part of our roadmap.

  • Build a CI/CD pipeline that automates build, test, and code integration. Our focus is on automating the testing of BFT and blockchain systems, as they can undergo arbitrary failures. Thus, we plan to provide access to a suite of unit test cases that can ensure the system is safe and available under guarantees provided by the PBFT protocol.
  • Create a developer wiki that accompanies the existing codebase. At present, we provide documentation as inline code comments and ResilientDB blogs.
  • Set up a code review process for new releases and pull requests.
  • Add use cases and applications for blockchain systems to attract developers to the project. We are working towards designing an NFT system on top of ResilientDB.
  • Enable scalable geo-scale consensus in ResilientDB by implementing the GeoBFT [VLDB’20] protocol. Unlike PBFT, GeoBFT protocol prevents degradation of throughput and latency if the replicas are spread across the globe. 
  • Enable speculative consensus in ResilientDB by implementing the PoE [EDBT’21] protocol. Unlike PBFT, which requires two phases of quadratic communication complexity (all-to-all communication among the replicas), PoE requires two phases of linear communication complexity and rollback transactions when necessary.
  • Enable parallel consensus in ResilientDB by implementing the RCC [ICDE’21] protocol. Unlike PBFT, which supports only one proposer and orders only one block of transactions per round of consensus, RCC supports multiple proposers and orders multiple blocks per round of consensus. This helps to utilize resources across all replicas.
  • Serve as a ledger notary to protect existing PoS-based blockchain systems, such as Ethereum, Algorand, Quorum, and so on, from long-range attacks. These long-range attacks aim to rewrite the ledger, and the PoC protocol prevents the same.
  • Enable integration with SGX toolkit to design consensus protocols, such as FlexiTrust protocols [EuroSys’23], that employ trusted execution environments to prevent malicious replicas from equivocation.
  • Enable data sharding by dividing replicas into subsets such that each subset manages a unique data shard. To support multi-shard transactions, we plan to integrate the RingBFT [EDBT’22] consensus protocol.
  • Enable view-change-less consensus as the view-change mechanism reduces the system throughput to zero; PBFT requires replicas to halt processing new client transactions until the successful replacement of the faulty leader and establishment of a new view. PVP [arxiv’23] presents a multi-leader view-change-less consensus protocol.


Initial Goals

  • To build a homogeneous community of rigorous developers and users of the ResilientDB.
  • To advocate the use of multi-party decentralized platforms that can guard against malicious attacks.


Current Status

The current version of ResilientDB was founded at ExpoLab, UC Davis, in 2018 and was open-sourced in late 2019. Version v1.0 of ResilientDB was released on November 24, 2019, and on September 30, 2021, we released ResilientDB v-3.0. In 2022, ResilientDB was rewritten and re-architected, paving the way for a new sustainable foundation, which we refer to as NexRes (Next Generation ResilientDB). On September 30, 2022, we released NexRes-v1.0.0.

Meritocracy

The current developers of ResilientDB are aware of the meritocracy of Apache products. We chose Apache because it upholds the principle of open-source development. Existing ResilientDB developers have contributed in the past to open-source projects and will honor the principles of ASF.

Community

The growing interest in blockchain technology has led to extensive research in untrusted multi-party decentralized systems. This has led to a spurt of new blockchain startups and a rise in the number of distributed systems developers. However, none of these organizations gives developers the freedom to design and test new protocols and data structures. Our goal is to attract this set of developers and researchers and to foster open-source collaboration. Radix DLT is one such startup that has utilized ResilientDB to test its designs. A discussion on this collaboration can be found here: analysis and partnership.

Core Developers

Although the core development of ResilientDB has occurred at the Expolatory Systems Lab at UC Davis and at UC Berkeley, ResilientDB has a diverse developer community with active contributors from developers at various universities (UC Irvine, UPenn, UC Santa Barbara, McMaster, and Umm Al-Qura University) and industries (Oracle, AWS, Radix DLT, and Cisco).


Alignment

ASF (Apache Software Foundation) is a natural home for ResilientDB as it hosts several distributed systems projects, such as Thrift, Zookeeper, and Ratis. Although these projects cater to crash failures, like ResilientDB, these projects also aim to deploy a reliable distributed system. As Byzantine failures are a superset of crash failures, we believe ResilientDB aligns with the interest of the existing Apache community. ResilientDB offers the existing Apache community a mechanism to deploy distributed client-server applications, which can guard against arbitrary faults, namely, Byzantine attacks.

Known Risks

Orphaned Products

Given the frequency of research surrounding ResilientDB, we envision ResilientDB as a long-term project of interest at Exploratory Systems Lab, UC Davis, which has also been used successfully as an educational tool inside the classroom at both graduate and undergraduate levels since 2018. Additionally, ResilientDB is being deployed in several upcoming projects at UC Berkeley. Thus, there is a small probability of this project being abandoned. Further, the growing use of ResilientDB at other universities reduces its chances of being orphaned. Additionally, blockchain technology is still at a nascent stage, and with Apache’s tag, ResilientDB will attract further attention.



Inexperience with Open Source:

The core team is fully aware of the merits of open source and, in the past, has contributed to existing open source projects. Moreover, ResilientDB was open-sourced in 2019 and has remained open-source for the past five years.

Homogenous Developers

As described in the Alignment section, ResilientDB currently receives support from student and postdoc researchers and engineers at UC Davis, UC Berkeley, UPenn, and UC Irvine. However, many other universities UC Santa Barbara, McMaster, and Umm Al-Qura University have cloned/forked ResilientDB and continued to enhance it in their own private repositories. Additionally, ResilientDB has been used by developers at Radix DLT, Oracle, AWS, Cisco, and Mysten Labs. We plan to continue encouraging developers from all over the globe to contribute to the ResilientDB codebase.


Developers

ResilientDB is developed and maintained by researchers primarily from UC Davis, UC Berkeley, UPenn, and UC Irvine along with broader collaborators from both universities (e.g., McMaster, UPenn) and industries (e.g., Oracle, Radix Works Ltd, Mysten Labs).

Relationships with Other Apache Products

Numerous established Apache projects, including Apache Ratis, and Apache Zookeeper, are aimed at addressing fault tolerance. This project continues to align with this direction by targeting Byzantine fault tolerance. As Byzantine failures are a superset of crash failures, we believe this project will interest the existing Apache community, as it complements the current solution stack offered by Apache. 


Documentation

As stated in the Roadmap, we provide documentation regarding ResilientDB in the codebase, on the Webpage, and on our Blog. Additionally, we plan to create a developer wiki. 


Initial Source

The initial source is available on GitHub under the Apache License 2.0

Source and Intellectual Property Submission Plan

All code currently hosted in the ResilientDB public folder on GitHub will be contributed.




External Dependencies

All dependencies (libevent, Boost), which are standard widely-used open-source libraries, have Apache-compatible licenses. These include BSD, CDDL, CPL, MPL, and MIT-licensed. dependencies.

 

Cryptography

Traffic Server uses cryptopp to implement signatures support. There is no other cryptographic code in the source tree.

Required Resources

Mailing lists

  • resilientdb-private (with moderated subscriptions)
  • resilientdb-dev
  • resilientdb-commits
  • resilientdb-user


Git Repositories

Git is the preferred source control system: git://git.apache.org/resilientdb

Issue Tracking & Project Management

Trello ResilientDB


Initial Committers

Mohammad Sadoghi <msadoghi at ucdavis dot edu>

Junchao Chen <juccchen at ucdavis dot edu>

Dakai Kang <dakang at ucdavis dot edu> 

Suyash Gupta <suyash.gupta at berkeley dot> [Now at UC Berkeley]

Sajjad Rahnama <srahnama at ucdavis dot edu> [Now at Oracle]

Jelle Hellings <jhellings at mcmaster dot ca> [Now at McMaster University]

Thamir Qadah <tmqadah at uqu dot edu dot sa> [Umm Al-Qura University]

Shesha Vishnu Prasad <sdharanaiah at ucdavis dot edu> [Now at Path]

Jinxiao Yu <jnxyu at ucdavis dot edu> [Now at Amazon  AWS]

Arindaam Roy <aroy at ucdavis dot edu> [Now at Square]

Divjeet Singh Jas <djas at ucdavis dot edu>

Wayne Wang <wjawang at ucdavis dot edu> [Now at Hesai Technology]

Julieta Duarte <juduarte at ucdavis dot edu>

Glenn Chen <gjjchen at ucdavis dot edu>

Apratim Shukla <aprshukla at ucdavis dot edu>

Priyal Soni <pdsoni at ucdavis dot edu> [Now at Amazon  AWS]

Rohan Sogani <rmsogani at ucdavis dot edu> [Now at Oracle]

Dhruv Krishnan <dkrishnan at ucdavis dot edu > [Now at Amazon  AWS]

Priya Holani <pmholani at ucdavis dot edu > [Now at Amazon  AWS]

Shubham Pandey <shupandey at ucdavis dot edu> [Now at Cisco]

Haojun (Howard) Zhu <hajzhu at ucdavis dot edu> 

Robert HE <mjhe at ucdavis dot edu>  [Now at Amazon  AWS]

Shreenath Iyer  <shriyer at ucdavis dot edu>  [Now at Amazon  AWS]

Domenic Cianfichi  <djcianfichi at ucdavis dot edu>  [Now at Amazon  AWS]

Erik Linsenmayer <ehlinsenmayer at ucdavis dot edu>  [Now at General Atomics]

Shreyan Mohanty <shmohanty at ucdavis dot edu>  [Now at General Atomics]

Xinyuan Sun <sxysun at ucdavis dot edu>  [Now at CertiK]

Patrick Liao <pjliao at ucdavis dot edu>  [Now at Juniper Networks]

Kaustubh Shete <kshete at ucdavis dot edu>

Gopal Nambiar <gnambiar at ucdavis dot edu>

Tim Huang <timhwang at ucdavis dot edu>

Haskell Lark Macaraig <hbmacaraig at ucdavis dot edu>

Steve Chen <sstchen at ucdavis dot edu>

Jared Givens <jcgivens at ucdavis dot edu>

Saipranav Kotamreddy <saikotamreddy at ucdavis dot edu>

Aditya Bej <akbej at ucdavis dot edu>

Seongwoo Choi <shjchoi at ucdavis dot edu >

Sponsors

Champion

Atri Sharma <atri at apache com org>

Nominated Mentors

Junping Du <junping_du at apache com org> 

Calvin Kirs <kirs at apache com org> 

Kevin Ratnasekera <djkevincr at apache com org>

Roman Shaposhnik <rvs at apache com org> 

Christian Grobmeier <grobmeier at apache dot org>

Sponsoring Entity

The Apache Incubator


  • No labels