Abstract

Kvrocks is a distributed key-value NoSQL database, supporting the rich data structure and be widely used by many internet business for example, the recommendation service can use hashes to store user characteristics and the sorted set to sort them. For supporting more users, Kvrocks also have good integration and ecosystem compatibility with other NoSQL database API and script.

In comparison to current open source NoSQL database like Redis, which also has rich data structure, Kvrocks data be stored in disk storage, but the redis data be stored in memory, it means the cost is higher and the capacity is limited when compare to disk storage solution, this is why we created Kvrocks project to solve the cost and capacity issues by using disk storage solution, the following are key features of Kvrocks:

  • Compatibly support Redis protocol and data structures (String/List/Hash/Set/Sorted-set/Bitmap/GEO/Sorted-Int)
  • Disk-Based, all data and updates always persist on the disk
  • Proxyless centralized cluster
  • Stored procedures based on Lua script
  • Transaction(no strict ACID)
  • Publish-Subscribe
  • Tunable consistency(In-Progress)

Proposal

Kvrocks was designed to decrease the cost of memory and increase the capacity while compared to Redis. Also, Kvrocks aims to become a distributed multi-model NoSQL database in the future. We will provide more features to expand its usage scenarios but all data structures on disk. Kvrocks is fully compatible with Redis in term of protocol.

We believe that the Kvrocks project will provide the diversity value for community if Kvrocks is introduced into the Apache incubator.

Background

Kvrocks NoSQL database was originally developed by Meitu, Kvrocks provides rich data structures like string, hash, list, set, sorted set, bitmap, geospatial index on disk, intends to decrease the cost of memory and increase the capacity.

Rationale

Generally, most of the distributed key-value databases on disk were designed with only the simple API like Get/Set/Delete, some key-value databases are also compatible with Redis protocol, but they have many differences in behavior or lack many features. Kvrocks is designed to offer rich data structures on disk, as well as compatible with most of Redis behavior and protocol.

Initial Goals

Our initial goals are to bring Kvrocks into the ASF, transition internal engineering processes into the open, and foster a collaborative development model according to the "Apache Way".

Some areas we want to focus on during the Apache incubation stage include:

  • High performance, the current architecture has investigated and applied many performance improvements on RocksDB, but we still need to improve query performance with a more effective memory cache policy.
  • Better usability: cluster management, control system, toolchains, web UI, etc...
  • Multi-model database: native support more data models, like JSON, Time series
  • Cloud friendly: disaggregate compute and storage, easier to run Kvrocks on Kubernetes

Current Status

Meritocracy

We intend to radically expand the initial developer and user community by running the project in accordance with the "Apache Way". Users and new contributors will be treated with respect and welcomed. By participating in the community and providing quality patches/support that move the project forward, they will earn merit. They also will be encouraged to provide non-code contributions (documentation, events, community management, etc.) and will gain merit for doing so. Those with a proven support and quality track record will be encouraged to become committers. 

Users

Many companies are deploying and using Kvrocks in the production environment, such as Meitu(3+ years), Trip.com(2+ years), Baidu(1.5+ years), Xueqiu, BaishanCloud. In other countries, there also are numerous users, U-Next in Japan, Coinindex Ltd in the United Kingdom, etc. We also received some questions or bug reports from developers in Europe and America. So we can think that Kvrocks is widely used around the world.

Community

If Kvrocks accepted for incubation, the primary initial goal is to build a large community. We really trust that Kvrocks will become a good NoSQL database, we hope to grow the base of contributors by inviting all those who offer contributions through The Apache Way.

Core Developers

The core developers are all experienced, open-source developers. They have operated the Kvrocks Community for 2 years.

Known Risks

Project Name

We have checked and believe the name is suitable and the project has legal permission to continue using its current name. There are no other projects found using this name through Google search.

Orphaned products

Inexperience with Open Source:

Kvrocks has been open-sourced on GitHub for 3 years, during which committers submitted code and documents in full compliance with open-source specifications and requirements.

Length of Incubation:

Expect to enter incubation in 2 months and graduate in about 3 years.

Homogenous Developers:

The developers on the current list come from several different companies plus many independent volunteers. They are experienced with other open-source projects, e.g. Redis, and RocksDB.

Reliance on Salaried Developers:

Most of the developers are paid by their employers to contribute to this project. Given some volunteer developers and committers' sense of ownership for the code, the project could continue even if no salaried developers contributed to the project.

Relationships with Other Apache Products:

Apache has several key-value services, such as Pegasus and Cassandra, Kvrocks has the rich data structure, the followings are concrete differences from other Apache software:

  • For the design goal, Kvrocks was intended to decrease the cost of memory and increase the capacity compared to Redis. It implemented Redis compatible protocol and rich data structures like String, Hash, Sorted Set, List, Bitmap, GEO, and so on, except for data were stored on disk instead of memory. 
  • For replication and cluster mode, Kvrocks use eventual consistency instead of strong consistency and will support tunable consistency soon. Also, Kvrocks implements the pre-sharding cluster mode and is compatible with the Redis cluster client.
  • For the future roadmap, Kvrocks aims to become a multi-model database that would support data structures like JSON/TimeSeries.

An Excessive Fascination with the Apache Brand:

Although we expect that the Apache brand may help attract more contributors, our interest in starting this project is based on the factors mentioned in the fundamentals section. We are interested in joining ASF to increase our connections in the open-source world. Based on extensive collaboration, it is possible to build a community of developers and committers that live longer than the founder.

Documentation

Kvrocks documentation is provided on https://github.com/KvrocksLabs/kvrocks/wiki

Initial Source

This project source code: https://github.com/KvrocksLabs/kvrocks

Source and Intellectual Property Submission Plan

Most of source code has been refactored to Apache License Version 2.0, and have been verified there are no intellectual property or license issues when being released to open source by Meitu in 2019. Meitu will provide SGA and all committers will sign ICLA after Kvrocks is accepted into the Incubator.

External Dependencies:

Kvrocks has the following external dependencies:

  • BSD
    • libevent
    • jemalloc
    • snappy
    • glog
  • MIT
    • lua
  • Apache
    • rocksdb

Cryptography

None

Required Resources

Mailing lists:

Git Repository:

Issue Tracking:

The community would like to continue using GitHub Issues (will be moved to github.com/apache).

Other Resources:

  • The community has already chosen GitHub actions as continuous integration tools.
  • The community has already chosen GitHub Wiki as the wiki page.

Initial Committers

Affiliations 

The initial committers and contributors are from several organizations and two individual developers.  

  • AfterShip
    • git-hulk
  • Baidu
    • ShooterIT
    • Alfejik
    • ChrisZ
    • Xiongda
    • Myth
  • Ctrip
    • Brad Lee
    • Colin Chamber
    • Popunit
  • Huawei
    • Liang Chen
  • Meitu
    • JianBing Lai
  • Meituan
    • Xiaoqiao He
  • Xiaomi
    • Nioshield
  • Individual Developer
    • Karelrooted
    • WyattJia

Sponsors

Champion:

Liang Chen  [chenliang613@apache.org]

Nominated Mentors:

Jean-Baptiste Onofre  [jbonofre@apache.org]

Xiaoqiao He [hexiaoqiao@apache.org]

tison [tison@apache.org]

Von Gosling [vongosling@apache.org]

Sponsoring Entity:

We are expecting the Apache Incubator could sponsor this project.

  • No labels

2 Comments

  1. Matt Sicker

    Can you update the initial committer list with public names for the individuals? Matching online handles to public names is difficult otherwise for ASF admins when making accounts.

    1. Zili Chen

      Updated.