Abstract
OpenDAL means “Open Data Access Layer”. It’s a Rust library that helps developers access data freely, painlessly, and efficiently over multiple services, including AWS S3, HDFS, POSIX-compatible file systems, and so on.
Proposal
OpenDAL provides the following features to support developers accessing data freely, painlessly, and efficiently:
- Freely
- Access different storage services in the same way
- Behavior tests for all services
- Cross-language/project bindings (working in progress)
- Painlessly
- Efficiently
- Zero cost: mapping to underlying API calls directly
- Best effort: auto-pick the best
read/seek/nextimplementations based on services - Auto metadata reuse: avoid extra
metadatacalls
OpenDAL was originally designed to be used by the Databend project but is now being used by Mozilla's sccache, DeepETH's mars, and several database startups.
We believe that the OpenDAL project will provide diversity value to the community if OpenDAL is brought into the Apache incubator.
Background
OpenDAL is being developed by an open-source community from day one and is owned by DatafuseLabs. The project has been launched in February 2022.
Rationale
OpenDAL provides a unified storage abstraction layer that simplifies the interfacing of different storage services. In addition, OpenDAL provides further advanced storage encapsulation, enabling enhancements such as automatic retry, request optimization, and observability. OpenDAL makes it possible to develop once and run on any storage service.
Initial Goals
By transferring ownership of the project to the ASF, OpenDAL expects to ensure its neutrality and further encourage and facilitate the adoption of OpenDAL by the community.
Some of the areas we would like to focus on during the Apache incubation phase include:
- A healthier community: more maintainers and contributors will be able to participate in OpenDAL and own different modules.
- Wider adoption: OpenDAL can be adopted by more open source/commercial projects, which in turn drives its own functionality.
- Richer integration: OpenDAL enables greater integration of storage services and offers a wider range of language bindings.
Current Status
Meritocracy
We intend to radically expand the initial developer and user community by running the project the 'Apache way'. Users and new contributors will be respected and welcomed. They will earn credit by participating in the community and providing quality patches/support to move the project forward. They will also be encouraged to provide non-code contributions (documentation, events, community management, etc.) and will be rewarded accordingly. Those with a proven track record of support and quality will be encouraged to become committers.
Community
Contributors: 37
Users:
- Databend: A cloud data warehouse
- GreptimeDB: A time-series database
- Sccache: ccache with cloud storage
- RisingWave: A Distributed SQL Database for Stream Processing
Core Developers
The core developers are all experienced open-source developers. They have been running the OpenDAL community for 1 year.
Alignment
Known Risks
Project Name
We have checked and believe that the name is appropriate and that the project has legal permission to continue using its current name. There are no other projects with this name found in a Google search.
Orphan Products
Inexperience with Open Source
OpenDAL's core developers are all experienced open source contributors, and its main maintainer Xuanwo has 10 years of open source experience, having worked on a number of open source projects including Hexo, TiDB, TiKV, Databend, Sccache, and others.
Length of Incubation
Expect to enter incubation in two months and graduate in about two years.
Homogenous Developers
OpenDAL developers come from a variety of backgrounds and contribute to the OpenDAL project for different usage scenarios.
Reliance on Salaried Developers
Most developers are paid by their employers to contribute to this project. It's a big risk indeed, we expect to attract more maintainers and contributors from outside the DatafuseLabs to address this.
Relationships with Other Apache Products
- OpenDAL can be used to operate files such as parquet, avro, …
- OpenDAL provides a shim that can be used with arrow object_store
An Excessive Fascination with the Apache Brand
Documentation
The document of OpenDAL is hosted at https://opendal.databend.rs/. And opendal’s document is self-contained, all its current and historical versions could be found at https://docs.rs/opendal/latest/opendal/.
Initial Source
The project currently holds a GitHub repository and a Cargo crate:
The crate will retain its name, while the repository will be moved to apache org, and the website will be permanently moved to opendal.apache.org if the proposal gets accepted.
Source and Intellectual Property Submission Plan
External Dependencies
Generated by
cargo deny list
- Apache-2.0 (197):
- ahash@0.7.6
- anes@0.1.6
- anyhow@1.0.68
- async-channel@1.8.0
- async-compat@0.2.1
- async-trait@0.1.60
- autocfg@1.1.0
- backon@0.2.0
- base64@0.13.1
- base64@0.21.0
- bitflags@1.3.2
- block-buffer@0.10.3
- bumpalo@3.11.1
- cast@0.3.0
- cc@1.0.78
- cfg-if@1.0.0
- ciborium@0.2.0
- ciborium-io@0.2.0
- ciborium-ll@0.2.0
- clap@3.2.23
- clap_lex@0.2.4
- concurrent-queue@2.0.0
- core-foundation@0.9.3
- core-foundation-sys@0.8.3
- cpufeatures@0.2.5
- criterion@0.4.0
- criterion-plot@0.5.0
- crossbeam-channel@0.5.6
- crossbeam-deque@0.8.2
- crossbeam-epoch@0.9.13
- crossbeam-utils@0.8.14
- crypto-common@0.1.6
- ctor@0.1.26
- deadpool@0.9.5
- deadpool-runtime@0.1.2
- diff@0.1.13
- digest@0.10.6
- dirs@4.0.0
- dirs-sys@0.3.7
- either@1.8.0
- encoding_rs@0.8.31
- env_logger@0.10.0
- event-listener@2.5.3
- fastrand@1.8.0
- flagset@0.4.3
- fnv@1.0.7
- form_urlencoded@1.1.0
- futures@0.3.25
- futures-channel@0.3.25
- futures-core@0.3.25
- futures-executor@0.3.25
- futures-io@0.3.25
- futures-lite@1.12.0
- futures-macro@0.3.25
- futures-sink@0.3.25
- futures-task@0.3.25
- futures-timer@3.0.2
- futures-util@0.3.25
- getrandom@0.1.16
- getrandom@0.2.8
- half@1.8.2
- hashbrown@0.12.3
- hermit-abi@0.1.19
- hermit-abi@0.2.6
- hex@0.4.3
- hmac@0.12.1
- http@0.2.8
- http-types@2.12.0
- httparse@1.8.0
- httpdate@1.0.2
- humantime@2.1.0
- hyper-rustls@0.23.2
- idna@0.3.0
- indexmap@1.9.2
- io-lifetimes@1.0.3
- ipnet@2.7.0
- itertools@0.10.5
- itoa@1.0.5
- js-sys@0.3.60
- lazy_static@1.4.0
- libc@0.2.139
- linux-raw-sys@0.1.4
- lock_api@0.4.9
- log@0.4.17
- md-5@0.10.5
- mime@0.3.16
- num-bigint@0.4.3
- num-integer@0.1.45
- num-traits@0.2.15
- num_cpus@1.15.0
- once_cell@1.17.0
- opendal@0.25.1
- openssl-probe@0.1.5
- opentelemetry@0.17.0
- opentelemetry-jaeger@0.16.0
- opentelemetry-semantic-conventions@0.9.0
- os_str_bytes@6.4.1
- parking@2.0.0
- parking_lot@0.12.1
- parking_lot_core@0.9.5
- paste@1.0.11
- percent-encoding@2.2.0
- pin-project@1.0.12
- pin-project-internal@1.0.12
- pin-project-lite@0.2.9
- pin-utils@0.1.0
- ppv-lite86@0.2.17
- pretty_assertions@1.3.0
- proc-macro2@1.0.49
- quote@1.0.23
- rand@0.7.3
- rand@0.8.5
- rand_chacha@0.2.2
- rand_chacha@0.3.1
- rand_core@0.5.1
- rand_core@0.6.4
- rand_hc@0.2.0
- rayon@1.6.1
- rayon-core@1.10.1
- regex@1.7.0
- regex-syntax@0.6.28
- reqsign@0.8.1
- reqwest@0.11.13
- rustix@0.36.6
- rustls@0.20.7
- rustls-native-certs@0.6.2
- rustls-pemfile@1.0.1
- ryu@1.0.12
- scopeguard@1.1.0
- sct@0.7.0
- security-framework@2.7.0
- security-framework-sys@2.6.1
- serde@1.0.152
- serde_derive@1.0.152
- serde_json@1.0.91
- serde_qs@0.8.5
- serde_urlencoded@0.7.1
- sha1@0.10.5
- sha2@0.10.6
- smallvec@1.10.0
- socket2@0.4.7
- syn@1.0.107
- thiserror@1.0.38
- thiserror-impl@1.0.38
- thread_local@1.1.4
- threadpool@1.8.1
- thrift@0.15.0
- time@0.3.17
- time-core@0.1.0
- time-macros@0.2.6
- tinytemplate@1.2.1
- tinyvec@1.6.0
- tinyvec_macros@0.1.0
- tokio-rustls@0.23.4
- typenum@1.16.0
- unicase@2.6.0
- unicode-bidi@0.3.8
- unicode-ident@1.0.6
- unicode-normalization@0.1.22
- ureq@2.6.1
- url@2.3.1
- uuid@1.2.2
- version_check@0.9.4
- waker-fn@1.1.0
- wasi@0.11.0+wasi-snapshot-preview1
- wasi@0.9.0+wasi-snapshot-preview1
- wasm-bindgen@0.2.83
- wasm-bindgen-backend@0.2.83
- wasm-bindgen-futures@0.4.33
- wasm-bindgen-macro@0.2.83
- wasm-bindgen-macro-support@0.2.83
- wasm-bindgen-shared@0.2.83
- web-sys@0.3.60
- winapi@0.3.9
- winapi-i686-pc-windows-gnu@0.4.0
- winapi-x86_64-pc-windows-gnu@0.4.0
- windows-sys@0.36.1
- windows-sys@0.42.0
- windows_aarch64_gnullvm@0.42.0
- windows_aarch64_msvc@0.36.1
- windows_aarch64_msvc@0.42.0
- windows_i686_gnu@0.36.1
- windows_i686_gnu@0.42.0
- windows_i686_msvc@0.36.1
- windows_i686_msvc@0.42.0
- windows_x86_64_gnu@0.36.1
- windows_x86_64_gnu@0.42.0
- windows_x86_64_gnullvm@0.42.0
- windows_x86_64_msvc@0.36.1
- windows_x86_64_msvc@0.42.0
- wiremock@0.5.16
- yansi@0.5.1
- io-lifetimes@1.0.3
- linux-raw-sys@0.1.4
- rustix@0.36.6
- wasi@0.11.0+wasi-snapshot-preview1
- wasi@0.9.0+wasi-snapshot-preview1
- BSD-3-Clause (3)
- BSL-1.0 (1):
- ISC (8):
- MIT (242)
- ahash@0.7.6
- aho-corasick@0.7.20
- anes@0.1.6
- anyhow@1.0.68
- assert-json-diff@2.0.2
- async-channel@1.8.0
- async-compat@0.2.1
- async-trait@0.1.60
- atty@0.2.14
- autocfg@1.1.0
- base64@0.13.1
- base64@0.21.0
- bincode@2.0.0-rc.2
- bincode_derive@2.0.0-rc.2
- bitflags@1.3.2
- block-buffer@0.10.3
- bumpalo@3.11.1
- byteorder@1.4.3
- bytes@1.3.0
- cast@0.3.0
- cc@1.0.78
- cfg-if@1.0.0
- clap@3.2.23
- clap_lex@0.2.4
- concurrent-queue@2.0.0
- core-foundation@0.9.3
- core-foundation-sys@0.8.3
- cpufeatures@0.2.5
- criterion@0.4.0
- criterion-plot@0.5.0
- crossbeam-channel@0.5.6
- crossbeam-deque@0.8.2
- crossbeam-epoch@0.9.13
- crossbeam-utils@0.8.14
- crypto-common@0.1.6
- ctor@0.1.26
- deadpool@0.9.5
- deadpool-runtime@0.1.2
- diff@0.1.13
- digest@0.10.6
- dirs@4.0.0
- dirs-sys@0.3.7
- dlv-list@0.3.0
- dotenvy@0.15.6
- either@1.8.0
- encoding_rs@0.8.31
- env_logger@0.10.0
- event-listener@2.5.3
- fastrand@1.8.0
- fnv@1.0.7
- form_urlencoded@1.1.0
- futures@0.3.25
- futures-channel@0.3.25
- futures-core@0.3.25
- futures-executor@0.3.25
- futures-io@0.3.25
- futures-lite@1.12.0
- futures-macro@0.3.25
- futures-sink@0.3.25
- futures-task@0.3.25
- futures-timer@3.0.2
- futures-util@0.3.25
- generic-array@0.14.6
- getrandom@0.1.16
- getrandom@0.2.8
- h2@0.3.15
- half@1.8.2
- hashbrown@0.12.3
- hermit-abi@0.1.19
- hermit-abi@0.2.6
- hex@0.4.3
- hmac@0.12.1
- http@0.2.8
- http-body@0.4.5
- http-types@2.12.0
- httparse@1.8.0
- httpdate@1.0.2
- humantime@2.1.0
- hyper@0.14.23
- hyper-rustls@0.23.2
- idna@0.3.0
- indexmap@1.9.2
- infer@0.2.3
- integer-encoding@3.0.4
- io-lifetimes@1.0.3
- ipnet@2.7.0
- is-terminal@0.4.2
- itertools@0.10.5
- itoa@1.0.5
- js-sys@0.3.60
- jsonwebtoken@8.2.0
- lazy_static@1.4.0
- libc@0.2.139
- linux-raw-sys@0.1.4
- lock_api@0.4.9
- log@0.4.17
- matchers@0.1.0
- md-5@0.10.5
- memchr@2.5.0
- memoffset@0.7.1
- mime@0.3.16
- mime_guess@2.0.4
- mio@0.8.5
- nu-ansi-term@0.46.0
- num-bigint@0.4.3
- num-integer@0.1.45
- num-traits@0.2.15
- num_cpus@1.15.0
- once_cell@1.17.0
- oorandom@11.1.3
- openssl-probe@0.1.5
- ordered-float@1.1.1
- ordered-multimap@0.4.3
- os_str_bytes@6.4.1
- output_vt100@0.1.3
- overload@0.1.1
- parking@2.0.0
- parking_lot@0.12.1
- parking_lot_core@0.9.5
- paste@1.0.11
- pem@1.1.0
- percent-encoding@2.2.0
- pin-project@1.0.12
- pin-project-internal@1.0.12
- pin-project-lite@0.2.9
- pin-utils@0.1.0
- plotters@0.3.4
- plotters-backend@0.3.4
- plotters-svg@0.3.3
- ppv-lite86@0.2.17
- pretty_assertions@1.3.0
- proc-macro2@1.0.49
- quick-xml@0.27.1
- quote@1.0.23
- rand@0.7.3
- rand@0.8.5
- rand_chacha@0.2.2
- rand_chacha@0.3.1
- rand_core@0.5.1
- rand_core@0.6.4
- rand_hc@0.2.0
- rayon@1.6.1
- rayon-core@1.10.1
- redox_syscall@0.2.16
- redox_users@0.4.3
- regex@1.7.0
- regex-automata@0.1.10
- regex-syntax@0.6.28
- reqwest@0.11.13
- retain_mut@0.1.9
- rust-ini@0.18.0
- rustix@0.36.6
- rustls@0.20.7
- rustls-native-certs@0.6.2
- rustls-pemfile@1.0.1
- same-file@1.0.6
- schannel@0.1.20
- scopeguard@1.1.0
- sct@0.7.0
- security-framework@2.7.0
- security-framework-sys@2.6.1
- serde@1.0.152
- serde_derive@1.0.152
- serde_json@1.0.91
- serde_qs@0.8.5
- serde_urlencoded@0.7.1
- sha1@0.10.5
- sha2@0.10.6
- sharded-slab@0.1.4
- size@0.4.1
- slab@0.4.7
- smallvec@1.10.0
- socket2@0.4.7
- spin@0.5.2
- syn@1.0.107
- termcolor@1.1.3
- textwrap@0.16.0
- thiserror@1.0.38
- thiserror-impl@1.0.38
- thread_local@1.1.4
- threadpool@1.8.1
- time@0.3.17
- time-core@0.1.0
- time-macros@0.2.6
- tinytemplate@1.2.1
- tinyvec@1.6.0
- tinyvec_macros@0.1.0
- tokio@1.23.1
- tokio-macros@1.8.2
- tokio-rustls@0.23.4
- tokio-util@0.7.4
- tower-service@0.3.2
- tracing@0.1.37
- tracing-core@0.1.30
- tracing-log@0.1.3
- tracing-opentelemetry@0.17.4
- tracing-subscriber@0.3.16
- try-lock@0.2.3
- typenum@1.16.0
- unicase@2.6.0
- unicode-bidi@0.3.8
- unicode-ident@1.0.6
- unicode-normalization@0.1.22
- ureq@2.6.1
- url@2.3.1
- uuid@1.2.2
- valuable@0.1.0
- version_check@0.9.4
- virtue@0.0.8
- waker-fn@1.1.0
- walkdir@2.3.2
- want@0.3.0
- wasi@0.11.0+wasi-snapshot-preview1
- wasi@0.9.0+wasi-snapshot-preview1
- wasm-bindgen@0.2.83
- wasm-bindgen-backend@0.2.83
- wasm-bindgen-futures@0.4.33
- wasm-bindgen-macro@0.2.83
- wasm-bindgen-macro-support@0.2.83
- wasm-bindgen-shared@0.2.83
- web-sys@0.3.60
- winapi@0.3.9
- winapi-i686-pc-windows-gnu@0.4.0
- winapi-util@0.1.5
- winapi-x86_64-pc-windows-gnu@0.4.0
- windows-sys@0.36.1
- windows-sys@0.42.0
- windows_aarch64_gnullvm@0.42.0
- windows_aarch64_msvc@0.36.1
- windows_aarch64_msvc@0.42.0
- windows_i686_gnu@0.36.1
- windows_i686_gnu@0.42.0
- windows_i686_msvc@0.36.1
- windows_i686_msvc@0.42.0
- windows_x86_64_gnu@0.36.1
- windows_x86_64_gnu@0.42.0
- windows_x86_64_gnullvm@0.42.0
- windows_x86_64_msvc@0.36.1
- windows_x86_64_msvc@0.42.0
- winreg@0.10.1
- wiremock@0.5.16
- yansi@0.5.1
- MPL-2.0 (1):
- Unicode-DFS-2016 (1):
- Unlicense (8):
- Zlib (2):
- Unlicensed (1):
-
ring is licensed under a mix of MIT, ISC, and OpenSSL’s licenses. OpenSSL itself is licensed under Apache 2.0. So ring is Apache 2.0 compatible.
-
Cryptography
N/A
Required Resources
Mailing Lists
- private@opendal.apache.org
- dev@opendal.apache.org
- commits@opendal.apache.org
- issues@opendal.apache.org
- build@opendal.apache.org
Subversion Directory
N/A
Git Repositories
From https://github.com/datafuselabs/opendal
Issue Tracking
The community would like to continue using GitHub Issues.
Other Resources
The community has already chosen GitHub actions as continuous integration tools.
Initial Committers
- Hao Ding [github@xuanwo.io]
- Lue Cai [cailue@bupt.edu.cn]
- Zhuoran Shang [psiace@outlook.com]
- Benwang Li [libenw1992@gmail.com]
Sponsors
Champion
- tison [tison@apache.org]
Nominated Mentors
- hexiaoqiao [hexiaoqiao@apache.org]
- ningjiang [ningjiang@apache.org]
- tedliu [tedliu@apache.org]
- tison [tison@apache.org]
- wusheng [wusheng@apache.org]
Sponsor Entity
The Incubator