GeodeProposal

Abstract

Geode is a data management platform that provides real-time, consistent access to data-intensive applications throughout widely distributed cloud architectures.

Geode pools memory (along with CPU, network and optionally local disk) across multiple processes to manage application objects and behavior. It uses dynamic replication and data partitioning techniques for high availability, improved performance, scalability, and fault tolerance. Geode is both a distributed data container and an in-memory data management system providing reliable asynchronous event notifications and guaranteed message delivery.

Proposal

The goal of this proposal is to bring the core of Pivotal Software, Inc.’s (Pivotal) Pivotal GemFireⓇ codebase into the Apache Software Foundation (ASF) in order to build a vibrant, diverse and self-governed open source community around the technology. Pivotal will continue to market and sell Pivotal GemFire based on Geode. Geode and Pivotal GemFire will be managed separately. This proposal covers the Geode source code (mainly written in Java), Geode documentation and other materials currently available on GitHub.

While Geode is our primary choice for a name of the project, in order to facilitate PODLINGNAMESEARCH we have come up with two alternatives:

Haptic
FIG

Background

GemFire is an extremely mature and robust product that can trace its legacy all the way back to one of the first Object Databases for Smalltalk: GemStone. The GemFire code base has been maintained by the same group of engineers as a closed source project. Because of that, even though the engineers behind GemFire are the de-facto knowledge leaders for distributed in-memory management, they have had little exposure to the open source governance process. The original company developing GemStone and GemFire was acquired by VMWare in 2010 and later spun off as part of Pivotal Software in 2013. Today GemFire is used by over 600 enterprise customers. An example deployment includes China National Railways that uses Pivotal GemFire to run railway ticketing for the entire country of China with a 10 node cluster that manages 2 gigabytes "hot data" in memory, and 10 backup nodes for high availability and elastic scale.

Rationale

Modern-day data management architectures require a robust in-memory data grid solution to handle a variety of use cases, ranging from enterprise-wide caching to real-time transactional applications at scale. In addition, as memory size and network bandwidth growth continues to outpace those of disk, the importance of managing large pools of RAM at scale increases. It is essential to innovate at the same pace and Pivotal strongly believes that in the Big Data space, this can be optimally achieved through a vibrant, diverse, self-governed community collectively innovating around a single codebase while at the same time cross-pollinating with various other data management communities. ASF is the ideal place to meet these ambitious goals.

Initial Goals

Our initial goals are to bring Geode into the ASF, transition internal engineering processes into the open, and foster a collaborative development model according to the "Apache Way." Pivotal plans to develop new functionality in an open, community-driven way. To get there, the existing internal build, test and release processes will be refactored to support open development.

Current Status

Currently, the project code base is licensed for evaluation purposes and is available for download from Pivotal.io (https://network.pivotal.io/products/project-geode). The documentation and wiki pages are available as public GitHub repositories under Project Geode organization on GitHub (https://github.com/project-geode). Although Pivotal GemFire was developed as a proprietary, closed-source product, the internal engineering practices adopted by the development team lend themselves well to an open, collaborative and meritocratic environment.

The Pivotal GemFire team has always focused on building a robust end user community of paying and non-paying customers. The existing documentation along with StackOverflow and other similar forums are expected to facilitate conversions between our existing users so as to transform them into an active community of Geode members, stakeholders and developers.

Meritocracy

Our proposed list of initial committers include the current GemFire R&D team, Pivotal Field Engineers, and several existing customers and partners. This group will form a base for the broader community we will invite to collaborate on the codebase. We intend to radically expand the initial developer and user community by running the project in accordance with the "Apache Way". Users and new contributors will be treated with respect and welcomed. By participating in the community and providing quality patches/support that move the project forward, they will earn merit. They also will be encouraged to provide non-code contributions (documentation, events, community management, etc.) and will gain merit for doing so. Those with a proven support and quality track record will be encouraged to become committers.

Community

If Geode is accepted for incubation, the primary initial goal will be transitioning the core community towards embracing the Apache Way of project governance. We would solicit major existing contributors to become committers on the project from the start.

Core Developers

While a few core developers are skilled in working in openly governed Apache communities. Most of the core developers are currently NOT affiliated with the ASF and would require new ICLAs before committing to the project.

Alignment

The following existing ASF projects can be considered when reviewing Geode proposal:

Apache HadoopⓇ is a distributed storage and processing framework for very large datasets focusing primarily on batch processing for analytic purposes. Geode is a data management platform that provides real-time, consistent, and transactional access to data-intensive applications. Our roadmap includes plans to provide close integration with HDFS.

Apache HBase offers tabular data stored in Hadoop based on the Google Bigtable model. HBase uses a key-based partitioning scheme and column family data model that can work well for scan intensive workloads but is not as broadly applicable as the rich object model, OQL querying, and hash partitioning provided by Geode. Geode will use the HFile format for storing data in HDFS.

Apache Spark is a fast engine for processing large datasets, typically from a Hadoop cluster, and performing batch, streaming, interactive, or machine learning workloads. Geode supports high throughput streaming ingest application patterns and data parallel algorithms. Geode also ensures that data is highly available through redundancy while Spark recovers from faults using RDD lineage recomputation. Our roadmap includes plans for providing integration with the Spark platform.

Apache Ignite (incubating) offers distributed in-memory processing capabilities which in some ways overlap with Geode. However, Geode has been in this field for more than 10 years and the product API's, design, and implementation details are quite different. In addition, Geode offers highly optimized shared-nothing disk persistence for in-memory data which does not appear to be available with Ignite.

Apache Cassandra is a highly scalable, distributed key-value store that focuses on eventual consistency. It uses log-structured merge trees to handle write-heavy workloads. The Geode distributed in-memory cluster provides highly performant and advanced capabilities including network partition detection and recovery, version-based state synchronization and conflict resolution, and single IO disk operations.

Apache ActiveMQ and its sub project Apache Apollo offers a powerful message queue framework that is being considered for an open source implementation of Geode's WAN replication capabilities.

Apache Storm is a streaming engine that processes events through a directed graph of computation. It requires Apache Zookeeper for coordination and Apache Kafka to reliably store the streaming data source. Geode provides builtin capabilities for these functions. In addition, Geode offers data locality for related entities and in-process access to reference data typically used during processing.

Apache Kafka offers distributed and durable publish-subscribe messaging. Geode expands beyond publish-subscribe messaging to support data oriented notifications delivered from highly available, fault tolerant event queues that can be easily correlated to related data for further processing. Apache Samza is a distributed processing framework heavily dependent on Apache Kafka and Apache Hadoop YARN for messaging and distributed fault-tolerant processing. Geode is data source agnostic and leverages its own technology and implementation for such use cases.

Known Risks

Development has been sponsored mostly by a single company (or its predecessors) thus far and coordinated mainly by the core Pivotal GemFire team.

For the project to fully transition to the Apache Way governance model, development must shift towards the meritocracy-centric model of growing a community of contributors balanced with the needs for extreme stability and core implementation coherency.

The tools and development practices in place for the Pivotal GemFire product are compatible with the ASF infrastructure and thus we do not anticipate any on-boarding pains. Migration from the current GitHub repository is also expected to be straightforward.

The project currently includes a modified version of the JGroups software toolkit. JGroups was initially released under the LGPL license. Although we have complied with the terms of the LGPL by making the modified source available, LGPL is classified as an incompatible license by the ASF. The JGroups project has since been re-licenced under ALv2 (see http://www.jgroups.org/license.html) and we plan to engage with the licensor to overcome this license incompatibility. If the license incompatibility cannot be overcome through our discussions with the licensor, we will need to rework this portion of the project based upon a newer version of the JGroups toolkit or via other alternative development efforts.

Orphaned products

Pivotal is fully committed to Pivotal GemFire and the product will continue to be based on the Geode project. Moreover, Pivotal has a vested interest in making Geode succeed by driving its close integration with sister ASF projects. We expect this to further reduces the risk of orphaning the product.

Inexperience with Open Source

Pivotal has embraced open source software since its formation by employing contributors/committers and by shepherding open source projects like Cloud Foundry, Spring, RabbitMQ and MADlib. Pivotal also supports other open source projects such as Redis, Chorus, Groovy and Grails. We have experience with the formation of vibrant communities around open technologies with the Cloud Foundry Foundation. Although some of the initial committers have not been developers on an entirely open source, community-driven project, we expect to bring to bear the open development practices that have proven successful on longstanding Pivotal open source projects to the Geode project. Additionally, several ASF veterans agreed to mentor the project and are listed in this proposal. The project will rely on their guidance and collective wisdom to quickly transition the entire team of initial committers towards practicing the Apache Way.

Homogeneous Developers

While most of the initial committers are employed by Pivotal, we have already seen a healthy level of interest from our existing customers and partners. We intend to convert that interest directly into participation and will be investing in activities to recruit additional committers from other companies.

Reliance on Salaried Developers

Most of the contributors are paid to work in the Big Data space. While they might wander from their current employers, they are unlikely to venture far from their core expertises and thus will continue to be engaged with the project regardless of their current employers.

Relationships with Other Apache Products

As mentioned in the Alignment section, Geode may consider various degrees of integration and code exchange with Apache Ignite (incubating), Apache Hadoop, Apache Storm, Apache Spark and Apache Kafka/Samza. Given the success that the Pivotal GemFire product enjoyed as an embedded service, we expect integration points to be inside and outside the project. We look forward to collaborating with these communities as well as other communities under the Apache umbrella.

An Excessive Fascination with the Apache Brand

While we intend to leverage the Apache ‘branding’ when talking to other projects as testament of our project’s ‘neutrality’, we have no plans for making use of Apache brand in press releases nor posting billboards advertising acceptance of Geode into Apache Incubator.

Documentation

See documentation for the current state of the project documentation available as part of the GitHub repository at https://github.com/project-geode/docs and live at http://geode-docs.cfapps.io/

Initial Source

Pivotal is releasing the source code for Geode under an Evaluation License at https://network.pivotal.io/products/project-geode . We encourage ASF community members interested in this proposal to download the source code, review and try out the software.

Source and Intellectual Property Submission Plan

As soon as Geode is approved to join Apache Incubator, the source code will be transitioned via the Software Grant Agreement onto ASF infrastructure and in turn made available under the Apache License, version 2.0. We know of no legal encumberments that would inhibit the transfer of source code to the ASF.

External Dependencies

Embedded dependencies (relocated):

json
jgroups
joptsimple

Runtime dependencies:

antlr
classmate
commons-fileupload
commons-io
commons-lang
commons-modeler
fastutil
findbugs annotations
guava
jackson
jansi
javax.activation
javax.mail-api
javax.resource-api
javax.servlet-api
javax.transaction-api
jetty
jline
jna
json4s
log4j
mx4j
paranamer
scala
slf4j
snappy-java
spring
swagger

Module or optional dependencies:

None

Build only dependencies:

None

Test only dependencies:

cglib
hamcrest
jmock
junit
multithreadedtc
objenesis

Cryptography N/A

Required Resources

Mailing lists

private@geode.incubator.apache.org (moderated subscriptions)
commits@geode.incubator.apache.org
dev@geode.incubator.apache.org
issues@geode.incubator.apache.org
user@geode.incubator.apache.org

Git Repository

https://git-wip-us.apache.org/repos/asf/incubator-geode.git

Issue Tracking

JIRA Project Geode (GEODE)

Other Resources

Means of setting up regular builds for Geode on builds.apache.org

Initial Committers

Amey Barve
Adib Saikali
Alan Strait
Amogh Shetkar
Anil Gingade
Anilkumar Gingade
Anthony Baker
Ashvin Agrawal
Asif Shahid
Avinash Dongre
Barry Oglesby
Ben Reser
Bruce Schuchardt
Bruce Szalwinski
Catherine Johnson
Chip Childers
Christian Tzolov
Dan Smith
Darrel Schneider
Dave Muirhead
David Yozie
Dick Cavender
Edin Zulich
Eric Shu
Gideon Low
Greg Chase
Hemant Bhanawat
Henry Saputra
Hitesh Khamesra
Jacob Barrett
Jags Ramnarayan
Jan Iversen
Jason Huynh
Jens Deppe
Jianxia Chen
John Blum
Justin Erenkrantz
Ketan Deshpande
Kirk Lund
Kishor Bachhav
Konstantin Boudnik
Konstantin Ignatyev
Lise Storc
Luke Shannon
Lyndon Adams
Lynn Gallinat
Lynn Hughes-Godfrey
Mark Bretl
Michael Schubert
Namrata Thanvi
Neeraj Kumar
Nilkanth Patel
Qihong Chen
Rahul Diyewar
Randy May
Roman Shaposhnik
Severine Tymon
Shatarupa Nandi
Shirish Deshmukh
Sonal Agarwal
Soubhik Chakraborty
Sourabh Bansod
Stephane Maldini
Stuart Williams
Sudhir Menon
Sunil Jigyasu
Supriya Pillai
Suranjan Kumar
Suyog Bhokare
Swapnil Bawaskar
Swati Sawant
Tushar Khairnar
Udo Kohlmeyer
Vince Ford
Vinesh Prasanna Manoharan
Vivek Bhaskar
Wes Williams
William A. Rowe Jr.
William Markito
Will Schipp
Xiaojian Zhou
Yogesh Mahajan

Affiliations

WANDisco: Konstantin Boudnik
Bloomberg LP: Justin Erenkrantz
Cloud Foundry Foundation: Chip Childers
NASA JPL: Chris Mattmann
Unaffiliated: Jan Iversen
CDK Global: Ben Reser, Konstantin Ignatyev, Bruce Szalwinski
Pivotal: everyone else on this proposal

Page tree