Hivemall is a library for machine learning implemented as Hive UDFs/UDAFs/UDTFs.
Hivemall runs on Hadoop-based data processing frameworks, specifically on Apache Hive, Apache Spark, and Apache Pig, that support Hive UDFs as an extension mechanism.
Hivemall is a collection of machine learning algorithms and versatile data analytics functions. It provides a number of ease of use machine learning functionalities through user-defined function (UDF), user-defined aggregate function (UDAFs), and/or user-defined table generating functions (UDTFs) of Apache Hive. It offers a variety of functionalities: regression, classification, recommendation, anomaly detection, k-nearest neighbor, and feature engineering. Hivemall supports state-of-the-art machine learning algorithms such as Soft Confidence Weighted, Adaptive Regularization of Weight Vectors, Factorization Machines, and AdaDelta. Hivemall is mainly designed to run on Apache Hive but it also supports Apache Pig and Apache Spark for the runtime.
Hivemall started as a research project of the main developer at National Institute of Advanced Industrial Science and Technology (AIST) in 2013 and the initial version was released on 2 Oct, 2013 on Github: https://github.com/myui/hivemall.
After the main developer moving to Treasure Data in 2015, the project has been actively developed as an open source product and changed the license from GNU LGPL v2.1 to Apache License v2 on Mar 16, 2015. The project copyright holders agreed to change the license then.
The community is growing incrementally and the project has 15 contributors, 431 stars, and 131 forks on Github as of Aug 15, 2016. The project was awarded for the InfoWorld Bossie Awards (the best open source big data tools) in 2014.
Past main contributions by external contributors includes Apache Pig supports from Daniel Dai (Hortonworks), Apache Spark porting and an integration to Apache YARN from Takeshi Yamamuro (NTT). Hivemall was originally designed for Apache Hive but it now supports Apache Spark and Apache Pig.
User-defined function is a powerful mechanism to enrich the expressive power of declarative query languages like SQL, HiveQL, PigLatin, Spark SQL. Hive UDF interface is now becoming the de-facto standard for SQL-on-Hadoop platforms; Apache Spark and Apache Pig have full supports for Hive UDFs/UDAFs/UDTFs, and Apache Impala, Apache Drill, and Apache Tajo also have limited supports for Hive UDFs/UDAFs.
Hivemall can be considered as a cross platform library for machine learning as Hivemall is implemented as cross platform Hive UDFs/UDAFs/UDTFs; prediction models built by a batch query of Apache Hive can be used on Apache Spark/Pig, and conversely, prediction models build by Apache Spark can be used from Apache Hive/Pig.
Several database vendors are trying to offer machine learning functionality in relational databases, so that the costs of moving data can be eliminated. Apache MADlib, a machine learning library for HAWQ and PostgreSQL, is accepted as an Apache Incubator project. MADlib is implemented using PostgreSQL UDF interface.
Apache Hive has a JIRA ticket in HIVE-7940 to support machine learning functionalities. So, we consider this proposal is useful for the community. We consider that Hivemall is better to be a separated project to the Apache Hive because 1) we target other data processing frameworks such as Apache Spark as well for the runtime of Hivemall, and 2) the current codebase is large enough to be separated. Separation of concerns is good for project governance (e.g., release management). For example, Apache Datafu is data mining and statistics library for Apache Pig and a separated project to Apache Pig.
We consider that Hivemall would be a similar position to Apache Datafu but there are large differences in features and target runtimes. The target runtime of Apache Datafu is Apache Pig but Hivemall targets Apache Hive, Apache Spark, and Apache Pig for the target runtime. Apache Datafu is more likely to be statistics library and does not support machine learning features such as classification and regression but Hivemall is a machine learning library supporting them.
The initial goals are as follows:
- Establish the project governance in the Apache way and broaden the community
- Improve documentations.
- Adding more unit/scenario tests.
- Handover of code and copyrights
- get I-CLA from the initial committers
- get SGA from other individuals not listed in the initial committers and AIST.
- list all copyrights and licenses in NOTICE file and LICENSE file, respectively.
Hivemall has several on-going WIP features.
Making a parameter server (a kind of distributed key-value store) as Apache YARN application is a major issue. Hivemall’s parameter server is currently a standalone application. Parameter servers on Apache YARN enables to use Hadoop cluster resource efficiently and makes management of parameter servers easier.
Another major WIP issue is integrating XGBoost into Hivemall. We need more works and tests, e.g., supporting cross compilation of native JNI objects of XGBoost.
The project members understand the importance of letting motivated individuals contribute to the project. Since Hivemall was initially released in 2014, it has received contributions from 14 contributors.
Our intent of this incubator proposal is building a diverse developer community following the Apache meritocracy model. We welcome external contributions and plan to elect committers from those who contribute significantly to the project.
While there are 15 contributors in total, there are 3-4 active developers continuously involved for the major feature development at the moment. We hope to extend our contributor base and encourages suggestions and contributions from any potential user.
The current main developers are from employees of Treasure Data, NTT and Hortonworks. Some of them are Hadoop/Pig PMCs and/or Hive committers.
Incubating at ASF is the natural choice for the Hivemall project because the Hivemall is targeting to run on Apache Hive, Apache Spark, and Apache Pig. We encourage integrations with other ASF data processing frameworks like Apache Impala and Apache Drill.
The contributions of the main developer is significant at the moment but the dependencies would decrease as the community grows.
While the main developer is developing Hivemall as a full-time job at TreasureData, the company is well being aware of the open source philosophy and the importance of open governance of open source products. Orphanining ASF product can be considered itself as a risk. Hence, we think the the risks of it being orphaned are minimal.
Inexperience with Open Source
Hivemall also has been developed as an open source project since 2013. The majority of the project member have jobs developing open source products and some of them are working on other ASF projects like Apache Hadoop and Apache Pig. We thus considered that the project members have enough experiences for open source development.
The current list of committers consists of developers from three different companies. The committers are geographically distributed across the U.S. and Asia. They are experienced with working in a distributed environment.
While not included in the initial committer, there are other external contributors to the project. So, we hope to establish a developer community that includes those contributors from several other corporations during the incubation process.
Reliance on Salaried Developers
The major developer is paid by his employer to contribute to this project and the other developers are payed by their employers for Hadoop-related open source development. While they might change their affiliations over time, they are willing to have their expertise for the open source development. So, the project would continue regardless their affiliations.
Relationships with Other Apache Products
Hivemall is a collection for machine learning functions on Apache Hive, Apache Spark, and Apache Pig. Apache MADlib is a collection of machine learning functions for relational databases, i.e., Apache HAWQ and PostgreSQL. There is no conflict in their target runtimes.
A Excessive Fascination with the Apache Brand
Our interest for this incubation is attracting more contributors, building a strong community with open governance, and increasing the visibility of Hivemall in the market/community. We will be sensitive to inadvertent abuse of the Apache brand for any commercial use and will work with the Incubator PMC and project mentors to ensure the brand policies are respected.
Information on Hivemall can be found at: https://github.com/myui/hivemall/wiki
We released the initial version of Hivemall in 2013 at https://github.com/myui/hivemall and introduced Hivemall at the Hadoop Summit 2014.
Source and Intellectual Property Submission Plan
We know no legal encumberment to transfer of the source to Apache. We are going to get Contributor License Agreement (CLA) for all property of Hivemall.
Also, we plan to get a sign from AIST for Software Grant Agreement (SGA).
Hivemall depends on the following third party libraries:
- netty (The MIT License)
- smile (Apache License v2.0)
- org.takuaani.xz (Public Domain)
- xgboost (Apache License v2.0)
- hadoop (Apache License v2.0)
- hive (Apache License v2.0)
- log4j (Apache License v2.0)
- guava (Apache License v2.0)
- lucene-analyzers-kuromoji (Apache License v2.0)
- junit (Eclipse Public License v1.0)
- mockito (The MIT License)
- powermock (Apache License v2.0)
- kryo (BSD License)
Hivemall on Spark:
- spark (Apache License v2.0)
- commons-cli (Apache License v2.0)
- commons-logging (Apache License v2.0)
- commons-compress (Apache License v2.0)
- scala-library (BSD License)
- scalatest (Apache License v2.0)
- xerial-core (Apache License v2.0)
The dependencies all have Apache compatible licenses.
firstname.lastname@example.org (with moderated subscriptions)
JIRA project Hivemall (HIVEMALL)
Makoto Yui (email@example.com)
Takeshi Yamamuro (firstname.lastname@example.org)
Daniel Dai (email@example.com)
Tsuyoshi Ozawa (firstname.lastname@example.org)
Kai Sasaki (email@example.com)
- Makoto Yui
- Kai Sasaki
- Takeshi Yamamuro
- Tsuyoshi Ozawa Apache Hadoop PMC member
- Daniel Dai (ASF member) Apache Pig PMC member
- Roman Shaposhnik (Pivotal, ASF member, IPMC member) Apache Bigtop/Incubator PMC member
- Reynold Xin (Dataricks, ASF member) Apache Spark PMC member
- Markus Weimer (Microsoft, ASF member) Apache REEF PMC member
- Xiangrui Meng (Databricks, ASF member) Apache Spark PMC member
We are requesting the Incubator to sponsor this project.