Differences between revisions 15 and 16
Revision 15 as of 2015-02-10 14:34:16
Size: 4439
Comment: update release numbers
Revision 16 as of 2016-03-03 17:00:04
Size: 4407
Comment:
Deletions are marked like this. Additions are marked like this.
Line 7: Line 7:
in Spark mode using [[http://spark.apache.org/|Apache Spark]], and in Flink mode using [[http://flink.apache.org|Apache Flink]] (only in MRQL-0.9.4-incubating). in Spark mode using [[http://spark.apache.org/|Apache Spark]], and in Flink mode using [[http://flink.apache.org|Apache Flink]].
Line 9: Line 9:
The latest stable MRQL version, MRQL-0.9.4-incubating, is compatible with the following Apache releases: The latest stable MRQL version, MRQL-0.9.6-incubating, is compatible with the following Apache releases:
Line 12: Line 12:
The BSP and Spark modes are optional. The BSP mode is compatible with Apache Hama 0.6.2, 0.6.3, and 0.6.4. The BSP and Spark modes are optional. The BSP mode is compatible with Apache Hama 0.6.2, 0.6.3, 0.6.4, and 0.7.0.
Line 14: Line 14:
The Spark mode is compatible with Apache Spark 1.0.0 through 1.2.0. The Spark mode is compatible with Apache Spark 1.0.0 through 1.6.0.
Line 16: Line 16:
The Flink mode is compatible with Apache Flink 0.6-incubating through 0.8.0 in local and Yarn modes. The Flink mode is compatible with Apache Flink 0.10.1 and 0.10.2 in local and Yarn modes.
Line 25: Line 25:
The MRQL 0.9.6 binary release uses Hadoop 2.7.1 (Yarn), Hama 0.7.0, Spark 1.6.0, and Flink 0.10.2
Line 54: Line 55:
First, start the Flink application manager on Yarn using {{{${FLINK_HOME}/bin/yarn-session.sh -n #_of_nodes}}} (see [[http://flink.apache.org/docs/0.8/yarn_setup.html|Yarn Setup]]).
The run the [[Pagerank|PageRank example]] or the [[Kmeans|k-means clustering example]] using the {{{bin/mrql.flink}}} script.
Change the configuration file {{{conf/mrql-env.sh}}} to match your Flink installation. Then run the [[Pagerank|PageRank example]] or the [[Kmeans|k-means clustering example]] using the {{{bin/mrql.flink}}} script.
Line 80: Line 80:
Currently, the "mvn install" builds MRQL with Hadoop 1.2.1, Hama 0.6.4, Spark 1.2.0, and Flink 0.8.0.
To build MRQL on some other Hadoop 1.x, such as 1.0.3, use:
Currently, the "mvn install" in MRQL 0.9.6 builds MRQL using Hadoop 2.7.1 (Yarn), Hama 0.7.0, Spark 1.6.0, and Flink 0.10.2.
To build MRQL on Hadoop 1.x, such as 1.0.3, use:
Line 84: Line 84:
mvn -Dhadoop.version=1.0.3 clean install mvn -Dhadoop1 -Dhadoop.version=1.0.3 clean install
Line 87: Line 87:
To build MRQL on Hadoop 2.x (yarn) use: To build MRQL on another Hadoop 2.x (yarn) use:
Line 90: Line 90:
mvn -Dyarn -Dyarn.version=2.2.0 -Dhadoop.version=1.2.1 clean install mvn -Dyarn.version=2.2.0 clean install

Getting Started with MRQL

Installation Instructions

Apache MRQL can run in 4 modes: in Map-Reduce mode using Apache Hadoop, in BSP mode (Bulk Synchronous Parallel mode) using Apache Hama, in Spark mode using Apache Spark, and in Flink mode using Apache Flink.

The latest stable MRQL version, MRQL-0.9.6-incubating, is compatible with the following Apache releases: The MRQL MapReduce mode is compatible with Apache Hadoop releases 1.x and 2.x (Yarn). You can download the Hadoop tarball from Apache Hadoop. The BSP and Spark modes are optional. The BSP mode is compatible with Apache Hama 0.6.2, 0.6.3, 0.6.4, and 0.7.0. You can download the latest tarball from Apache Hama. The Spark mode is compatible with Apache Spark 1.0.0 through 1.6.0. You can download the latest tarball prebuilt for Hadoop1 or Hadoop2 from Apache Spark. The Flink mode is compatible with Apache Flink 0.10.1 and 0.10.2 in local and Yarn modes. You can download the latest tarball prebuilt for Hadoop2 from Apache Flink.

The following instructions assume that you have already installed Hadoop MapReduce and you have deployed it on your cluster successfully.

How to install MRQL

Download the latest stable MRQL binary release from http://www.apache.org/dyn/closer.cgi/incubator/mrql and extract the files. The MRQL 0.9.6 binary release uses Hadoop 2.7.1 (Yarn), Hama 0.7.0, Spark 1.6.0, and Flink 0.10.2 The scripts bin/mrql, bin/mrql.bsp, bin/mrql.spark, and bin/mrql.flink evaluate MRQL queries in Hadoop, Hama, Spark, and Flink modes, respectively.

How to run MRQL on a Hadoop MapReduce cluster:

Change the configuration file conf/mrql-env.sh to match your Hadoop installation. For a test, run the PageRank example or the k-means clustering example on a small Hadoop MapReduce cluster.

How to run MRQL on a Hama cluster:

  • Follow the instructions in Getting Started with Hama to set up and start Hama.

  • Change the configuration file conf/mrql-env.sh to match your Hama installation.

For a test, run the PageRank example or the k-means clustering example on a Hama cluster.

How to run MRQL on a Spark standalone cluster:

  • Follow the instructions in Spark Standalone Mode to set up and start Spark in standalone deploy mode.

  • Change the configuration file conf/mrql-env.sh to match your Spark installation.

For a test, run the PageRank example or the k-means clustering example on a Spark cluster.

How to run MRQL in Spark mode on a Yarn cluster:

Set SPARK_MASTER=yarn-client in conf/mrql-env.sh (see Running Spark on YARN).

Change the configuration file conf/mrql-env.sh to match your Flink installation. Then run the PageRank example or the k-means clustering example using the bin/mrql.flink script.

How to Recompile MRQL

Download the latest stable MRQL source release from http://www.apache.org/dyn/closer.cgi/incubator/mrql and extract the files. You can get the latest source code using:

git clone https://git-wip-us.apache.org/repos/asf/incubator-mrql.git

To build MRQL using maven, use

mvn clean install

To validate the installation, use

mvn -DskipTests=false clean install

which runs the queries in tests/queries in memory, local Hadoop mode, local Hama mode, local Spark mode, and local Flink mode.

Currently, the "mvn install" in MRQL 0.9.6 builds MRQL using Hadoop 2.7.1 (Yarn), Hama 0.7.0, Spark 1.6.0, and Flink 0.10.2. To build MRQL on Hadoop 1.x, such as 1.0.3, use:

mvn -Dhadoop1 -Dhadoop.version=1.0.3 clean install 

To build MRQL on another Hadoop 2.x (yarn) use:

mvn -Dyarn.version=2.2.0 clean install 

GettingStarted (last edited 2016-03-03 17:00:04 by LeonidasFegaras)