The following provides instructions on how to install Metron on an existing Ambari-managed HDP 2.4 cluster. These are not recommendations for production deployment but rather for dev/testing environments. 

Prerequisites

For the purposes of this discussion, there will be 7 types of server roles:

1. Hadoop Master
2. Hadoop Slave
3. Ambari Server
4. Metron Installer
5. Metron Search
6. Metron Probe
7. Metron Web

Specifications for Existing Hadoop Cluster

The following are the specifications for the Hadoop related nodes:

1. All cluster nodes must be running CentOS 6.x.
2. The cluster must be running HDP 2.4 managed by Ambari 2.2.x.

3. The cluster must have a minimum of 2 Hadoop Master nodes and 4 Hadoop slave nodes.

4. The minimum spec for each of the Hadoop Slave and Master nodes should be (for Dev/Test/POC envs):
   • 8-16 GB RAM
   • 4 cores
   • 100 GB of storage
5. The following services must be installed across the Hadoop Master and Slave nodes:
   1. 
      

6. Each of the following components must be installed on at least 4 Hadoop Slave Nodes:

   1. 
7. No other workloads should be running on the cluster. In other words, this cluster should be dedicated for Metron jobs.

Specifications for Metron Nodes

1. At least three nodes must be dedicated for Metron-specific components.
2. You should have root access on all Metron nodes.

3. All Metron nodes (Search, Probes, and Web) must have the following:

   • Centos 6.7
   • Oracle JDK 1.8
4. The minimum spec for each of the Metron nodes should be (for Dev/Test/POC envs):
   • 8-16 GB RAM
   • 4 cores
   • 100 GB of storage

5. The Metron Installer node must have Docker installed. (Note: If you are using Centos, 7.x is preferred for the Metron Installer node.)

6. The Metron Installer node requires SSH access to all Hadoop Cluster and Metron nodes. Passwordless is preferred.

Sample Deployment Architecture

Based on the above specs, here is a sample deployment architecture.

Install Instructions

Step 1: Setup

For the following instructions, set the following environmental variables. Plug in the necessary values for your deployment.

export METRON_HOME=/root/metron/incubator-metron-Metron_0.2.0BETA_rc2
export METRON_0_2_RELEASE_BINARIES=https://dist.apache.org/repos/dist/dev/incubator/metron/0.2.0BETA-RC2-incubating/apache-metron-0.2.0BETA-RC2-incubating.tar.gz
export INSTALL_CONF_DIRECTORY=my_install

Step 2: Install or Validate Existing Hadoop Cluster

1. Install a new HDP 2.4 using the following instructions or ensure you have an existing HDP cluster that follows the specs outlined in the pre-requisites section. 

Step 3: Download and Build Metron

1. Log into the Metron Installer node.
2. Verify Docker is installed and running. 
3. Ensure that this node has ssh access to each of the Metron and Hadoop/HDP nodes. 
4. Download the Metron Binaries from $METRON_0_2_RELEASE_BINARIES
5. Untar it: tar -zxvf apache-metron-0.2.0BETA-RC2-incubating.tar.gz
6. cd $METRON_HOME
7. Run Docker installer image:
    docker run -it -v $METRON_HOME:/root/incubator-metron dlyle65535/metron-ansible:2.0.0.2 bash
8. Build Metron
   cd /root/incubator-metron/
   mvn clean package -DskipTests 

Step 4: Configure Metron for Install

1. cd /root/incubator-metron/metron-deployment/inventory
2. create a directory called $INSTALL_CONF_DIRECTORY that will hold all of your metron deployment configuration
3. Copy the sample configs into the new directory:
   1. cp - R /root/incubator-metron/metron-deployment/inventory/metron_example/* /root/incubator-metron/metron-deployment/inventory/$INSTALL_CONF_DIRECTORY
4. Setup your inventory hosts by editing the file $METRON_HOME/metron-deployment/inventory/$INSTALL_CONF_DIRECTORY/hosts

   1. Configure with your cluster details by defining which hosts the  Metron services will be installed on by updating these groups:

      [ambari_master] - host running Ambari
      [ambari_slaves] - all Ambari-managed hosts
      [metron_hbase_tables] - host used to create the HBase tables required by Metron. Requires a HBase client.
      [metron_kafka_topics] - host used to create the Kafka topics required by Metron. Requires a Kafka broker.
      [enrichment] - submits the topology code to Storm and requires a Storm client
      [search] - host(s) where Elasticsearch will be installed
      [web] - host where the Metron UI and underlying services will be installed
      [sensors] - host where network data will be collected and published to Kafka

       

   2. Setup your environment configuration by editing the file $METRON_HOME/metron-deployment/inventory/$INSTALL_CONF_DIRECTORY/group_vars/all 

      1. Commonly Used Settings:

         Ansible

         • • • • ansible_ssh_private_key_file: /Path/to/private/key/file Point to the private key file for ssh user on the target hosts
               • ansible_ssh_user: root The name of the ssh user on the target hosts (requires sudo)

         Ambari

         • • • • ambari_port: 8080 Change if your Ambari instance uses a non-default port
               • ambari_user: admin Change to user on your Ambari instance
               • ambari_password: admin Change to password for your Ambari user above

         Kafka

         • • • • num_partitions: 3 Change to your desired number of partitions
               • retention_in_gb: 25 Change to your desired retention size

         Metron

         • • • • java_home: /usr/jdk64/jdk1.8.0_40 Location of Java on all hosts

         Sensors

         • • • • sensor_test_mode: True Change to false if not running traffic replay
               • sniff_interface: eth0 Interface that the Metron sensors will sniff on the [sensors] host

         Search

         • • • • elasticsearch_network_interface: eth0  Bind interface for the Elasticsearch host(s)
   3. Ensure that you do't have duplicate entries of sniff_interface

Step 5: Install/Deploy Metron to the Cluster

Note: Adding -u user -k to the below command lines allows deployment in environments where passwordless SSH has not been enabled.

1. cd /root/incubator-metron/metron-deployment/playbooks
2. Run the following ansible command to deploy Metron with the standard services (Metron Topologies, WebUI, and Monit)
   ansible-playbook -i /root/incubator-metron/metron-deployment/inventory/$INSTALL_CONF_DIRECTORY metron_install.yml --skip-tags="solr"
3. Run the following Ansible command to deploy Metron without installing Sensors
    ansible-playbook -i /root/incubator-metron/metron-deployment/inventory/$INSTALL_CONF_DIRECTORY metron_install.yml --skip-tags="solr,sensors"