Configuring Ambari Metrics service in distributed mode

In distributed mode, Metric Collector writes go to HDFS of the cluster. Currently distributed mode does not support multi-mode Metric Collector, however the plan is to allow Metric Collector to scale horizontally to allow a multi-node HBase storage layer.

Note: Make sure there is a local Datanode hosted with the Collector, it provides AMS HBase the distinct advantage of write and reads sharded across the data volumes available to the DN.

Following steps need to be performed either at install time or after deploy to configure Metric Collector in distributed mode. Note: If configuring after install, the data will not be automatically copied over to HDFS.

  1. Edit ams-site, Set timeline.metrics.service.operation.mode = distributed
  2. Edit ams-hbase-site,
    1. Set hbase.rootdir = hdfs://<namenode-host>:8020/user/ams/hbase [ If NN HA is enabled, hdfs://<nameservice-id>/user/ams/hbase ]
      (Note: /user/ams/hbase here is the directory where metric data will be stored in HDFS)
    2. Set hbase.cluster.distributed = true
    3. Add dfs.client.read.shortcircuit = true (This is an optimization with a local DN present)
  3. Restart Metrics Collector

Note: In Ambari 2.0.x, there is a bug in deploying AMS in distributed mode, if Namenode HA is enabled. Please follow the instruction listed in this JIRA as workaround steps: (AMBARI-10707)

Note: In Ambari 2.2.1, stack advisor changes the dependent configs for distributed mode automatically through recommendations. Ideally, the only config that needs to be changed is timeline.metrics.service.operation.mode = distributed. The other configs - hbase.rootdir, hbase.cluster.distributed and dfs.client.read.shortcircuit will be changed automatically.

Migrating data from embedded to distributed mode

Steps to migrate existing metric data to HDFS and starting AMS in distributed mode:

  • Stop AMS Metric Collector

  • Create hdfs directory for ams user, Example:

    su - hdfs -c 'hdfs dfs -mkdir /user/ams'
    su - hdfs -c 'hdfs dfs -chown ams:hadoop /user/ams'

  • Copy the metric data from the AMS local directory (existing value of hbase.rootdir in ams-hbase-site) to HDFS directory. Example:

    cd /var/lib/ambari-metrics-collector/
    su - hdfs -c 'hdfs dfs -copyFromLocal hbase hdfs://<namnode-http-address>:8020/user/ams/'
    su - hdfs -c 'hdfs dfs -chown -R ams:hadoop /user/ams/hbase'

  • Start the Metric Collector after making the changes needed for distributed mode.

  • No labels