You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Next »

Overview

Hive Replication builds on the MetaStore event and ExIm features to provide a framework for replicating Hive metadata and data changes between clusters. There is no requirement for the source cluster and replica to run the same Hadoop distribution, Hive version, or MetaStore RDBMS. The replication system has a fairly 'light touch', exhibiting a low degree of coupling and using the hive-metastore Thrift service as an integration point. However, the current implementation is not an 'out of the box' solution. In particular it is necessary to provide some kind of orchestration service that is responsible for requesting replication tasks and executing them.

Prerequisites

  • You must be running Hive 1.1.0 or later at your replication source (for DbNotificationListener support).
  • You must be running Hive 0.8.0 or later at your replication destination (for IMPORT support).
  • You'll require Hive 1.2.0 or later JAR dependencies to instantiate and execute ReplicationTasks. This is not a cluster requirement, you'll need this only for the service orchestrating the replication.
  • You will initially require administration privileges on the source cluster to enable the writing of notifications to the MetaStore database.

Limitations

  • While the MetaStore events feature allows the sinking of notifications to anything implementing MetaStoreEventListener, the implementation of Replication feature can only source events from the MetaStore database and hence the DbNotificationListener must be used.
  • Data appended to tables or partitions using the HCatalogWriters will not be automatically replicated as they do not currently generate MetaStore notifications (HIVE-9577). This is likely only a consideration if data is being written to table by processes outside of Hive.

Configuration

To configure the persistence of MetaStore notification events it is necessary to set the following hive-site.xml properties on the source cluster. A restart of the MetaStore service will be required for the settings to take effect.

hive-site.xml configuration for replication
  <property>
    <name>hive.metastore.event.listeners</name>
    <value>org.apache.hive.hcatalog.listener.DbNotificationListener</value>
  </property>
  <property>
    <name>hive.metastore.event.db.listener.timetolive</name>
    <value>86400s</value>
  </property>

Typical mode of operation

  • With the MetaStore event configuration in place on the source cluster, the NOTIFICATION_LOG table in the MetaStore will be populated with events on the successful execution of metadata operations such as CREATE, ALTER, and DROP.
  • These events can be read and converted into ReplicationTasks using org.apache.hive.hcatalog.api.HCatClient.getReplicationTasks(long, int, String, String).
  • ReplicationTasks encapsulate a set of commands to execute on the source Hive instance (typically to export data) and another set to execute on the replica instance (typically to import data). The commands are provided as HQL strings.
  • The ReplicationTask also serves as a place where database and table names mappings can be declared and StagingDirectoryProvider implementations configured for the resolution of paths at both the source and destination:
    • org.apache.hive.hcatalog.api.repl.ReplicationTask.withDbNameMapping(Function<String, String>)
    • org.apache.hive.hcatalog.api.repl.ReplicationTask.withTableNameMapping(Function<String, String>)
    • org.apache.hive.hcatalog.api.repl.ReplicationTask.withSrcStagingDirProvider(StagingDirectoryProvider)
    • org.apache.hive.hcatalog.api.repl.ReplicationTask.withDstStagingDirProvider(StagingDirectoryProvider)
  • The HQL commands provided by the tasks must then be executed against the source Hive and then the destination (aka the replica). One way of doing this is to open up a JDBC connection to the respective HiveServer and submit the tasks HQL queries.
  • It is necessary to maintain the position within the notification log so that replication tasks are applied only once. This can be achieved by maintaining a record of the last successfully executed event's id (task.getEvent().getEventId()) and providing this as an offset this when sourcing the next batch of events.

Replication to AWS/EMR/S3

At this time it is not possible to replicate to tables on EMR that have a path location on S3. This is due to a bug in the dependency of the IMPORT command in the EMR distribution (checked in AMI-4.2.0).

 

  • No labels