HBase wire compatibility meeting notes

2/21/2012

Versioning:

0.94: just make it a performance release

keep wire compatibility with 92 for:

  • 0.92 clients to 0.94 servers
  • 0.94 clients to 0.92 servers (we’ll check if this is possible, but we think so.. maybe one or two spots. Lars and Stack to investigate)
  • rolling upgrade within the cluster for advanced users only (eg they have to turn off prefix compression during upgrade, etc)

0.96: “the singularity”

  • aim to have stuff moved over to protobuf at this time, and probably remove old writable-based calls.
  • try to remove ROOT, maybe HBASE-2600 if it got done?
  • migration: perhaps offline, but has to be in-place, and preferably automatically done (eg master rewrites META)

ideas for compatibility of old clients to 0.96 (to help people move to singularity):

  • maybe we backport a forward-compatible client onto 0.94 or 0.92 releases (rather than trying to make 0.96 back-compatible). eg. if we remove ROOT, add a forward-compatibility path which can use either ROOTless or with-ROOT, make people upgrade to 0.94.X client, then can upgrade server

Tasks:

Core types:

Actual conversion to protobuf RPCs:

  • HRegionInterface - Jimmy
  • HMasterInterface - Greg
  • HMasterRegionInterface - Greg (while doing this, collapse RPCs where necessary)

Zookeeper data:

  • Client-facing stuff: data in ROOT region location node, master address node (Stack)
  • Cluster-internal stuff:
  • /hbase/regionservers znodes
  • distributed log splitting status, etc

Filters: (tricky because they’re user-definable?)

  • todd to investigate

Coprocessors:

  • this is somewhat tricky
  • Enis to investigate

Security:

salesforce folks to investigate (might come for free)

Replication:

  • there’s stuff in ZK, also uses client APIs (Lars/Chris/Jesse)

RPC payload + envelope

  • Devaraj
  • (move the headers, etc to Protobuf)

Management stuff for multi-version:

  • add version string for each node in the cluster to ZK
  • display this info on web UIs, JMX, etc

HBase shell: (Chris)

  • has to know to deserialize the proto types instead, for example? we’re not sure if it ever reads/deserializes stuff directly

On-disk formats:

  • eg reference files use Writable
  • HLog: HLogKey, WALEdits, etc (affects unclean upgrade, and replication)
  • HFile metadata - currently it’s a MapWritable - need to figure out how to keep this back-compat against Writable
  • /hbase/hbase.version file

How to integrate with Thrift? (facebook to investigate)

  • No labels