user6676898 user6676898 - 1 month ago 16
Java Question

A connection error in remote mode of Titan-1.0.0+Hbase-0.98.20 using java

I am learning Titan database. I have run it successfully in local-mode. Now, I am trying to use Titan database in "Remote Server Mode" introduced in Titan-documentation. My Titan version is Titan-1.0.0-hadoop1.

I have clusters in my LAN including

cloud12
and
cloud13
. I installed hadoop-1.2.1 on it, the master is
cloud12
and the slave is
cloud13
.

I want to test the performance about create a graph, so I design to start my Hbase-0.98.20 in pseudo-distributed-mode on machine
cloud12
with independent zookeeper-3.4.6 and elasticsearch on
cloud12
.(I modified
hbase-env.sh
, and use default port
2181
in
zoo.cfg
)

Hadoop and HBase are seems like working regularly, I checked two servers by Jps and I also checked HBase through HBase shell.

Here are my configuration of
hbase-site.xml
:

<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://cloud12:9000/hbase</value>
</property>

<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>

<property>
<name>hbase.master.port</name>
<value>60000</value>
</property>

<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/home/Titan/hbase/zookeeperDir</value>
</property>

<property>
<name>hbase.tmp.dir</name>
<value>/home/Titan/hbase/tmpDir</value>
</property>

<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>

<property>
<name>zookeeper.znode.parent</name>
<value>/hbase-unsecure</value>
</property>

<property>
<name>hbase.zookeeper.property.maxClientCnxns</name>
<value>600</value>
</property>
</configuration>


Although when I ran my program in Eclipse on the other machine in LAN which named
cloud6
(I installed Titan-1.0.0 on this machine), a confused error information printed in my output logs.

Here is the error constantly repeated in my output


789 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=192.168.12.148:2181 sessionTimeout=90000 watcher=hconnection-0x12d3a4e9, quorum=192.168.12.148:2181, baseZNode=/hbase

870 [main] INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Process identifier=hconnection-0x12d3a4e9 connecting to ZooKeeper ensemble=192.168.12.148:2181

878 [main-SendThread(192.168.12.148:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 192.168.12.148/192.168.12.148:2181. Will not attempt to authenticate using SASL (unknown error)

1030 [main-SendThread(192.168.12.148:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to 192.168.12.148/192.168.12.148:2181, initiating session

1049 [main-SendThread(192.168.12.148:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server 192.168.12.148/192.168.12.148:2181, sessionid = 0x15654717951001b, negotiated timeout = 40000

1054 [main] INFO org.apache.hadoop.hbase.client.ZooKeeperRegistry - ClusterId read in ZooKeeper is null


ClusterId read in ZooKeeper is null? I really confused about it. I tried to solve problem and I found my error are similar to many others, but I can't get clear answer from Google or other websites. Is there an error in my design of architecture? or configuration error?

I can ensure that my hosts and the time of cluster are all correct. Here is a part of my program to connect Hbase on
cloud12
from
cloud6
, is anything wrong or lackness in my code ?

public static final String INDEXNAME = "search";
...
BaseConfiguration conf=new BaseConfiguration();
conf.setProperty("storage.backend", "hbase");
conf.setProperty("storage.hostname", "192.168.12.148");//ip of cloud12
conf.setProperty("storage.tablename", "graph1");
conf.setProperty("index." + INDEXNAME + ".backend", "elasticsearch");
conf.setProperty("index." + INDEXNAME + ".hostname", "192.168.12.148");
conf.setProperty("index." + INDEXNAME + ".elasticsearch.local-mode", false);
conf.setProperty("index." + INDEXNAME + ".elasticsearch.client-only", true);

Answer

Some things to check since you are using an external Zookeeper ensemble:

  1. Add the hbase.zookeeper.quorum property value in hbase-site.xml. The value should be a comma-separated list of your Zookeeper nodes. It defaults to localhost.

  2. Add export HBASE_MANAGES_ZK=false to the hbase-env.sh. It defaults to true.

  3. Make sure that the zookeeper.znode.parent property value in hbase-site.xml matches the value in your Titan configuration with storage.hbase.ext.zookeeper.znode.parent. It defaults to /hbase. If these values don't match, the Titan connection to HBase will hang.

  4. Make sure that the clientPort property value in zoo.cfg matches the hbase.zookeeper.property.clientPort value in hbase-site.xml and also matches the storage.hbase.ext.hbase.zookeeper.property.clientPort value in your Titan configuration. It defaults to 2181. If these don't match, you would see connection exceptions in the logs.

  5. Make sure that the Zookeeper nodes are listening on the clientPort using an accessible IP address (not localhost).

Once you've verified that you can connect to your Titan table, I think that the INFO messages are safe to ignore. You can toggle the logging level through settings in log4j.properties:

log4j.logger.org.apache.zookeeper=WARN
log4j.logger.org.apache.hadoop.hbase.client=WARN
log4j.logger.org.apache.hadoop.hbase.zookeeper=WARN

There is also an open issue against Titan to investigate why the Zookeeper client connections are happening so frequently.