Frankie Frankie - 2 months ago 22
Java Question

Deadlock when connect to Hbase in Spark

I am trying to connect to HBase in Spark application. I am using HDP(hortonworks data platform). versions: Spark 1.5.2, HBase 1.1.2.
my codes are below:

import org.apache.spark.{SparkConf, Logging, SparkContext}
import org.apache.spark.sql.SQLContext
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.{HColumnDescriptor, HTableDescriptor, TableName, HBaseConfiguration}
import org.apache.hadoop.hbase.client.{HBaseAdmin, Put, ConnectionFactory, Table}

object HBaseSource {
def main(args: Array[String]){
val sparkConf = new SparkConf().setAppName("HBase Test")
val sc = new SparkContext(sparkConf)
val sqlContext = new SQLContext(sc)

import sqlContext._
import sqlContext.implicits._

val hConf = HBaseConfiguration.create
if(hConf == null){
println("null pointer")
}else{
println("not null pointer")
}

hConf.set("zookeeper.znode.parent", "/hbase-secure")
hConf.set("hbase.zookeeper.quorum", "zookeeper quorum's IP address")
hConf.set("hbase.master", "master's IP address")
hConf.set("hbase.zookeeper.property.clientPort", "2181")
hConf.addResource("/usr/hdp/curent/hbase-master/conf/hbase-site.xml")
hConf.addResource("/usr/hdp/current/hbase-master/conf/hdfs-site.xml")
hConf.addResource("/usr/hdp/current/hbase-master/conf/core-site.xml")

println("beginning of the test")
HBaseAdmin.checkHBaseAvailable(hConf)
}
}


I am using HBaseAdmin.checkHBaseAvailable(hConf) to check if HBase is good.
I build with maven and my maven POM file is shown below:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.kpmg.tabi</groupId>
<artifactId>kshc1</artifactId>
<version>0.0.1-SNAPSHOT</version>

<build>
<plugins>
<!-- any other plugins -->
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
</plugin>
</plugins>
</build>

<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.5.2</version>
</dependency>

<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>1.5.2</version>
</dependency>

<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>1.1.2</version>
<exclusions>
<exclusion>
<groupId>asm</groupId>
<artifactId>asm</artifactId>
</exclusion>
<exclusion>
<groupId>org.jboss.netty</groupId>
<artifactId>netty</artifactId>
</exclusion>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty</artifactId>
</exclusion>
<exclusion>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
</exclusion>
<exclusion>
<groupId>org.jruby</groupId>
<artifactId>jruby-complete</artifactId>
</exclusion>
</exclusions>
</dependency>

</dependencies>

</project>


I use
spark-submit
to execute my job. command to submit is
spark-submit --class HBaseSource appName-jar-with-dependencies.jar
. Then I got a deadlock, this is part of log information. After this, this app gets stuck and never goes forward.

16/08/15 17:03:02 INFO Executor: Starting executor ID driver on host localhost
16/08/15 17:03:02 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 56566.
16/08/15 17:03:02 INFO NettyBlockTransferService: Server created on 56566
16/08/15 17:03:02 INFO BlockManagerMaster: Trying to register BlockManager
16/08/15 17:03:02 INFO BlockManagerMasterEndpoint: Registering block manager localhost:56566 with 530.0 MB RAM, BlockManagerId(driver, localhost, 56566)
16/08/15 17:03:02 INFO BlockManagerMaster: Registered BlockManager
not null pointer
beginning of the test
16/08/15 17:03:02 INFO RecoverableZooKeeper: Process identifier=hconnection-0x5c60b0a0 connecting to ZooKeeper ensemble=10.1.188.121:2181
16/08/15 17:03:02 INFO ZooKeeper: Client environment:zookeeper.version=3.4.6-4--1, built on 02/11/2016 06:47 GMT
16/08/15 17:03:02 INFO ZooKeeper: Client environment:host.name=useomlxd00007.nix.us.kworld.kpmg.com
16/08/15 17:03:02 INFO ZooKeeper: Client environment:java.version=1.8.0_71
16/08/15 17:03:02 INFO ZooKeeper: Client environment:java.vendor=Oracle Corporation
16/08/15 17:03:02 INFO ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-1.b15.el6_7.x86_64/jre
16/08/15 17:03:02 INFO ZooKeeper: Client environment:java.class.path=/usr/hdp/current/spark-client/conf/:/usr/hdp/2.3.4.7-4/spark/lib/spark-assembly-1.5.2.2.3.4.7-4-hadoop2.7.1.2.3.4.7-4.jar:/usr/hdp/2.3.4.7-4/spark/lib/datanucleus-rdbms-3.2.9.jar:/usr/hdp/2.3.4.7-4/spark/lib/datanucleus-api-jdo-3.2.6.jar:/usr/hdp/2.3.4.7-4/spark/lib/datanucleus-core-3.2.10.jar:/usr/hdp/current/hadoop-client/conf/:/usr/hdp/2.3.4.7-4/hadoop/lib/hadoop-lzo-0.6.0.2.3.4.7-4.jar
16/08/15 17:03:02 INFO ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
16/08/15 17:03:02 INFO ZooKeeper: Client environment:java.io.tmpdir=/tmp
16/08/15 17:03:02 INFO ZooKeeper: Client environment:java.compiler=<NA>
16/08/15 17:03:02 INFO ZooKeeper: Client environment:os.name=Linux
16/08/15 17:03:02 INFO ZooKeeper: Client environment:os.arch=amd64
16/08/15 17:03:02 INFO ZooKeeper: Client environment:os.version=2.6.32-573.8.1.el6.x86_64
16/08/15 17:03:02 INFO ZooKeeper: Client environment:user.name=username
16/08/15 17:03:02 INFO ZooKeeper: Client environment:user.home=homePath
16/08/15 17:03:02 INFO ZooKeeper: Client environment:user.dir=dirPath
16/08/15 17:03:02 INFO ZooKeeper: Initiating client connection, connectString=ipAddr:2181 sessionTimeout=90000 watcher=hconnection-0x5c60b0a00x0, quorum=ipAddr:2181, baseZNode=/hbase-secure
16/08/15 17:03:02 INFO ClientCnxn: Opening socket connection to server ipAddr/ipAddr:2181. Will not attempt to authenticate using SASL (unknown error)
16/08/15 17:03:02 INFO ClientCnxn: Socket connection established to 10.1.188.121/10.1.188.121:2181, initiating session
16/08/15 17:03:02 INFO ClientCnxn: Session establishment complete on server 10.1.188.121/10.1.188.121:2181, sessionid = 0x1568e5a3fde008c, negotiated timeout = 40000
16/08/15 17:03:03 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
16/08/15 17:03:03 INFO ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x1568e5a3fde008c
16/08/15 17:03:03 INFO ZooKeeper: Session: 0x1568e5a3fde008c closed
16/08/15 17:03:03 INFO ClientCnxn: EventThread shut down


I am sure IP address and Port number are correct, because I am able to use Java API to connect to HBase successfully. My question is why this small simple spark app gets a deadlock. Are there other ways in spark connecting to HBase? Any help is appreciated.

Answer

This error is because version mismatch. I build my project with dependencies declared in my POM file and include external jars into my project by using this plugin.

      <plugin>
        <artifactId>maven-assembly-plugin</artifactId>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>single</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <descriptorRefs>
            <descriptorRef>jar-with-dependencies</descriptorRef>
          </descriptorRefs>
        </configuration>
      </plugin>

When I delete this plugin to exclude jars declared in maven and use libs provided in the cluster, the problem is solved. But I am still wondering why this happens because version of libs declared in the maven is then same as libs provided in the cluster.

Anyway this problem seems lib version mismatch.