Frangipanes Frangipanes - 23 days ago 7
Java Question

Unable to connect to a Kerberos-secured Phoenix datasource

I want to test pulling data from Apache HBase with a Java application. The application will use SQL-like queries via a JDBC to Apache Phoenix.

I've set up my Hadoop "cluster" on one machine using Ambari and the HortonWorks HDP 2.5 platform. I've also Kerberized the environment using Ambari's wizard, where my KDC is a seperate machine running Windows Active Directory.

Ambari shows no errors, and I am able to use

sqlline.py
to successfully make SQL-like calls to HBase through Phoenix. I set up some example tables this way (cf. HortonWorks Phoenix & ODBC tutorial, although I had to
kinit
etc. first).

However, I am having problems creating a JDBC datasource to be used by the Java application. In my case, I am planning to host the webapp on WildFly 10.1 and I am developing with Eclipse JEE with the JBoss Tools plugin.




These are the steps I used to create the datasource:

Datasource Explorer > Database Connections > New...


  • Connection Profile: Generic JDBC

  • URL:
    jdbc:phoenix:hdfs.eaa.local:2181/hbase-secure:HTTP/hbase.eaa.local@EAA.LOCAL:jboss.server.temp.dir/spnego.service.keytab

  • Username: hbase -I'm unsure what to put here-

  • Driver: I've created a new driver of the type "Generic JDBC Driver" and I had to add JAR files for all of the dependencies of
    phoenix-core-[version].jar
    . The Driver Class is
    org.apache.phoenix.jbdc.PhoenixDriver
    .



I got the connection string from an extant post in the HortonWorks community, which is why it includes the Kerberos principal and keytab used for the connection.




When I try to test the datasource connection, it churns for about 5 minutes before spitting out an error message (after something like 35 attempts). The client returns Java exceptions that the sockets are in a "closing state", and the Zookeeper logs are less helpful:

INFO [SyncThread:0:ZooKeeperServer@617] - Established session 0x157aef451560217 with negotiated timeout 40000 for client /192.168.40.3:52674
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /192.168.40.41:43860
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing ruok command from /192.168.40.41:43860
INFO [Thread-1448:NIOServerCnxn@1007] - Closed socket connection for client /192.168.40.41:43860 (no session established for client)

INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /192.168.40.41:43922
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to establish new session at /192.168.40.41:43922
INFO [SyncThread:0:ZooKeeperServer@617] - Established session 0x157aef451560218 with negotiated timeout 40000 for client /192.168.40.41:43922
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@118] - Successfully authenticated client: authenticationID=hbase/hdfs.eaa.local@EAA.LOCAL; authorizationID=hbase/hdfs.eaa.local@EAA.LOCAL.
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@134] - Setting authorizedID: hbase
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@964] - adding SASL authorization for authorizationID: hbase
INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@494] - Processed session termination for sessionid: 0x157aef451560218
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /192.168.40.41:43922 which had sessionid 0x157aef451560218

INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /192.168.40.41:44008
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing ruok command from /192.168.40.41:44008
INFO [Thread-1449:NIOServerCnxn@1007] - Closed socket connection for client /192.168.40.41:44008 (no session established for client)


NB. 192.168.40.3 is the VPN server, which my host machine is using to tunnel into the environment with the Hadoop cluster. 192.168.40.41 is the machine running the cluster,
hdfs.eaa.local
.

There are plenty of accepted socket connections which are then immediately closed. Occasionally the client authenticates successfully (so I'm confident in my Kerberos settings) but then there is a session termination immediately afterward.

I've also tried to deploy the Datasource directly in WildFly with
jboss-cli
and
standalone.xml
and
module.xml
modifications. But I get lots of problems with missing dependencies that I'm not sure how to resolve without creating a new module for each required JAR (and there are a lot) for
phoenix-core-[version].jar
. I followed this guide.

What can I do to fix the issue or diagnose further? I've been pulling my hair out for a couple of days now.

Answer

You need to add hbase-site.xml and core-site.xml to your classpath.

See How to connect to a Kerberos-secured Apache Phoenix data source with WildFly? for more information.