Wenyuan Chen Wenyuan Chen - 21 days ago 16
Java Question

Phoenix connection throws error message "Failed to find any Kerberos tgt" after running for some time

I am using Phoenix to connect to secure HBase. It runs well when the client starts up. I can connect to HBase an query the data from HBase. The Phoenix JDBC connection is in the connection pool and doesn't close. But after a few hours, I can't query anything from HBase using the same connection. I have to restart the client in order to make my application work. The error messages are:

Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions:
Fri Nov 11 06:24:01 CST 2016, RpcRetryingCaller{globalStartTime=1478867041301, pause=100, retries=1}, java.io.IOException: Could not set up IO Streams to <regionserver>/<ip_address>:60020
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147) ~[hbase-client-1.0.1.jar:1.0.1]
at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:95) ~[hbase-client-1.0.1.jar:1.0.1]
at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callMethod(CoprocessorRpcChannel.java:56) ~[hbase-client-1.0.1.jar:1.0.1]
at org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService$Stub.getTable(MetaDataProtos.java:11769) ~[phoenix-core-4.5.1-HBase-1.0.jar:4.5.1-HBase-1.0]
at org.apache.phoenix.query.ConnectionQueryServicesImpl$7.call(ConnectionQueryServicesImpl.java:1301) ~[phoenix-core-4.5.1-HBase-1.0.jar:4.5.1-HBase-1.0]
at org.apache.phoenix.query.ConnectionQueryServicesImpl$7.call(ConnectionQueryServicesImpl.java:1288) ~[phoenix-core-4.5.1-HBase-1.0.jar:4.5.1-HBase-1.0]
at org.apache.hadoop.hbase.client.HTable$16.call(HTable.java:1737) ~[hbase-client-1.0.1.jar:1.0.1]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_79]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79]
... 1 common frames omitted
Caused by: java.io.IOException: Could not set up IO Streams to <regionserver>/<ip_address>:60020
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:772) ~[hbase-client-1.0.1.jar:1.0.1]
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:880) ~[hbase-client-1.0.1.jar:1.0.1]
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:849) ~[hbase-client-1.0.1.jar:1.0.1]
at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1173) ~[hbase-client-1.0.1.jar:1.0.1]
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:216) ~[hbase-client-1.0.1.jar:1.0.1]
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:300) ~[hbase-client-1.0.1.jar:1.0.1]
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.execService(ClientProtos.java:31913) ~[hbase-protocol-1.0.1.jar:1.0.1]
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1605) ~[hbase-client-1.0.1.jar:1.0.1]
at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:92) ~[hbase-client-1.0.1.jar:1.0.1]
at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:89) ~[hbase-client-1.0.1.jar:1.0.1]
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126) ~[hbase-client-1.0.1.jar:1.0.1]
... 10 common frames omitted
Caused by: java.lang.RuntimeException: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$1.run(RpcClientImpl.java:672) ~[hbase-client-1.0.1.jar:1.0.1]
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_79]
at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_79]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) ~[hadoop-common-2.6.0.jar:na]
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.handleSaslConnectionFailure(RpcClientImpl.java:630) ~[hbase-client-1.0.1.jar:1.0.1]
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:738) ~[hbase-client-1.0.1.jar:1.0.1]
... 20 common frames omitted
Caused by: javax.security.sasl.SaslException: GSS initiate failed
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) ~[na:1.7.0_79]
at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179) ~[hbase-client-1.0.1.jar:1.0.1]
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:604) ~[hbase-client-1.0.1.jar:1.0.1]
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:153) ~[hbase-client-1.0.1.jar:1.0.1]
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:730) ~[hbase-client-1.0.1.jar:1.0.1]
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:727) ~[hbase-client-1.0.1.jar:1.0.1]
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_79]
at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_79]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) ~[hadoop-common-2.6.0.jar:na]
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:727) ~[hbase-client-1.0.1.jar:1.0.1]
... 20 common frames omitted
Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[na:1.7.0_79]
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) ~[na:1.7.0_79]
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[na:1.7.0_79]
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223) ~[na:1.7.0_79]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[na:1.7.0_79]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[na:1.7.0_79]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193) ~[na:1.7.0_79]
... 29 common frames omitted

Answer

The credentials may have expired. Do klist at the client and regionserver to see valid tickets when:

  • you can connect
  • you cannot connect
Comments