user3859408 user3859408 - 9 months ago 53
Java Question

Permission denied for hive scratch directory while trying to establish connection using Hive streaming api

I am writing a sample program using HCatalogue Streaming api

I have a running hadoop, Hiveserver and a Hivemetastore server.

I write a java program to connect to hive metastore.The source code is given below

public class HCatalogueStreamingclient {
public static void main(String[] args) {
System.setProperty("hadoop.home.dir", "E:\\midhun\\hadoop\\hive\\winutils");
String dbName = "hive_streaming";
String tblName = "alerts";
ArrayList<String> partitionVals = new ArrayList<String>(2);
HiveEndPoint hiveEP = new HiveEndPoint("thrift://", dbName, tblName, partitionVals);
HiveConf conf = new HiveConf();
conf.set("hive.exec.scratchdir", "/tmp/hivetmp");

try {
StreamingConnection connection = hiveEP.newConnection(true,conf);
} catch (ConnectionError e) {
} catch (InvalidPartition e) {
} catch (InvalidTable e) {
} catch (PartitionCreationFailed e) {
} catch (ImpersonationFailed e) {
} catch (InterruptedException e) {

While running the program I am getting following exception

Exception in thread "main" java.lang.RuntimeException: The root scratch dir: /tmp/hivetmp on HDFS should be writable. Current permissions are: rw-rw-rw-
at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(
at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(
at org.apache.hadoop.hive.ql.session.SessionState.start(
at org.apache.hadoop.hive.ql.session.SessionState.start(
at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.createPartitionIfNotExists(
at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(
at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(
at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnectionImpl(
at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(
at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(
at com.mj.HCatalogueStreamingclient.main(

Any one have an Idea on how to grand write permission to the hdfs folder /tmp/hivetmp

Answer Source

The problem is identified.

  • We need to grand all permission for hadoop directory /tmp/hivetmp.The command for granding permission is $HADOOP_HOME/bin/hadoop fs -chmod 777 -R /tmp/hivetmp
  • I was running the program in Windows 7 64 bit OS.We need to download winutils appropriate to the operating system and version.

After doing above given points I was able to establish connection using Hive HCatalogue Streaming api