stackster stackster - 3 months ago 17
Bash Question

store hdfs command output to a shell variable

I am trying to the hdfs command output to a shell variable but am getting error. My script is as below:

#!/bin/bash
default=$1
date="";
HDFS_PATH='/data/max_emp_value' (This is a directory)
if $(hadoop fs -test -e $HDFS_PATH/*); then
date = "`hadoop fs -cat HDFS_PATH/*)`"
else
date=$default
fi
echo date = ${date}


Thanks,
stacky

Answer

Does this Work - copy and paste it as is.

#!/bin/bash
default=$1
date=""
HDFS_PATH='/data/max_emp_value' #(This is a directory)
if $(hadoop fs -test -e $HDFS_PATH/*)
then
    date="`hadoop fs -cat $HDFS_PATH/*)`"
else
    date=$default
fi
echo "date is ${date}"

Why do you want to check all files in a directory? $HDFS_PATH/* the * refers to all files in a directory. Possibly you are trying to check if the files have arrived - in that case you should have one touch file usually marked as _success after the transfer is complete. Otherwise you risk of consuming the data in transit.