Maximilian Kohl Maximilian Kohl - 9 months ago 109
Bash Question

Bash - how to check if file has been uploaded to hdfs?

I created a folder in hdfs. Now, I tried to upload a file from my local filesystem to the tmp folder, like so:

hdfs dfs -put opt/ibm/labfiles/ /user/root/input/tmp

But I get the message:

put: '/user/root/input/tmp': File exists

What I don't understand: When I run the test if the file is there, I don't see the file listed, but the folder has the size of the file (4811 bites). Yet again, tmp might not even be a folder:
enter image description here

  • How could I see if tmp is a folder?

  • How could I see if the file IS in tmp, or if not, how could I put it there?

  • When starting SparkShell from here, I cannot access the file (Input path does not exist). Why?


According to that tmp is a file. If it was a directory it would show drw_r__r__ notice the d, yours doesnt have that, so its a file.

You should have run:

hdfs dfs -put opt/ibm/labfiles/ /user/root/input/tmp/