Maximilian Kohl Maximilian Kohl - 4 months ago 20
Bash Question

Bash - how to check if file has been uploaded to hdfs?

I created a folder in hdfs. Now, I tried to upload a file from my local filesystem to the tmp folder, like so:

hdfs dfs -put opt/ibm/labfiles/README.md /user/root/input/tmp


But I get the message:

put: '/user/root/input/tmp': File exists


What I don't understand: When I run the test if the file is there, I don't see the file listed, but the folder has the size of the file (4811 bites). Yet again, tmp might not even be a folder:
enter image description here


  • How could I see if tmp is a folder?

  • How could I see if the README.md file IS in tmp, or if not, how could I put it there?

  • When starting SparkShell from here, I cannot access the file (Input path does not exist). Why?


Answer

According to that tmp is a file. If it was a directory it would show drw_r__r__ notice the d, yours doesnt have that, so its a file.

You should have run:

hdfs dfs -put opt/ibm/labfiles/README.md /user/root/input/tmp/README.md

Comments