Prem Anand Prem Anand - 6 months ago 9
Linux Question

File system in hadoop

Recently I have started to learn hadoop multi cluster

How the file system work's in hadoop.

For eg. If i have one test.txt file in fully of farmer details.
Whether I need to upload the file in master HDFS server or in slave server.

Answer

To know how the Hadoop filesystem works, please refer to book Hadoop: The Definitive Guide

For time being, to load a file into HDFS, you only need to run the -put or -copyFromLocal command from edgenode of the cluster(meaning from where you can run hadoop command) and rest will be taken care by Hadoop framework. Your command may look like

hadoop fs set

$ hadoop fs -copyFromLocal /home/user1/farmer_details.txt /user/user1

or

$ hadoop fs -put /home/user1/farmer_details.txt /user/user1

You can also try hdfs dfs set

$ hdfs dfs -copyFromLocal /home/user1/farmer_details.txt /user/user1

or

$ hdfs dfs -put /home/user1/farmer_details.txt /user/user1

where /home/user1/farmer_details.txt is the source location in your local file system and /user/user1 is the destination location in hadoop filesystem.

To verify the uploaded file, you can run command

$ hadoop fs -ls /user/user1

You should get /user/user1/farmer_details.txt file.

Comments