Prem Anand Prem Anand - 7 months ago 25
Linux Question

File system in hadoop

Recently I have started to learn hadoop multi cluster

How the file system work's in hadoop.

For eg. If i have one test.txt file in fully of farmer details.
Whether I need to upload the file in master HDFS server or in slave server.


To know how the Hadoop filesystem works, please refer to book Hadoop: The Definitive Guide

For time being, to load a file into HDFS, you only need to run the -put or -copyFromLocal command from edgenode of the cluster(meaning from where you can run hadoop command) and rest will be taken care by Hadoop framework. Your command may look like

hadoop fs set

$ hadoop fs -copyFromLocal /home/user1/farmer_details.txt /user/user1


$ hadoop fs -put /home/user1/farmer_details.txt /user/user1

You can also try hdfs dfs set

$ hdfs dfs -copyFromLocal /home/user1/farmer_details.txt /user/user1


$ hdfs dfs -put /home/user1/farmer_details.txt /user/user1

where /home/user1/farmer_details.txt is the source location in your local file system and /user/user1 is the destination location in hadoop filesystem.

To verify the uploaded file, you can run command

$ hadoop fs -ls /user/user1

You should get /user/user1/farmer_details.txt file.