sarthak sarthak - 26 days ago 7
Scala Question

Unable to read file from HDFS

I am trying to read files from HDFS. I am using the following code:

val sam = sc.wholeTextFiles("hdfs://localhost:9000"+inputFolder,4)


I am getting the following error:

java.lang.IllegalArgumentException: Wrong FS: hdfs://localhost, expected: hdfs://localhost:9000


I had referenced this question for adding the URI in my file path:

Cannot Read a file from HDFS using Spark

But I am still not able to read the file due t the above error. How can I resolve this?

Answer

Can you check adding winutils.exe in your system and setting a environment variable for the same . Spark needs winutils.exe to do hdfs operations.

Comments