view raw
sarthak sarthak - 6 months ago 35
Scala Question

Unable to read file from HDFS

I am trying to read files from HDFS. I am using the following code:

val sam = sc.wholeTextFiles("hdfs://localhost:9000"+inputFolder,4)

I am getting the following error:

java.lang.IllegalArgumentException: Wrong FS: hdfs://localhost, expected: hdfs://localhost:9000

I had referenced this question for adding the URI in my file path:

Cannot Read a file from HDFS using Spark

But I am still not able to read the file due t the above error. How can I resolve this?


Can you check adding winutils.exe in your system and setting a environment variable for the same . Spark needs winutils.exe to do hdfs operations.