sarthak sarthak - 1 year ago 112
Scala Question

Unable to read file from HDFS

I am trying to read files from HDFS. I am using the following code:

val sam = sc.wholeTextFiles("hdfs://localhost:9000"+inputFolder,4)

I am getting the following error:

java.lang.IllegalArgumentException: Wrong FS: hdfs://localhost, expected: hdfs://localhost:9000

I had referenced this question for adding the URI in my file path:

Cannot Read a file from HDFS using Spark

But I am still not able to read the file due t the above error. How can I resolve this?

Answer Source

Can you check adding winutils.exe in your system and setting a environment variable for the same . Spark needs winutils.exe to do hdfs operations.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download