I have a Spark program (in Scala) and a
Thanks to marios and kostya, but there are few steps to writing a text file into HDFS from Spark.
// Hadoop Config is accessible from SparkContext val fs = FileSystem.get(sparkContext.hadoopConfiguration); // Output file can be created from file system. val output = fs.create(new Path(filename)); // But BufferedOutputStream must be used to output an actual text file. val os = BufferedOutputStream(output) os.write("Hello World".getBytes("UTF-8")) os.close()
FSDataOutputStream, which has been suggested, is a Java serialized object output stream, not a text output stream. The
writeUTF method appears to write plaint text, but it's actually a binary serialization format that includes extra bytes.