Erik Barajas Erik Barajas - 3 days ago 3
Scala Question

Null Fields Omitted with Split Function - Spark Scala

I'm new interacting with the world of Spark and Scala.

I have a file with the following data:

1.1|AGDT970HH|02|2011|N|||||
110000000000|0.00|0.00|0.00|1485976093.32|
110100000000|0.00|0.00|0.00|248871036.86|
110200000000|0.00|0.00|0.00|229610905.15|
110201000000|0.00|0.00|0.00|256660759.36|
110202000000|0.00|0.00|0.00|234105514.15|
110203000000|0.00|0.00|0.00|-261155368.36|
110300000000|0.00|0.00|0.00|280990.61|


Always the first row of the file will have a different layout in comparison to the other rows.

For that reason I want to give it a different treatment, so I try to get the values of the first row (including the null ones):

val fileToRead = sc.textFile("FileName")
val firstRow = fileToRead.first.split("\\|")


But when my firstRow field is created, it doesn't contain the null values:
enter image description here

It only saves the ones that contains a value.

Does anyone knows how to keep those null values?

Answer

You can get it as val firstRow = fileToRead.first.split("\\|", -1)