Rani Rani - 11 days ago 5
Scala Question

expression of type Seq[unit] does not conform to expected type Seq[DataFrame] in scala

In my function, I am returning a finalDF, a sequence of data frames. In the loop shown below, map returns Seq[DataFrame] and it is being stored in finalDF to be able to return to the caller, but in some cases where there is further processing, I would like to store the filtered dataframe for each iteration and pass it to next loop. How do I do it? If I try to assign it to some temp val, it throws and error that expression of type Seq[unit] does not conform to expected type Seq[DataFrame]

var finalDF: Seq[DataFrame] =null

for (i <- 0 until stop){
finalDF=strataCount(i).map(x=> {
df.filter(df(cols(i)) === x)

//how to get the above data frame to pass on to the next computation?
}
)

}


Regards

Answer

Maybe this is helpful:

val finalDF: Seq[DataFrame] = (0 to stop).flatMap(i => strataCount(i).map(x => df.filter(df(cols(i)) === x))).toSeq

flatMap to flatten the Seq(Seq).