I would like to display the entire Apache Spark SQL DataFrame with the Scala API. I can use the
It is generally not advisable to display an entire DataFrame to stdout, because that means you need to pull the entire DataFrame (all of its values) to the driver (unless
DataFrame is already local, which you can check with
Unless you know ahead of time that the size of your dataset is sufficiently small so that driver JVM process has enough memory available to accommodate all values, it is not safe to do this. That's why DataFrame API's
show() by default shows you only the first 20 rows.
You could use the
df.collect which returns
Array[T] and then iterate over each line and print it:
but you lose all formatting implemented in
df.showString(numRows: Int) (that
show() internally uses).
So no, I guess there is no better way.