Vedom Vedom - 28 days ago 12
Scala Question

How to sort by column in descending order in Spark SQL?

I tried

df.orderBy("col1").show(10)
but it sorted in ascending order.
df.sort("col1").show(10)
also sorts in descending order. I looked on stackoverflow and the answers I found were all outdated or referred to RDDs. I'd like to use the native dataframe in spark.

Answer

It's in org.apache.spark.sql.DataFrame for sort method:

df.sort($"col1", $"col2".desc)

Note $ and .desc inside sort for the column to sort the results by.