Feng Chen Feng Chen - 1 year ago 90
Scala Question

How to display the results brough from column functions using spark/scala like what show() does to dataframe

I just start how to use dataframe and column in spark/scala. I know if I want to show sth on the screen, I can just do like df.show() for that. But how can I do this to a column. For example,

scala> val dfcol = df.apply("sgan")
dfcol: org.apache.spark.sql.Column = sgan

this can find a column called "sgan" from the dataframe df then give it to dfcol, so dfcol is a column. Then, if I do

scala> abs(dfcol)
res29: org.apache.spark.sql.Column = abs(sgan)

I just got the result shown on the screen like above. How can I show the result of this function on the screen like df.show() does? Or, in other words, how can I know the results of the functions like abs, min and so forth?

Answer Source

You should always use a dataframe, Column objects are not meant to be investigated this way. You can use select to create a dataframe with the column you're interested in, and then use show():

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download