lte__ lte__ - 4 months ago 35
Java Question

Spark SQL - JAVA syntax of CASE-THEN?

I noticed I can use CASE-THEN with Spark if I use an

SQLContext
and the
.sql()
function. Is there a way to use this in a JAVA syntax, directly on dataframes, too? How?
Now, I write:

SparkConf sparkConf = new SparkConf();
JavaSparkContext ctx = new JavaSparkContext(sparkConf);
SQLContext sqlContext = new SQLContext(ctx);
DataFrame df = //some imported data
df.registerTempTable("df");
sqlContext.sql("SELECT *use case-then in here* FROM df");


I'm looking for something like

df.select(case("this").then("that"));

Answer

Just import org.apache.spark.sql.functions then use when(Column col, Object obj).

import org.apache.spark.sql.functions;

df.select(functions.when(df.col("colName").equalTo("this"), "that").otherwise("something"));
Comments