lte__ lte__ - 1 year ago 146
Java Question

Apache Spark - Adding two columns

Is there a way to add two columns so that the first one is a date and the second one contains the number of days that need to be added? I'm trying

date_add(res.col("date"), res.col("days"));

But this doesn't work, since
requires a
and an
, while I have two columns.

Thank you!

Answer Source

This is a limitation of DataFrame DSL not engine itself. It is not optimal but you can replace function call with expr:

import org.apache.spark.sql.functions.{expr, col}

val df = Seq(("2012-04-05", 6))
  .toDF("date", "days")
  .withColumn("date", col("date").cast("date"))"date_add(date, days)"))
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download