DilTeam DilTeam - 3 months ago 85
Scala Question

Spark sql (1.4) : Get current timestamp

I know there's a 'current_timestamp' function in Spark 1.5 & above, but we're still on 1.4. How do I add a column to a DataFrame that is set to 'current timestamp'?

df.withColumn("date_inserted", ????)

Answer

Simple timestamp literal should do the trick:

df.withColumn(
  "date_inserted",
  lit(java.sql.Timestamp.from(java.time.Instant.now)))