Avijit Avijit - 1 year ago 82
Scala Question

Error: not found: value lit/when - spark scala

I am using scala, spark, IntelliJ and maven.

I have used below code :

val joinCondition = when($"exp.fnal_expr_dt" >= $"exp.nonfnal_expr_dt",

val score = exprDF.as("exp").join(scoreDF.as("score"),joinCondition,"inner")


val score= list.withColumn("scr", lit(0))

But when try to build using maven, getting below errors -

error: not found: value when


error: not found: value lit

I have used
import sqlContext.implicits.StringToColumn
and it is working fine. No error occurred at the time of maven build.But for
what I need to import or is there any other way resolve the issue.

Answer Source

Let's consider the following context :

val sqlContext: SQLContext = new SQLContext(sc)
val list: DataFrame = ???

To use when and lit, you'll need to import the proper functions :

import org.apache.spark.sql.functions.{col, lit, when}

list.select(when(col("column_name"), 1))

import org.apache.spark.sql.functions.lit

val score = list.withColumn("scr", lit(0))
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download