Avijit Avijit - 2 months ago 17
Scala Question

error: not found: value lit....Spark Scala

I am Using Scala, Spark, IntelliJ, Maven.

I have used below code -

val joinCondition = when($"exp.fnal_expr_dt" >= $"exp.nonfnal_expr_dt",
$"exp.manr_cd"===$"score.MANR_CD")

val score = exprDF.as("exp").join(scoreDF.as("score"),joinCondition,"inner")


and

val score= list.withColumn("scr", lit(0))


But when try to build using maven, getting below errors -


error: not found: value when


and


error: not found: value lit


For
$
and
===
I have used
import sqlContext.implicits.StringToColumn
and it is working fine. No error occured at the time of maven build.But for
lit(0)
and
when
what I need to import or is there any other way resolve the issue.

Answer

Let's consider the following context :

val sqlContext: SQLContext = new SQLContext(sc)
val list: DataFrame = ???

To use when and lit, you'll need to import the proper functions :

import org.apache.spark.sql.functions.{col, when}

list.select(when(col("column_name"), 1))

import org.apache.spark.sql.functions.lit

val score = list.withColumn("scr", lit(0))
Comments