choroukka choroukka - 1 year ago 277
Scala Question

Cannot resolve reference StructField with such signature

i've copied a working example of and i've changed it a little, but the core is always the same, but i got always this error in the StructField point:

cannot resolve reference StructField with such signature

And also gives me this one, inside the signature:

Type mismatch, expected: Datatype, actual StringType

Here is the part of my code where i got problems:

import org.apache.avro.generic.GenericData.StringType
import org.apache.spark
import org.apache.spark.sql.types.StructField
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.sql.types._
object Test{

def main(args: Array[String]): Unit = {

val file = "/home/ubuntu/spark/MyFile"
val conf = new SparkConf().setAppName("Test")
val sc = new SparkContext(conf)
val read = sc.textFile(file)
val header = read.first().toString
//generate schema from first csv row
val fields = header.split(";").map(fieldName => StructField(fieldName.trim, StringType, true))
val schema = StructType(fields)

I cannot understand where i'm wrong.
I'm using Spark version 2.0.0


Answer Source

It looks like GenericData.StringType is an issue. Use an alias:

import org.apache.avro.generic.GenericData.{StringType => AvroStringType}

or remove this import (you don't use it).

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download