harshita g harshita g - 3 years ago 202
Scala Question

Spark scala split string and convert to dataframe with two columns

test = "a1-b1,a2-b2"

I want this string to be converted to a dataframe as

with columns A and B holding respective a1,a2 and b1,b2

Answer Source

You can convert the string into a RDD which is then converted into a DataFrame:

val s = "a1-b1,a2-b2"

val df = sc.parallelize(
    s.split(",").map(_.split("-")).map{ case Array(a, b) => (a, b) }
  ).toDF("A", "B")

|  A|  B|
| a1| b1|
| a2| b2|
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download