Spark scala split string and convert to dataframe with two columns

Question

test = "a1-b1,a2-b2"

I want this string to be converted to a dataframe as

with columns A and B holding respective a1,a2 and b1,b2

Recommended for you: Get network issues from WhatsUp Gold. Not end users.

Answer 1

You can convert the string into a RDD which is then converted into a DataFrame:

val s = "a1-b1,a2-b2"

val df = sc.parallelize(
    s.split(",").map(_.split("-")).map{ case Array(a, b) => (a, b) }
  ).toDF("A", "B")

df.show
+---+---+
|  A|  B|
+---+---+
| a1| b1|
| a2| b2|
+---+---+

Spark scala split string and convert to dataframe with two columns

Sign up