Steven Li Steven Li - 1 month ago 8
Scala Question

How can I extact multi-line record from a text file with Spark

I have a text file. The records are all multiple-line records which are delimited by "\n\n\n\n\n". The text file looks like below:

name: Steven
gender: male
title: mr.
company: ABC

cell 647-777-****
home 905-000-****
work 289-***-1111





name: Al
gender: male
title: mr.
company: DEF

home 905-111-****
cell 289-991-****


What I have done is the code below:

val contact_raw = sc.wholeTextFiles("/user/data/contact.txt").flatMap(x => x._2.split("\n\n\n\n\n"))
val contact = contact_raw.map(contacts => {
val per_person = contacts.split("\n\n")
(per_person(0), per_person(1))
}).map(
contact_info => {
val personal_info = contact_info._1.split("\n")
var name = ""
var company = ""
var gender = ""
var title = ""
for (x <- personal_info) {
if(x.startsWith("name:")){
name = x.split("name:")(1).trim
} else if(x.startsWith("gender:")){
gender = x.split("gender:")(1).trim
} else if(x.startsWith("title:")){
title = x.split("title:")(1)
} else if(x.startsWith("company:")){
company = x.split("company:")(1)
}
}
val phone_info = contact_info._2.split("\n").map(
pair => {
val phone_pair = pair.split("\\s")
(phone_pair(0), phone_pair(1))
}
)
(name, gender, title, company, phone_info)
}
).toDF("name", "gender", "title", "company", "phone_info")


The output is:

scala> contact.show
+------+------+-----+-------+--------------------+
| name|gender|title|company| phone_info|
+------+------+-----+-------+--------------------+
|Steven| male| mr.| ABC|[[cell,647-777-**...|
| Al| male| mr.| DEF|[[home,905-111-**...|
+------+------+-----+-------+--------------------+


And the schema is:

scala> contact.printSchema
root
|-- name: string (nullable = true)
|-- gender: string (nullable = true)
|-- title: string (nullable = true)
|-- company: string (nullable = true)
|-- phone_info: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- _1: string (nullable = true)
| | |-- _2: string (nullable = true)


The expected output is:

+------+------+-----+-------+-------------+------------+
| name|gender|title|company| phone_type|number |
+------+------+-----+-------+-------------+------------+
|Steven| male| mr.| ABC| cell|647-777-****|
|Steven| male| mr.| ABC| home|905-000-****|
|Steven| male| mr.| ABC| work|289-***-1111|
| Al| male| mr.| DEF| home|905-111-****|
| Al| male| mr.| DEF| cell|289-991-****|
+------+------+-----+-------+-------------+------------+


Who can tell me how to modify my code to get the output I need? Thanks.

Answer

The following would work:

val contact = contact_raw.map(contacts => {
      val per_person = contacts.split("\n\n")
      (per_person(0), per_person(1))
    }).flatMap(
      contact_info => {
        val personal_info = contact_info._1.split("\n")
        var name = ""
        var company = ""
        var gender = ""
        var title = ""
        for (x <- personal_info) {
          if (x.startsWith("name:")) {
            name = x.split("name:")(1).trim
          } else if (x.startsWith("gender:")) {
            gender = x.split("gender:")(1).trim
          } else if (x.startsWith("title:")) {
            title = x.split("title:")(1)
          } else if (x.startsWith("company:")) {
            company = x.split("company:")(1)
          }
        }
        contact_info._2.split("\n").map(
          pair => {
            val phone_pair = pair.split("\\s")
            (name, gender, title, company, phone_pair(0), phone_pair(1))
          }
        )
      }
    ).toDF("name", "gender", "title", "company", "phone_info")
Comments