LearningSlowly LearningSlowly - 3 months ago 13
Scala Question

Map Method sql.row extraction - data type error in Spark

I have a

.sql.Row
type
RDD
.

I am using the
map
method to unpack from the
row
and create a new
RDD
.

This question builds directly from what I learnt in Scala RDD String manipulation.

In the original
RDD
,
vertices_raw
I have a field
Metrics
that I use map on.

val vertices = vertices_raw.rdd.map(row=> (row.getAs[String]("Metrics").map(_.stripPrefix("name").toLong))


Resulting in:

error: value stripPrefix is not a member of Char


The same goes for
.drop
method for removing
name
.

I have tried adding a
toString
in attempt to convert from
Char
to
String
with no change.

val vertices = vertices_raw.rdd.map(row=> (row.getAs[String]("Metrics").toString.map(_.stripPrefix("name").toLong))


Instead of
getAs[String]
I also tried
getString
.

val vertices = vertices_raw.rdd.map(row=> (row.getString("Metrics").map(_.stripPrefix("name").toLong))
error: type mismatch;
found : String("toid")
required: Int


My practical question is how do I convert from
Char
to
String
? before performing the
map
function?

My fundamental question is what exactly does
getAs[String]
mean if it doesn't literally mean get and in String format? I have a misunderstood this clearly.

Answer

You are performing a map on a string - that is going to map over each char of the string. If you just want to call stripPrefix then just do that without the map.

row.getAs[String]("Metrics").stripPrefix("name").toLong
Comments