Dinky Dinky - 1 year ago 63
Java Question

Transforming RDD to List of Map

I need to transform each RDD to a NavigableMap and store in

List<NavigableMap<byte[], List<Map<String, String>>>>
in an anonymous function. I am doing work on a
JavaPairDStream<ImmutableBytesWritable, Put> puts

What I have so far:

puts.foreachRDD(r ->
List<NavigableMap<byte[], List<Cell>>> l = r.map(t ->
return null;

This throws a NotSerializableException at
is not serializable.

So I need to somehow transform
to a
Map<String, String
in the anonymous function and return
List<NavigableMap<byte[], List<Map<String, String>>>>
for me to use

Not having much Java 8 experience, I am pretty much stuck at this point. Any help will be appreciated.

Answer Source

This is how I ended up going about it. I needed primitives before collecting. I was trying to collect the return value of the .getFamilyCellMap() which is a map and it was not serializable.

Then I turned it back into a map before the assertions.

Here is the Java code:

puts.foreachRDD(r -> {
        List<String> l = r.flatMap(t -> {
                    Collection<List<Cell>> collection = t._2().getFamilyCellMap().values();
            return collection.stream()

        //Mapping for testing
        ObjectMapper objectMapper = new ObjectMapper();
        Map<String, Object> map1 = objectMapper.readValue(l.get(1), new TypeReference<Map<String, Object>>(){});
        Map<String, Object> map2 = objectMapper.readValue(l.get(2), new TypeReference<Map<String, Object>>(){});

        Assert.assertEquals(map1.get("attribute1").toString(), expected1);
        Assert.assertEquals(map2.get("attribute2").toString(), expected2);
    return null;

Hope this helps someone in need.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download