Dinky Dinky - 2 months ago 7
Java Question

Transforming RDD to List of Map

I need to transform each RDD to a NavigableMap and store in

List<NavigableMap<byte[], List<Map<String, String>>>>
in an anonymous function. I am doing work on a
JavaPairDStream<ImmutableBytesWritable, Put> puts
.

What I have so far:

puts.foreachRDD(r ->
List<NavigableMap<byte[], List<Cell>>> l = r.map(t ->
t._2().getFamilyCellMap()).collect();
return null;
});


This throws a NotSerializableException at
.collect()
because
Cell
is not serializable.

So I need to somehow transform
Cell
to a
Map<String, String
in the anonymous function and return
List<NavigableMap<byte[], List<Map<String, String>>>>
for me to use
.collect()
.

Not having much Java 8 experience, I am pretty much stuck at this point. Any help will be appreciated.

Answer

This is how I ended up going about it. I needed primitives before collecting. I was trying to collect the return value of the .getFamilyCellMap() which is a map and it was not serializable.

Then I turned it back into a map before the assertions.

Here is the Java code:

puts.foreachRDD(r -> {
        List<String> l = r.flatMap(t -> {
                    Collection<List<Cell>> collection = t._2().getFamilyCellMap().values();
            return collection.stream()
                    .flatMap(Collection::stream)
                    .map(CellUtil::cloneValue))
                    .collect(Collectors.toList());
                }).collect();

        //Mapping for testing
        ObjectMapper objectMapper = new ObjectMapper();
        Map<String, Object> map1 = objectMapper.readValue(l.get(1), new TypeReference<Map<String, Object>>(){});
        Map<String, Object> map2 = objectMapper.readValue(l.get(2), new TypeReference<Map<String, Object>>(){});

        System.out.println(map1);
        Assert.assertEquals(map1.get("attribute1").toString(), expected1);
        //etc
        System.out.println(map2);
        Assert.assertEquals(map2.get("attribute2").toString(), expected2);
        //etc
    }
    return null;
});

Hope this helps someone in need.

Comments