Béatrice Moissinac Béatrice Moissinac - 1 year ago 125
Javascript Question

Spark 2.0.0 - JSON malformed output

I am processing data with Spark and Scala, and saving it in



The output looks like this:


I use the data to create a visualization using
, using

d3.json("mydata.json", function(d){

My problem is that
to be formatted as follow:


Who is wrong? Spark or
? What can I do to alleviate this situation without having to manually add

Answer Source

I don't know Spark, but I can say that this is not a valid JSON, you have just a bunch of objects that are not wrapped in an array. So, for "who is wrong?", I'd say Spark.

But there is a (ugly) workaround. Use d3.text to load that thing (that bunch of objects):

d3.text("data.json", function(data){});

Then, your data will be a string. The next step is splitting the string by new lines:

data = data.match(/[^\r\n]+/g);

And, then, we transform this in an array of objects:

data = data.map(function(d){
    return JSON.parse(d)

All together:

d3.text("data.json", function(data){

    data = data.match(/[^\r\n]+/g);
    data = data.map(function(d){
        return JSON.parse(d)
    //now you can use 'data' here


Check the console in this plunker: https://plnkr.co/edit/ER1oXyWZL62dwxlgaenP?p=preview

And, now that you have an array of objects, you can pass it to your D3 code.

PS: This may not work if you have dates in the data.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download