FredMaggiowski FredMaggiowski - 4 months ago 52
Android Question

Android huge JSONObject toString goes OutOfMemoryError

I need to read a lot of data from a SQLite database and create a properly formatted JSON.

I'm currently achieving this by having an object called

RequestPayload
 which contains some
ArrayList
s in which I put the data I read from SQLite.

The
RequestPayload
class has a
parseJson()
method which returns a
JSONObject
on which I eventually call the
toString()
method to obtain my JSON String that finally got written on a file.

Now, when I've got "small" quantities of data in SQLite everything goes fine. When I've got a lot of data this is what happens:

06-28 09:55:34.121 10857-6799/it.example.sampler E/AndroidRuntime: FATAL EXCEPTION: Thread-195240
Process: it.example.sampler, PID: 10857
Theme: themes:{com.cyanogenmod.trebuchet=overlay:system, com.android.settings=overlay:system, default=overlay:system, iconPack:system, fontPkg:system, com.android.systemui=overlay:system, com.android.systemui.navbar=overlay:system}
java.lang.OutOfMemoryError: Failed to allocate a 52962812 byte allocation with 16764752 free bytes and 41MB until OOM
at java.lang.StringFactory.newStringFromChars(Native Method)
at java.lang.AbstractStringBuilder.toString(AbstractStringBuilder.java:629)
at java.lang.StringBuilder.toString(StringBuilder.java:663)
at org.json.JSONStringer.toString(JSONStringer.java:430)
at org.json.JSONObject.toString(JSONObject.java:690)
at it.example.sampler.controllers.network.RequestBodyEncoder.serialise(RequestBodyEncoder.java:69)
at it.example.sampler.controllers.network.RequestBodyEncoder.createPacket(RequestBodyEncoder.java:50)
at it.example.sampler.services.FileStoreRunnable.run(FileStoreRunnable.java:57)
at java.lang.Thread.run(Thread.java:818)

06-28 09:55:34.509 10857-10857/it.example.sampler E/WindowManager: android.view.WindowLeaked: Activity it.example.sampler.ui.StartSamplingActivity has leaked window com.android.internal.policy.PhoneWindow$DecorView{74710d V.E...... R......D 0,0-1026,494} that was originally added here
at android.view.ViewRootImpl.<init>(ViewRootImpl.java:372)
at android.view.WindowManagerGlobal.addView(WindowManagerGlobal.java:299)
at android.view.WindowManagerImpl.addView(WindowManagerImpl.java:86)
at android.app.Dialog.show(Dialog.java:319)
at it.example.sampler.ui.StartSamplingActivity.executeStop(StartSamplingActivity.java:305)
at it.example.sampler.ui.StartSamplingActivity.onClickedButton(StartSamplingActivity.java:256)
at it.example.sampler.ui.StartSamplingActivity$3.onClick(StartSamplingActivity.java:190)
at android.view.View.performClick(View.java:5204)
at android.view.View$PerformClick.run(View.java:21158)
at android.os.Handler.handleCallback(Handler.java:739)
at android.os.Handler.dispatchMessage(Handler.java:95)
at android.os.Looper.loop(Looper.java:148)
at android.app.ActivityThread.main(ActivityThread.java:5461)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:726)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:616)


Now the code:

the error line is from this method:

private String serialise() throws JSONException {
// Serialise
String sampleString = mPayload.parseJSON().toString(); // THIS LINE

Logger.get().d(LOG_TAG, "Serialised payload: -> " + sampleString);
return sampleString;
}


and if I got the LogCat correctly the error is during the execution of the
toString()
method and it appears that the JSON object is too big.

How can I handle this situation?

UPDATE:
To answer to @YashJain comment asking for the size of the JSONObject.

After sampling a sampling lasted 2 hours I had a JSON containing:


  • 12 JSONArray containing 155.432
    float
    s

  • 2 JSONArray containing 8.393
    float
    s

  • 1 JSONArray containing 8.393 custom object containing 5
    float
    s each (and a String)

  • 1 2D array containing 155.432
    int
    s



In terms of bytes since a Float is made by 4 bytes (I hope I've done the calculus correctly) I've got circa:

(12 * 155432 * 4) + (2 * 8393 * 4) + (1 * 8393 * 5) + (1 * 155432 * 4) ~= 8.191.573 bytes


UPDATE:
I've been asked about the used compression algorithm. I use
ZLIB
via the
Deflater
and
DeflaterOutputStream
 Java classes.

Thus the flow is:


  • Sample data -> Store them in
    SQLite

  • Read from
    SQLite
    -> Build
    RequestPayload
    object in memory

  • Convert
    RequestPayload
    object to
    JSONObject
    (with custom a custom parser).

  • Convert the
    JSONObject
    to String (using
    JSONObject
    toString()
    method)

  • Compress the String bytes (using
    getBytes()
    ) -> encode it in Base64

  • Send final Base64 string to server



EDIT:
To respond to the "possibile duplicate flag" I haven't seen that question during my researches by the way that didn't help me since it makes use of
largeHeap
directive (which I don't use). Also there's no accepted answer and the only answer doesn't actually provide a practical solution.

Answer

Sorry for not giving any more feedbacks I've been busy with other works and this got last in my priority queue.

Anyhow what actually worked for me is using Google GSON library. My serialisation has become:

private String serialise() {
    // Serialise
    return mPayload.toStringFromGSON();
}

And method toStringFromGSON() in class Payload is:

public String toStringFromGSON() {
    GsonBuilder gsonBuilder = new GsonBuilder();
    Gson gson = gsonBuilder.create();

    return gson.toJson(this);
}

Anyhow I noticed that the memory used by the application increases dramatically during this phase hence I think this solution only postponed the problem that will eventually occur with heavier loads..