Johan Johan - 1 year ago 74
iOS Question

Threading caused app to lag on iOS but not on Android

I'm building a face detection app in Unity3D for both iOS and Android. The app is supposed to detect 10 faces per second. However, each face detection takes about 50 ms, so I've decided to let the face detection run in the background on another thread in order to prevent small lag spikes (the apps need to update visual stuff every frame, 50 ms stops makes it laggy).

The problem is that threading caused the app to lag on iOS. Without threading it used to detect 10 faces per second, now 2 with threading.. It works fine on Android.

What's weird is that the app runs fine on iOS for the first 5 seconds. It doesn't matter if it detects faces or not, after 5 seconds comes heavy lag. EDIT: If I pause the app and then resume it the lag disappears for another 5 seconds like in the start. What is this!?

Down below is a coroutine where the face detection is handled, it runs 10 times per second. If the face detection thread doesn't finish in 100 ms it prevents another thread from starting.

IEnumerator FaceDetection()
while (true)
if (faceDetectionThreadDone)
rgbaMat = webCamTextureToMatHelper.GetMat();
if (rgbaMat != null)
OpenCVForUnityUtils.SetImage(faceLandmarkDetector, rgbaMat);
faceDetectionThreadDone = false;

new Thread(() =>
Thread.CurrentThread.IsBackground = true;

// heavy computing stuff (50 ms)
List<UnityEngine.Rect> detectResult = faceLandmarkDetector.Detect();
if (detectResult.Count > 0)
points = faceLandmarkDetector.DetectLandmark(detectResult[0]);

faceDetectionThreadDone = true;

yield return new WaitForSeconds(0.1f);

Is there some serious problem with how I create new threads? I've read somewhere that they get cleaned up by the garbage collector once they're done executing. Might that cause the lag? The build settings are set on high performance etc.

EDIT: Ok, I'm pretty sure now that the problem isn't related to memory management. I've run the internal profiler in Xcode and watched how much of the heap was used. The perfomance didn't get affected when the garbage collector kicked in.

Devices that I tested the app on: iPhone 6, Sony Z3, Samsung S4

Answer Source

Creating and starting Thread is expensive. Each time

Thread(() =>

is called, about 1 MB memory is allocated. Add other memory allocation that happens inside there such as List(detectResult) creation.

Also, yield return new WaitForSeconds(0.1f); allocates memory but I don't think this is the problem. You can prevent this by declaring WaitForSeconds before the while loop then using it.

For example,

//Create instance of WaitForSeconds once, before the while loop
WaitForSeconds waitForSeconds = new WaitForSeconds(0.1f);

while (true){
//your Thread code
yield return waitForSeconds; //No more memory allocation 

Don't really that that is the main problem but I put that let you know you could do that. For the Thread problem I mentioned above, you should create a while loop that processes your data then performs the output in the main Thread. Basically a Thread function that will loop forever instead of creating a new Thread each time.

There is a plugin called Unity Threading Helper that allows you to call Unity API functions from another Thread. You may want to use it. It has paid and free version. The free version does not includes examples in it. That's the only difference but you can download it and examples of it on the link I posted.

Below is a general example of how to use that API. Read the comments in the code. Your job is to put the codes in your question into the appropriate location which is easy to do.

bool keepProcessing;
UnityThreading.ActionThread myThread;
void Start()
    keepProcessing = true;
    //Create Thread once
    myThread = UnityThreadHelper.CreateThread(() =>
        //Will process Data forever!
        while (keepProcessing)
            //Do heavy computing stuff 

            //Computing Done! Do something with the output data
             UnityThreadHelper.Dispatcher.Dispatch(() =>
                   //You can safely call Unity API functions inside this codeblock

            //Wait so that Unity won't Freeze

void stopProcessing()
    keepProcessing = false;

Finally, I suggest you re-design your DetectLandmark function to take in array as parameter instead of List. Allocate enough array memory once then re-use it. This will completely remove memory allocation from your app.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download