Sebastian Rockefeller Sebastian Rockefeller - 4 months ago 18
Linux Question

Out of memory in a thread pool in C, Linux

I need to create infinite loop and with a thread pool create for example 200 threads to do the job from infinite loop.

I'm using this thread pool - https://github.com/Pithikos/C-Thread-Pool

In the same time I'm monitoring the server resources (with htop) and see that memory is increasing on 3 megabytes every second until the kernel kills the application.

The code:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <pthread.h>
#include "thpool.h"


#define MAX_IPv4 256


/* Args for thread start function */
typedef struct {
int octet1;
int octet2;
int octet3;
int octet4;
} args_struct;

/* Thread task */
void task1(void *args) {

args_struct *actual_args = args;

printf("%d.%d.%d.%d\n", actual_args->octet1, actual_args->octet2, actual_args->octet3, actual_args->octet4);

/* Do some job */
sleep(1);

/* Free the args */
free(args);
}


/* Main function */
int main( void ) {

int i=0, j=0, n=0, m=0;

/* Making threadpool n threads */
threadpool thpool = thpool_init(200);

/* Infinite loop start from the certain ip*/
while (1) {
for (i=0; i < MAX_IPv4; ++i) {
for (j=0; j < MAX_IPv4; ++j) {
for (n=0; n < MAX_IPv4; ++n) {
for (m=0; m < MAX_IPv4; ++m) {

/* Heap memory for the args different for the every thread */
args_struct *args = malloc(sizeof *args);
args->octet1 = i;
args->octet2 = j;
args->octet3 = n;
args->octet4 = m;

/* Create thread */
thpool_add_work(thpool, (void*)task1, (void*)args);
}
}
}
}

/* Start from 0.0.0.0 */
i=0;
j=0;
n=0;
m=0;
}

/* Wait until the all threads are done */
thpool_wait(thpool);

/* Destroy the threadpool */
thpool_destroy(thpool);

return 0;
}


How to solve this issue?

Answer

Looking at issues for your library ( especially this one about memory consumption ).

There is a recommendation to check the job queue length threadpool.jobqueue.len;

Maybe your code could check after adding your job to the queue

Unfortunately the threadpool is an opaque pointer and you could not access the value directly.

I would recommend adding a function for the threadpool in thpool.c :

int thpool_jobqueue_length(thpool_* thpool_p) {
    return thpool->jobqueue->len;
}

And don't forget the declaration in thpool.h

int thpool_jobqueue_length(threadpool);

Then modify your code

const int SOME_ARBITRARY_VALUE = 400
...
thpool_add_work(thpool, (void*)task1, (void*)args);
while( ( thpool_jobqueue_length(thpool) > SOME_ARBITRARY_VALUE ) ) {
    sleep(1);
}
...