Sebastian Rockefeller Sebastian Rockefeller - 2 months ago 6x
Linux Question

Out of memory in a thread pool in C, Linux

I need to create infinite loop and with a thread pool create for example 200 threads to do the job from infinite loop.

I'm using this thread pool -

In the same time I'm monitoring the server resources (with htop) and see that memory is increasing on 3 megabytes every second until the kernel kills the application.

The code:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <pthread.h>
#include "thpool.h"

#define MAX_IPv4 256

/* Args for thread start function */
typedef struct {
int octet1;
int octet2;
int octet3;
int octet4;
} args_struct;

/* Thread task */
void task1(void *args) {

args_struct *actual_args = args;

printf("%d.%d.%d.%d\n", actual_args->octet1, actual_args->octet2, actual_args->octet3, actual_args->octet4);

/* Do some job */

/* Free the args */

/* Main function */
int main( void ) {

int i=0, j=0, n=0, m=0;

/* Making threadpool n threads */
threadpool thpool = thpool_init(200);

/* Infinite loop start from the certain ip*/
while (1) {
for (i=0; i < MAX_IPv4; ++i) {
for (j=0; j < MAX_IPv4; ++j) {
for (n=0; n < MAX_IPv4; ++n) {
for (m=0; m < MAX_IPv4; ++m) {

/* Heap memory for the args different for the every thread */
args_struct *args = malloc(sizeof *args);
args->octet1 = i;
args->octet2 = j;
args->octet3 = n;
args->octet4 = m;

/* Create thread */
thpool_add_work(thpool, (void*)task1, (void*)args);

/* Start from */

/* Wait until the all threads are done */

/* Destroy the threadpool */

return 0;

How to solve this issue?


Looking at issues for your library ( especially this one about memory consumption ).

There is a recommendation to check the job queue length threadpool.jobqueue.len;

Maybe your code could check after adding your job to the queue

Unfortunately the threadpool is an opaque pointer and you could not access the value directly.

I would recommend adding a function for the threadpool in thpool.c :

int thpool_jobqueue_length(thpool_* thpool_p) {
    return thpool->jobqueue->len;

And don't forget the declaration in thpool.h

int thpool_jobqueue_length(threadpool);

Then modify your code

const int SOME_ARBITRARY_VALUE = 400
thpool_add_work(thpool, (void*)task1, (void*)args);
while( ( thpool_jobqueue_length(thpool) > SOME_ARBITRARY_VALUE ) ) {