massy8448 massy8448 - 5 months ago 26
Linux Question

SQLite3 - memory usage gradually increases until all available memory are consumed

I am experiencing a memory usage issue while running a simple single-threaded C++ program for long time (say over night). The program uses SQLite3 API to open a data base and writes some data in it within a loop.I am running the program on two different machines: a desktop Ubuntu Linux and an ARM based embedded device running a custom built Linux.
In both cases I am getting the same results: Memory is consumed gradually and not released while application is running. I am checking memory usage using a simple bash script running in background:

while true;
do free -m;
sleep 2;
done


It should be noted that I am also monitoring memory usage using SQLite provided API:

sqlite3_memory_used()


The API reports a fairly stable amount of used memory but the "free -m" report is different and increases gradually.

The SQLite source code is compiled with the following flags:

SQLITE_DEFAULT_TEMP_CACHE_SIZE=3
SQLITE_DEFAULT_WAL_AUTOCHECKPOINT=2
SQLITE_MAX_MMAP_SIZE=2048
SQLITE_ENABLE_MEMSYS5
SQLITE_ENABLE_MEMORY_MANAGEMENT
SQLITE_DEFAULT_CACHE_SIZE=3
SQLITE_DEFAULT_AUTOVACUUM=1
SQLITE_DEFAULT_PAGE_SIZE=512


Please note that at this stage I am not concerned about speed but my main concern is memory usage so I set parameters in such a way that minimum data is cached in memory and get them pushed to the disk as soon as possible.

I also use "PRAGMA shrink_memory" in each iteration.

To minimise dynamic memory allocation I have also provided static arrays for the following memory types:

SQLITE_CONFIG_HEAP
SQLITE_CONFIG_SCRATCH
SQLITE_CONFIG_PAGECACHE


And the code snippet which writes to database looks like this:

char SQL_Statement[100]={0};
char *ErrMsg = 0;
for (int i = 0; i < 1000000; i++)
{
sprintf(SQL_Statement, "INSERT INTO PointValue (TimeStamp, BlockId, PointId, Value) VALUES (%f, %d, %d, %d);",TimeStamp_ ,BlockId_, PointId_, Value_ );
check = sqlite3_exec(MyDB, SQL_Statement, callback, (void*)data, &ErrMsg);
sqlite3_free(ErrMsg);
}

Answer

Thanks everyone for your answers and comments. It is confirmed that The memory is consumed by linux page cache which is fine because Linux hopefully takes care of it and releases unnecessary pages when another application need more memory. just using a simple command:

cat /proc/meminfo