massy8448 massy8448 - 1 year ago 183
Linux Question

SQLite3 - memory usage gradually increases until all available memory are consumed

I am experiencing a memory usage issue while running a simple single-threaded C++ program for long time (say over night). The program uses SQLite3 API to open a data base and writes some data in it within a loop.I am running the program on two different machines: a desktop Ubuntu Linux and an ARM based embedded device running a custom built Linux.
In both cases I am getting the same results: Memory is consumed gradually and not released while application is running. I am checking memory usage using a simple bash script running in background:

while true;
do free -m;
sleep 2;

It should be noted that I am also monitoring memory usage using SQLite provided API:


The API reports a fairly stable amount of used memory but the "free -m" report is different and increases gradually.

The SQLite source code is compiled with the following flags:


Please note that at this stage I am not concerned about speed but my main concern is memory usage so I set parameters in such a way that minimum data is cached in memory and get them pushed to the disk as soon as possible.

I also use "PRAGMA shrink_memory" in each iteration.

To minimise dynamic memory allocation I have also provided static arrays for the following memory types:


And the code snippet which writes to database looks like this:

char SQL_Statement[100]={0};
char *ErrMsg = 0;
for (int i = 0; i < 1000000; i++)
sprintf(SQL_Statement, "INSERT INTO PointValue (TimeStamp, BlockId, PointId, Value) VALUES (%f, %d, %d, %d);",TimeStamp_ ,BlockId_, PointId_, Value_ );
check = sqlite3_exec(MyDB, SQL_Statement, callback, (void*)data, &ErrMsg);

Answer Source

Thanks everyone for your answers and comments. It is confirmed that The memory is consumed by linux page cache which is fine because Linux hopefully takes care of it and releases unnecessary pages when another application need more memory. just using a simple command:

cat /proc/meminfo
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download