C. Güzelhan C. Güzelhan - 16 days ago 4
C++ Question

Memory usage increases over time when writing to a file

I have a small Qt based C++ program and this program just writes a vector to a file until I click the "STOP" button.

In the Qt designer, I just put a push button, change the text to "STOP" and name the object "button".

The problem is that when I check the memory usage of the program by the watch command in terminal I see that the free memory decreases over time and clicking on the button does not affect it. It only goes to its initial if I close the application.

test.pro :

#-------------------------------------------------
#
# Project created by QtCreator 2016-10-12T13:56:45
#
#-------------------------------------------------

QT += core gui

greaterThan(QT_MAJOR_VERSION, 4): QT += widgets

QMAKE_CXXFLAGS += -std=c++11
QMAKE_CXXFLAGS += -D_GLIBCXX_USE_CXX11_ABI=0
QMAKE_CXXFLAGS += -pthread

TARGET = Test
TEMPLATE = app


SOURCES += main.cpp\
mainwindow.cpp \
writer.cpp

HEADERS += mainwindow.h \
writer.h

FORMS += mainwindow.ui


mainwindow.h :

#ifndef MAINWINDOW_H
#define MAINWINDOW_H

#include <QMainWindow>

#include <writer.h>

namespace Ui {
class MainWindow;
}

class MainWindow : public QMainWindow
{
Q_OBJECT

public:
explicit MainWindow(QWidget *parent = 0);
~MainWindow();

private slots:
void on_stopButton_clicked();

void on_startButton_clicked();

private:
Ui::MainWindow *ui;
writer *w;
};

#endif // MAINWINDOW_H


writer.h :

#ifndef WRITER_H
#define WRITER_H

#include <iostream>
#include <fstream>
#include <iterator>
#include <string>
#include <locale>
#include <sstream>
#include <thread>
#include <atomic>
#include <vector>

#define COUNTER_FILENAME "data_counter.txt"
#define DATA_PREFIX "data_no"
#define DATA_EXTENSION ".csv"

class writer
{
public:
// For labeling
unsigned int label_id;
bool label;
// Start writing frame features in a file
void write();
// Stop writing
void stopWriting();
// Default demo constructor, sets up the frame thread which will process frame data in a seperate thread
explicit writer();
// Default demo destructor, flags that the frame thread should stop and waits for the frame thread to join
~writer();

protected:
std::unique_ptr< std::thread > frameThread;
std::atomic< bool > stopFrameThread;
std::vector< float > frameFeatures;
std::string data_filename;
};

#endif // WRITER_H


main.cpp :

#include "mainwindow.h"
#include <QApplication>

int main(int argc, char *argv[])
{
QApplication a(argc, argv);
MainWindow w;
w.show();

return a.exec();
}


mainwindow.cpp :

#include "mainwindow.h"
#include "ui_mainwindow.h"

MainWindow::MainWindow(QWidget *parent) :
QMainWindow(parent),
ui(new Ui::MainWindow),
w(new writer())
{
ui->setupUi(this);


}

MainWindow::~MainWindow()
{
delete w;
delete ui;
}

void MainWindow::on_stopButton_clicked()
{
w->stopWriting();
}

void MainWindow::on_startButton_clicked()
{
w->write();
}


writer.cpp :

#include "writer.h"

writer::writer() :
label_id( 0 ),
label( false )
{
stopFrameThread = false;
// Resize the vector to put all frame features
frameFeatures.resize( 1743, 0.0 );
}

writer::~writer()
{
stopWriting();
}

void writer::write()
{
// Setup the frame thread
stopFrameThread = false;

// Create a new thread to retrieve all features of each frame
frameThread = std::unique_ptr< std::thread >( new std::thread( [ this ](){
// Read from counter file to get counter value
std::string counter;
std::ifstream counter_in (COUNTER_FILENAME);
if( counter_in.is_open() ){
std::getline( counter_in, counter );
counter_in.close();
}else{
counter = "0";
}

// Convert the counter into an integer
int c;
std::stringstream( counter ) >> c;

// Compute data filename
data_filename = DATA_PREFIX + counter + "_id" + applicant_id + DATA_EXTENSION;

std::ofstream data_file( data_filename );

if( data_file.is_open() ){
// Start the main processing loop
while( ! stopFrameThread ){
for(int index = 0; index < 1743; index++){
frameFeatures[ index ] = index;
}

for(std::vector< float >::iterator it = frameFeatures.begin(); it != frameFeatures.end(); ++it){
data_file << " " + std::to_string( *it );
}
data_file << "\n";
}

// Close the data file
data_file.close();

// Write incremented counter value on counter file
std::ofstream counter_out (COUNTER_FILENAME);
if( counter_out.is_open() ){
counter_out << ++c;
counter_out.close();
}else{
std::cerr << "Unable to write on counter file\n";
}
}else{
std::cerr << "Unable to open the data file\n";
}
} ) );
}

void writer::stopWriting()
{
// Flag that the frame thread should stop
stopFrameThread = true;

// Wait for the frame thread to stop
if( frameThread ){
if( frameThread->joinable() ){
frameThread->join();
}
}
}


Thank you for any help.

Answer

It is been a while since I fixed the issue but I thought I should write it here in case someone encounters a similar problem.

There was not any bug, nor any memory leak in the code. The program works perfectly. The issue was that the process of creating new vectors was faster than than the one writing to the file.

I solved it by having two separate threads to speed up the process. The first thread would create vectors and store them in a buffer. The second thread would take a vector from that buffer and write it.