cmannett85 cmannett85 - 2 months ago 26
C++ Question

boost::managed_mapped_file cannot allocate all of grown space

I'm trying to grow a memory mapped file, I successfully grow it, but I can't allocate all the extra space I've requested - I just get a

std::bad_alloc
instead.

Here is an example that shows the effect with g++ on Linux (I've seen the same on my 'real' code on MSVC too):

#include <memory>
#include <sstream>

#include <boost/interprocess/managed_mapped_file.hpp>
#include <boost/interprocess/allocators/allocator.hpp>
#include <boost/interprocess/containers/vector.hpp>
#include <boost/filesystem/operations.hpp>
#include <boost/uuid/uuid_io.hpp>
#include <boost/uuid/random_generator.hpp>

namespace
{
using MMapManager = boost::interprocess::basic_managed_mapped_file<
char,
boost::interprocess::rbtree_best_fit<boost::interprocess::null_mutex_family,
boost::interprocess::offset_ptr<void>,
16u>,
boost::interprocess::iset_index>;

using MMapAllocatorType = boost::interprocess::allocator<
std::size_t,
MMapManager::segment_manager>;

using MMapContainerType = boost::interprocess::vector<
std::size_t,
MMapAllocatorType>;

// I've measured this at 256 bytes for my example configuration, but it's not
// documented anywhere, so let's overcompensate
constexpr auto ManagedFileOverhead = 1024u;

boost::filesystem::path getTemporaryFilePath()
{
auto ss = std::stringstream{};
ss << "MMap_test_" << boost::uuids::random_generator{}();

return boost::filesystem::temp_directory_path() / ss.str();
}
}

int main()
{
// Create memory mapped file, initially for 100 items
auto capacity = 100u;
const auto size = (capacity * sizeof(std::size_t)) + ManagedFileOverhead;
const auto path = getTemporaryFilePath();

auto file = std::make_unique<MMapManager>(
boost::interprocess::create_only,
path.string().c_str(),
size);
auto data = file->construct<MMapContainerType>("data_")(file->get_segment_manager());

// Fill with stuff
data->reserve(capacity);
for (auto i = 0u; i < capacity; ++i) {
data->push_back(i);
}

// Let's grow to hold 162 items (100 * golden ratio)
capacity = 162u;
const auto newFileSize = (capacity * sizeof(std::size_t)) + ManagedFileOverhead;
const auto oldFileSize = boost::filesystem::file_size(path);
const auto extraBytes = newFileSize - oldFileSize;

// Unmap from the process, and grow
file.reset();
MMapManager::grow(path.string().c_str(), extraBytes);

// Reopen it to re-map it into this process
file = std::make_unique<MMapManager>(
boost::interprocess::open_only,
path.string().c_str());
data = file->find<MMapContainerType>("data_").first;

// Allocate it all
data->reserve(capacity); // Bang, you're dead

// Close down
file.reset();
boost::system::error_code ec;
boost::filesystem::remove(path, ec);

return EXIT_SUCCESS;
}


Setting the reserve (after growing) to 155 items works, just one more triggers a
std::bad_alloc
.

Why does this not work? Does growing incur extra management overhead within the mapped file causing me to run out of space earlier than expected?

Answer

You're simply assuming too much about the allocators.

Growing the mapped file will happen in place. Growing the vector does not. So, while you would need only extraBytes after increasing the reserved size, during the reserve you need enough space to hold both the old and the new allocations.

Prove it either by using:

MMapManager::grow(path.string().c_str(), oldFileSize + extraBytes);

Or by clearing the old container first:

{
    auto file = std::make_unique<MMapManager>(boost::interprocess::open_only, path.string().c_str());
    file->destroy<MMapContainerType>("data_");
    auto data = file->construct<MMapContainerType>("data_")(file->get_segment_manager());
}

MMapManager::grow(path.string().c_str(), extraBytes);
Comments