RX14 RX14 -4 years ago 58
Linux Question

Increase maximum virtual memory size above 256gb

I'm running a program which allocates 8mb stacks using mmap. While testing to see how many stacks I could allocate (aiming for 100,000), I see virtual memory size rise quickly as expected, and reserved size stay small (less than 1gb). The program then segfaults with

Cannot allocate new fiber stack: Cannot allocate memory (Errno)
. Using
gdb
to rescue the segfault and then looking at htop, I have discovered this happens at around 256GB of virtual memory.

I've tried using
prlimit --as=unlimited --rss=unlimited --memlock=unlimited --data=unlimited
when running the program, but it doesn't seem to make a difference.

Is there a way to increase this limit? Is it advisable to increase this limit? Is there a better way for crystal to allocate stacks?

Answer Source

Maybe you're hitting the maximum of /proc/sys/vm/max_map_count. This setting sets a maximum on the number of mmaps your process can have. The default value is 65536. So it's likely not the size of memory you want to malloc, but the number of malloc calls that causes the error Cannot allocate memory.

You can try to increase the maximum with:

sysctl -w vm.max_map_count=131070

See also NPTL caps maximum threads at 65528?

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download