Viren Viren - 4 months ago 23
Ruby Question

Memory Management in Ruby

I puzzled by some behaviour of ruby and how it manages memory.

I understand the Ruby GC (major or minor) behaviour i.e if the any objects count goes above there threshold value or limit (i.e heap_available_slots,old_objects_limit, remembered_shady_object_limit, malloc_limit). Ruby run/trigger a GC(major or minor).

And after GC if it can't find the enough memory Ruby allocate (basically malloc I assuming) more memory for the running program.

Also, It's a known fact that by does release back memory to the OS immediately.

Now ..

What I fail to understand how come Ruby releases memory (back to the OS) without triggering any GC.

Example

require 'rbtrace'

index = 1
array = []
while(index < 20000000) do
array << index
index += 1
end


sleep 10
print "-"
array=nil
sleep


Here is my example. If run the above code on ruby 2.2.2p95.
htop
display the RSS count of the process (test.rb PID 11483) reaching to 161MB.

enter image description here

GC.stat
(captured via
rbtrace
gem) look like (pay close attention to attention to
GC count
)

rbtrace -p 11843 -e '[Time.now,Process.pid,GC.stat]'

[Time.now,Process.pid,GC.stat]
=> [2016-07-27 13:50:28 +0530, 11843,
{
"count": 7,
"heap_allocated_pages": 74,
"heap_sorted_length": 75,
"heap_allocatable_pages": 0,
"heap_available_slots": 30162,
"heap_live_slots": 11479,
"heap_free_slots": 18594,
"heap_final_slots": 89,
"heap_marked_slots": 120,
"heap_swept_slots": 18847,
"heap_eden_pages": 74,
"heap_tomb_pages": 0,
"total_allocated_pages": 74,
"total_freed_pages": 0,
"total_allocated_objects": 66182,
"total_freed_objects": 54614,
"malloc_increase_bytes": 8368,
"malloc_increase_bytes_limit": 33554432,
"minor_gc_count": 4,
"major_gc_count": 3,
"remembered_wb_unprotected_objects": 0,
"remembered_wb_unprotected_objects_limit": 278,
"old_objects": 14,
"old_objects_limit": 10766,
"oldmalloc_increase_bytes": 198674592,
"oldmalloc_increase_bytes_limit": 20132659
}]
*** detached from process 11843

GC count => 7


Approximately 25 minutes later. Memory has drop down to 6MB but GC count is still 7.

enter image description here

[Time.now,Process.pid,GC.stat]
=> [2016-07-27 14:16:02 +0530, 11843,
{
"count": 7,
"heap_allocated_pages": 74,
"heap_sorted_length": 75,
"heap_allocatable_pages": 0,
"heap_available_slots": 30162,
"heap_live_slots": 11581,
"heap_free_slots": 18581,
"heap_final_slots": 0,
"heap_marked_slots": 120,
"heap_swept_slots": 18936,
"heap_eden_pages": 74,
"heap_tomb_pages": 0,
"total_allocated_pages": 74,
"total_freed_pages": 0,
"total_allocated_objects": 66284,
"total_freed_objects": 54703,
"malloc_increase_bytes": 3248,
"malloc_increase_bytes_limit": 33554432,
"minor_gc_count": 4,
"major_gc_count": 3,
"remembered_wb_unprotected_objects": 0,
"remembered_wb_unprotected_objects_limit": 278,
"old_objects": 14,
"old_objects_limit": 10766,
"oldmalloc_increase_bytes": 198663520,
"oldmalloc_increase_bytes_limit": 20132659
}]


Question: I was under the impression that Ruby Release memory whenever GC is triggered. But clearly that not the case over here.

Anybody can provide a detail on how (as in who triggered the memory releases surely its not GC.) the memory is released back to OS.

OS: OS X version 10.11.12

Answer

You are correct, it's not GC that changed the physical memory requirements, it's the OS kernel.

You need to look at the VIRT column, not the RES column. As you can see VIRT stays exactly the same.

RES is physical (resident) memory, VIRT is virtual (allocated, but currently unused) memory.

When the process sleeps it's not using its memory or doing anything, so the OS memory manager decides to swap out part of the physical memory and move it into virtual space.

Why keep an idle process hogging physical memory for no reason? So the OS is smart and swaps out as much unused physical memory as possible, that's why you see a reduction in RES.

I suspect you would see the same effect even without array = nil, by just sleeping long enough. Once you stop sleeping and access something in the array then RES will jump back up again.

You can read some more discussion through these:
What is RSS and VSZ in Linux memory management
http://www.darkcoding.net/software/resident-and-virtual-memory-on-linux-a-short-example/
What's the difference between "virtual memory" and "swap space"?
http://www.tldp.org/LDP/tlk/mm/memory.html