dsp_099 dsp_099 - 3 months ago 18
Ruby Question

How to read a large text file line-by-line and append this stream to a file line-by-line in Ruby?

Let's say I want to combine several massive files into one and then

uniq!
the one (THAT alone might take a hot second)

It's my understanding that
File.readlines()
loads ALL the lines into memory. Is there a way to read it line by line, sort of like how node.js
pipe()
system works?

Answer

One of the great things about Ruby is that you can do file IO in a block:

File.open("test.txt", "r").each_line do |row|
  puts row
end               # file closed here

so things get cleaned up automatically. Maybe it doesn't matter on a little script but it's always nice to know you can get it for free.