Count the number of lines in a file without reading entire file into memory?

It doesn't matter what language you're using, you're going to have to read the whole file if the lines are of variable length. That's because the newlines could be anywhere and theres no way to know without reading the file (assuming it isn't cached, which generally speaking it isn't).

If you want to indicate progress, you have two realistic options. You can extrapolate progress based on assumed line length:

assumed lines in file = size of file / assumed line size
progress = lines processed / assumed lines in file * 100%

since you know the size of the file. Alternatively you can measure progress as:

progress = bytes processed / size of file * 100%

This should be sufficient.

If you are in a Unix environment, you can just let wc -l do the work.

It will not load the whole file into memory; since it is optimized for streaming file and count word/line the performance is good enough rather then streaming the file yourself in Ruby.

SSCCE:

filename = 'a_file/somewhere.txt'
line_count = `wc -l "#{filename}"`.strip.split(' ')[0].to_i
p line_count

Or if you want a collection of files passed on the command line:

wc_output = `wc -l "#{ARGV.join('" "')}"`
line_count = wc_output.match(/^ *([0-9]+) +total$/).captures[0].to_i
p line_count

Reading the file a line at a time:

count = File.foreach(filename).inject(0) {|c, line| c+1}

or the Perl-ish

File.foreach(filename) {}
count = $.

or

count = 0
File.open(filename) {|f| count = f.read.count("\n")}

Will be slower than

count = %x{wc -l #{filename}}.split.first.to_i