Golang: How do I determine the number of lines in a file efficiently?
Here's a faster line counter using bytes.Count
to find the newline characters.
It's faster because it takes away all the extra logic and buffering required to return whole lines, and takes advantage of some assembly optimized functions offered by the bytes package to search characters in a byte slice.
Larger buffers also help here, especially with larger files. On my system, with the file I used for testing, a 32k buffer was fastest.
func lineCounter(r io.Reader) (int, error) {
buf := make([]byte, 32*1024)
count := 0
lineSep := []byte{'\n'}
for {
c, err := r.Read(buf)
count += bytes.Count(buf[:c], lineSep)
switch {
case err == io.EOF:
return count, nil
case err != nil:
return count, err
}
}
}
and the benchmark output:
BenchmarkBuffioScan 500 6408963 ns/op 4208 B/op 2 allocs/op
BenchmarkBytesCount 500 4323397 ns/op 8200 B/op 1 allocs/op
BenchmarkBytes32k 500 3650818 ns/op 65545 B/op 1 allocs/op
The most efficient way I found is using IndexByte of the byte packet, it is at least four times faster than using bytes.Count
and depending on the size of the buffer it uses much less memory.
func LineCounter(r io.Reader) (int, error) {
var count int
const lineBreak = '\n'
buf := make([]byte, bufio.MaxScanTokenSize)
for {
bufferSize, err := r.Read(buf)
if err != nil && err != io.EOF {
return 0, err
}
var buffPosition int
for {
i := bytes.IndexByte(buf[buffPosition:], lineBreak)
if i == -1 || bufferSize == buffPosition {
break
}
buffPosition += i + 1
count++
}
if err == io.EOF {
break
}
}
return count, nil
}
Benchmark
BenchmarkIndexByteWithBuffer 2000000 653 ns/op 1024 B/op 1 allocs/op
BenchmarkBytes32k 500000 3189 ns/op 32768 B/op 1 allocs/op