How to Read last lines from a big file with Go every 10 secs

I think a combination of File.Seek(0, 2) and File.Read() should work.

The Seek call gets you to the end of file. You can Seek to a position a bit before the EOF to get last few lines. Then you Read till the EOF and just sleep in your goroutine for 10 seconds; next Read has a chance to get you more data.

You can snatch the idea (and the scan-back logic for initially showing few last lines) from GNU tail's source.

Some people will come to this page looking for efficiently reading the last line of a log file (like the tail command line tool).

Here is my version to read the last line of a big file. It use two previous suggestions (using Seek and file Stat).

It read the file backward, byte by byte (no need to set a buffer size) until finding the beginning of a line or the beginning of the file.

func getLastLineWithSeek(filepath string) string {
    fileHandle, err := os.Open(filepath)

    if err != nil {
        panic("Cannot open file")
        os.Exit(1)
    }
    defer fileHandle.Close()

    line := ""
    var cursor int64 = 0
    stat, _ := fileHandle.Stat()
    filesize := stat.Size()
    for { 
        cursor -= 1
        fileHandle.Seek(cursor, io.SeekEnd)

        char := make([]byte, 1)
        fileHandle.Read(char)

        if cursor != -1 && (char[0] == 10 || char[0] == 13) { // stop if we find a line
            break
        }

        line = fmt.Sprintf("%s%s", string(char), line) // there is more efficient way

        if cursor == -filesize { // stop if we are at the begining
            break
        }
    }

    return line
}

You can use file.Seek() or file.ReadAt() to almost the end and then Reading forward. You can only estimate where to start seeking unless you can know that 2 lines = x bytes.

You can get the File length by using the os.Stat(name)

Here is an example based on ReadAt, Stat, and your sample log file:

package main

import (
    "fmt"
    "os"
    "time"
)

const MYFILE = "logfile.log"

func main() {
    c := time.Tick(10 * time.Second)
    for _ = range c {
        readFile(MYFILE)
    }
}

func readFile(fname string) {
    file, err := os.Open(fname)
    if err != nil {
        panic(err)
    }
    defer file.Close()

    buf := make([]byte, 62)
    stat, err := os.Stat(fname)
    start := stat.Size() - 62
    _, err = file.ReadAt(buf, start)
    if err == nil {
        fmt.Printf("%s\n", buf)
    }

}

Well, this is only a raw idea and maybe not the best way, you should check and improve it, but seems to work...

I hope that experienced Go users could contribute too..

With Stat you can get the size of the file and from it get the offset for use with ReadAt

func readLastLine(fname string) {
    file, err := os.Open(fname)
    if err != nil {
        panic(err)
    }
    defer file.Close()

    fi, err := file.Stat()
    if err != nil {
        fmt.Println(err)
    }

    buf := make([]byte, 32)
    n, err := file.ReadAt(buf, fi.Size()-int64(len(buf)))
    if err != nil {
        fmt.Println(err)
    }
    buf = buf[:n]
    fmt.Printf("%s", buf)

}