What is the fastest way to read the new lines in the dynamically updatable file?

This might be OS-dependent, but I would assume that:

oldPos = StreamPosition[f];
Close[f];
f = OpenRead["f:\\imgs\\logs.txt"];
SetStreamPosition[f, p];
Read[f, ConstantArray[Number, 2]]

should be faster than Skip, since it just moves the low-level file pointer.

Wolfram Support Answer

The input stream is "buffered" by C, which is to say the file's data is read into memory when the stream is opened. Then all "read" operations work on the buffered data in memory, so if the file changes in the meantime, those changes are not reflected. This is a lot faster than constantly checking the file on disk.

You could remember the current stream position (using StreamPosition), Close the file, re-open the file again (using OpenRead), and then move back to the saved stream position (using SetStreamPosition). Closing and re-opening would get rid of the in-memory buffer.

This is, of course, not dynamic and unfortunately, there is not a documented or supported method of doing this.

However, OpenRead with the option AppendCheck->True may be able to fill the purpose. The option will reset the end of file marker in most cases allowing new information from the file to be read. This option is not documented or supported, so please consider this when using it at your discretion as we would not be able to provide further information on it.

Benchmarking

f = OpenRead["all_trades.txt", AppendCheck -> True];

AbsoluteTiming[
 SetStreamPosition[f, 0];
 Skip[f, ConstantArray[Number, 10], 163658];
 ]

{0.246617, Null}

StreamPosition[f]

6382661

AbsoluteTiming[
 SetStreamPosition[f, 0];
 SetStreamPosition[f, 6382661];
 ]

{0.00020363, Null}

Add a new line to the file and save.

Read[f, ConstantArray[Number, 10]]; // AbsoluteTiming

{0.0000380343, Null}

So OpenRead[..., AppendCheck -> True] is the fastest way.