Python truncate lines as they are read

You can't. It is just not possible with actual text file implementations on current filesystems.

Text files are sequential, because the lines in a text file can be of any length. Deleting a particular line would mean rewriting the entire file from that point on.

Suppose you have a file with the following 3 lines;

'line1\nline2reallybig\nline3\nlast line'

To delete the second line you'd have to move the third and fourth lines' positions in the disk. The only way would be to store the third and fourth lines somewhere, truncate the file on the second line, and rewrite the missing lines.

If you know the size of every line in the text file, you can truncate the file in any position using .truncate(line_size * line_number) but even then you'd have to rewrite everything after the line.


Remove all lines after you've done with them:

with open('myfile.txt', 'r+') as file:
    for line in file:
        processLine(line)
    file.truncate(0)

Remove each line independently:

lines = open('myfile.txt').readlines()

for line in lines[::-1]: # process lines in reverse order
    processLine(line)
    del lines[-1]  # remove the [last] line

open('myfile.txt', 'w').writelines(lines)

You can leave only those lines that cause exceptions:

import fileinput, sys
    
for line in fileinput.input(['myfile.txt'], inplace=1):
    try: processLine(line)
    except Exception:
         sys.stdout.write(line) # it prints to 'myfile.txt'

In general, as other people already said it is a bad idea what you are trying to do.

Tags:

Python

File Io