Add lines to the beginning and end of the huge file
sed -i
uses tempfiles as an implementation detail, which is what you are experiencing; however, prepending data to the beginning of a data stream without overwriting the existing contents requires rewriting the file, there's no way to get around that, even when avoiding sed -i
.
If rewriting the file is not an option, you might consider manipulating it when it is read, for example:
{ echo some prepended text ; cat file ; } | command
Also, sed is for editing streams -- a file is not a stream. Use a program that is meant for this purpose, like ed or ex. The -i
option to sed is not only not portable, it will also break any symlinks to your file, since it essentially deletes it and recreates it, which is pointless.
You can do this in a single command with ed
like so:
ed -s file << 'EOF'
0a
prepend these lines
to the beginning
.
$a
append these lines
to the end
.
w
EOF
Note that depending on your implementation of ed, it may use a paging file, requiring you to have at least that much space available.
Note that if you want to avoid allocating a whole copy of the file on disk, you could do:
sed '
1i\
begin
$a\
end' < file 1<> file
That uses the fact that when its stdin/stdout is a file, sed
reads and writes by block. So here, it's OK for it to override the file it is reading as long as the first line you're adding is smaller than sed
's block size (should be something like 4k or 8k).
Note though that if for some reason sed
fails (killed, machine crash...), you'll end up with the file half processed which will mean some data the size of the first line missing somewhere in the middle.
Also note that unless your sed
is the GNU sed
, that won't work for binary data (but since you're using -i
, you are using GNU sed).
Here are some choices (all of which will create a new copy of the file so make sure you have enough space for that):
simple echo/cat
echo "first" > new_file; cat $File >> new_file; \ echo "last" >> new_file;
awk/gawk etc
gawk 'BEGIN{print "first\n"}{print}END{print "last\n"}' $File > NewFile
awk
and its ilk read files line by line. TheBEGIN{}
block is executed before the first line and theEND{}
block after the last line. So, the command above meansprint "first" at the beginning, then print every line in the file and print "last" at the end
.Perl
perl -ne 'BEGIN{print "first\n"} print;END{print "last\n"}' $File > NewFile
This is essentially the same thing as the gawk above just written in Perl.