copy command in postgresql to append data
Assuming a Unix system
with PostgreQSL 9.3 or newer
Use the PROGRAM clause with cat
as the program.
execute 'copy (select * from tableName) to PROGRAM ''cat >>/tmp/result.txt'''
This would append the results to the end of /tmp/result.txt
instead of overwriting the file.
with older versions of PostgreSQL
A named pipe can be used by a separate process to concatenate results. Example:
$ mkfifo /tmp/pgfifo
$ while true; do cat /tmp/pgfifo >>/tmp/result.txt; done
This last command will block. Let it run until all the results are accumulated, then it can be terminated with ^C or kill.
In SQL, the fifo has to be fed with:
execute 'copy (select * from tableName) to ''/tmp/pgfifo'''
I think you cannot (easily) do that (at least on your PostgreSQL version, see Daniel's answer). What you definitely can, however, is changing your approach, and do a UNION ALL
from all those tables in the query part of your COPY
. This means there is no looping, but you have to construct your query from the collected table names.
The result would look like
COPY (SELECT * FROM table1
UNION ALL
SELECT * FROM table2
...)
TO tmp/result.txt;
Notes:
- there is no such thing as a
psql
function.psql
is a client to PostgreSQL. It has built in commands (basically everything starting with\
), however. - it is not so clear what you mean by 'archive logs'. If it is the write-ahead log (WAL), then with the above approach you don't have to worry about it. Otherwise, you can use unlogged tables.
- if the tables are big, there is a chance your system will write (a lot of) temp files. This might make the execution not so fast.