Launching Multiple Queries With Bash Script
Due to https://mywiki.wooledge.org/BashPitfalls#Non-atomic_writes_with_xargs_-P (output from parallel jobs in xargs
risks being mixed), I would use GNU Parallel instead:
cat infile |
parallel -P0 -q curl {} -o /dev/null --silent --head --write-out "%{http_code} {}\n" > outfile
In this particular case it may be safe to use xargs
because the output is so short, so the problem with using xargs
is rather that if someone later changes the code to do something bigger, it will no longer be safe. Or if someone reads this question and thinks he can replace curl
with something else, then that may also not be safe.
This may be faster:
doit() {
while read LINE; do
curl -o /dev/null --silent --head --write-out "%{http_code} $LINE\n" "$LINE"
done
}
export -f doit
parallel -j0 --pipepart -a infile --block -10 doit > outfile