Is there something wrong with my script or is Bash much slower than Python?

Shell loops are slow and bash's are the slowest. Shells aren't meant to do heavy work in loops. Shells are meant to launch a few external, optimized processes on batches of data.

Anyway, I was curious how shell loops compare so I made a little benchmark:

#!/bin/bash

export IT=$((10**6))

echo POSIX:
for sh in dash bash ksh zsh; do
    TIMEFORMAT="%RR %UU %SS $sh"
    time $sh -c 'i=0; while [ "$IT" -gt "$i" ]; do i=$((i+1)); done'
done


echo C-LIKE:
for sh in bash ksh zsh; do
    TIMEFORMAT="%RR %UU %SS $sh"
    time $sh -c 'for ((i=0;i<IT;i++)); do :; done'
done

G=$((10**9))
TIMEFORMAT="%RR %UU %SS 1000*C"
echo 'int main(){ int i,sum; for(i=0;i<IT;i++) sum+=i; printf("%d\n", sum); return 0; }' |
   gcc -include stdio.h -O3 -x c -DIT=$G - 
time ./a.out

( Details:

  • CPU: Intel(R) Core(TM) i5 CPU M 430 @ 2.27GHz
  • ksh: version sh (AT&T Research) 93u+ 2012-08-01
  • bash: GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu)
  • zsh: zsh 5.2 (x86_64-unknown-linux-gnu)
  • dash: 0.5.7-4ubuntu1

)

The (abbreviated) results (time per iteration) are:

POSIX:
5.8 µs  dash
8.5 µs ksh
14.6 µs zsh
22.6 µs bash

C-LIKE:
2.7 µs ksh
5.8 µs zsh
11.7 µs bash

C:
0.4 ns C

From the results:

If you want a slightly faster shell loop, then if you have the [[ syntax and you want a fast shell loop, you're in an advanced shell and you have the C-like for loop too. Use the C like for loop, then. They can be about 2 times as fast as while [-loops in the same shell.

  • ksh has the fastest for ( loop at about 2.7µs per iteration
  • dash has the fastest while [ loop at about 5.8µs per iteration

C for loops can be 3-4 decimal orders of magnitude faster. (I heard the Torvalds love C).

The optimized C for loop is 56500 times faster than bash's while [ loop (the slowest shell loop) and 6750 times faster than ksh's for ( loop (the fastest shell loop).

Again, the slowness of shells shouldn't matter much though, because the typical pattern with shells is to offload to a few processes of external, optimized programs.

With this pattern, shells often make it much easier to write scripts with performance superior to python scripts (last time I checked, creating process pipelines in python was rather clumsy).

Another thing to consider is startup time.

time python3 -c ' '

takes 30 to 40 ms on my PC whereas shells take around 3ms. If you launch a lot of scripts, this quickly adds up and you can do very very much in the extra 27-37 ms that python takes just to start. Small scripts can be finished several times over in that time frame.

(NodeJs is probably the worst scripting runtime in this department as it takes about 100ms just to start (even though once it has started, you'd be hard pressed to find a better performer among scripting languages)).


This is a known bug in bash; see the man page and search for "BUGS":

BUGS
       It's too big and too slow.

;)

For an excellent primer on the conceptual differences between shell scripting and other programming languages, I highly recommend reading:

  • Why is using a shell loop to process text considered bad practice?

The most pertinent excerpts:

Shells are a higher level language. One may say it's not even a language. They're before all command line interpreters. The job is done by those commands you run and the shell is only meant to orchestrate them.

...

IOW, in shells, especially to process text, you invoke as few utilities as possible and have them cooperate to the task, not run thousands of tools in sequence waiting for each one to start, run, clean up before running the next one.

...

As said earlier, running one command has a cost. A huge cost if that command is not builtin, but even if they are builtin, the cost is big.

And shells have not been designed to run like that, they have no pretension to being performant programming languages. They are not, they're just command line interpreters. So, little optimisation has been done on this front.

Don't use big loops in shell scripting.


I did a bit of testing, and on my system ran the following--none made the order of magnitude speedup that would be needed to be competitive, but you can make it faster:

Test 1: 18.233s

#!/bin/bash
i=0
while [[ $i -le 4000000 ]]
do
    let i++
done

test2: 20.45s

#!/bin/bash
i=0
while [[ $i -le 4000000 ]]
do 
    i=$(($i+1))
done

test3: 17.64s

#!/bin/bash
i=0
while [[ $i -le 4000000 ]]; do let i++; done

test4: 26.69s

#!/bin/bash
i=0
while [ $i -le 4000000 ]; do let i++; done

test5: 12.79s

#!/bin/bash
export LC_ALL=C

for ((i=0; i != 4000000; i++)) { 
:
}

The important part in this last one is the export LC_ALL=C. I've found that many bash operations end up significantly faster if this is used, in particular any regex function. It also shows an undocumented for syntax to use the {} and the : as a no-op.

Tags:

Bash

Python3

Recent Posts