File not found error when launching a subprocess containing piped commands
You have to add shell=True
to execute a shell command. check_output
is trying to find an executable called: date | grep -o -w '"+tz+"'' | wc -w
and he cannot find it. (no idea why you removed the essential information from the error message).
See the difference between:
>>> subprocess.check_output('date | grep 1')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.4/subprocess.py", line 603, in check_output
with Popen(*popenargs, stdout=PIPE, **kwargs) as process:
File "/usr/lib/python3.4/subprocess.py", line 848, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.4/subprocess.py", line 1446, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'date | grep 1'
And:
>>> subprocess.check_output('date | grep 1', shell=True)
b'gio 19 giu 2014, 14.15.35, CEST\n'
Read the documentation about the Frequently Used Arguments for more information about the shell
argument and how it changes the interpretation of the other arguments.
Note that you should try to avoid using shell=True
since spawning a shell can be a security hazard (even if you do not execute untrusted input attacks like Shellshock can still be performed!).
The documentation for the subprocess module has a little section about replacing the shell pipeline.
You can do so by spawning the two processes in python and use subprocess.PIPE
:
date_proc = subprocess.Popen(['date'], stdout=subprocess.PIPE)
grep_proc = subprocess.check_output(['grep', '1'], stdin=date_proc.stdout, stdout=subprocess.PIPE)
date_proc.stdout.close()
output = grep_proc.communicate()[0]
You can write some simple wrapper function to easily define pipelines:
import subprocess
from shlex import split
from collections import namedtuple
from functools import reduce
proc_output = namedtuple('proc_output', 'stdout stderr')
def pipeline(starter_command, *commands):
if not commands:
try:
starter_command, *commands = starter_command.split('|')
except AttributeError:
pass
starter_command = _parse(starter_command)
starter = subprocess.Popen(starter_command, stdout=subprocess.PIPE)
last_proc = reduce(_create_pipe, map(_parse, commands), starter)
return proc_output(*last_proc.communicate())
def _create_pipe(previous, command):
proc = subprocess.Popen(command, stdin=previous.stdout, stdout=subprocess.PIPE)
previous.stdout.close()
return proc
def _parse(cmd):
try:
return split(cmd)
except Exception:
return cmd
With this in place you can write pipeline('date | grep 1')
or pipeline('date', 'grep 1')
or pipeline(['date'], ['grep', '1'])
The most common cause of FileNotFound
with subprocess, in my experience, is the use of spaces in your command. If you have just a single command (not a pipeline, and no redirection, wildcards, etc), use a list instead.
# Wrong, even with a valid command string
subprocess.run(['grep -o -w "+tz+"'])
# Fixed; notice also
subprocess.run(["grep", "-o", "-w", '"+tz+"'])
This change results in no more FileNotFound
errors, and is a nice solution if you got here searching for that exception with a simpler command.
If you need a pipeline or other shell features, the simple fix is to add shell=True
:
subprocess.run(
'''date | grep -o -w '"+tz+"'' | wc -w''',
shell=True)
However, if you are using python 3.5 or greater, try using this approach:
import subprocess
a = subprocess.run(["date"], stdout=subprocess.PIPE)
print(a.stdout.decode('utf-8'))
b = subprocess.run(["grep", "-o", "-w", '"+tz+"'],
input=a.stdout, stdout=subprocess.PIPE)
print(b.stdout.decode('utf-8'))
c = subprocess.run(["wc", "-w"],
input=b.stdout, stdout=subprocess.PIPE)
print(c.stdout.decode('utf-8'))
You should see how one command's output becomes another's input just like using a shell pipe, but you can easily debug each step of the process in python. Using subprocess.run is recommended for python > 3.5, but not available in prior versions.