What is the subprocess.Popen max length of the args parameter?
If you're passing shell=False, then Cmd.exe does not come into play.
On windows, subprocess will use the CreateProcess function from Win32 API to create the new process. The documentation for this function states that the second argument (which is build by subprocess.list2cmdline) has a max length of 32,768 characters, including the Unicode terminating null character. If lpApplicationName is NULL, the module name portion of lpCommandLine is limited to MAX_PATH characters.
Given your example, I suggest providing a value for executable (args[0]) and using args for the first parameter. If my reading of the CreateProcess documentation and of the subprocess module source code is correct, this should solve your problem.
[edit: removed the args[1:] bit after getting my hands on a windows machine and testing]
For Unix-like platforms, the kernel constant ARG_MAX
is defined by POSIX. It is required to be at least 4096 bytes, though on modern systems, it's probably a megabyte or more.
On many systems, getconf ARG_MAX
will reveal its value at the shell prompt.
The shell utility xargs
conveniently allows you to break up a long command line. For example, if
python myscript.py *
fails in a large directory because the list of files expands to a value whose length in bytes exceeds ARG_MAX
, you can work around it with something like
printf '%s\0' * |
xargs -0 python myscript.py
(The option -0
is a GNU extension, but really the only completely safe way to unambiguously pass a list of file names which could contain newlines, quoting characters, etc.) Maybe also explore
find . -maxdepth 1 -type f -exec python myscript.py {} +
Conversely, to pass a long list of arguments to subprocess.Popen()
and friends, something like
p = subprocess.Popen(['xargs', '-0', 'command'],
stdin=subprocess.PIPE, stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
out, err = p.communicate('\0'.join(long_long_argument_list))
... where in most scenarios you should probably avoid raw Popen()
and let a wrapper function like run()
or check_call()
do most of the work:
r = subprocess.run(['xargs', '-0', 'command'],
input='\0'.join(long_long_argument_list),
universal_newlines=True)
out = r.stdout
subprocess.run()
supports text=True
in 3.7+ as the new name of universal_newlines=True
. Older Python versions than 3.5 didn't have run
, so you need to fall back to the older legacy functions check_output
, check_call
, or (rarely) call
.
If you wanted to reimplement xargs
in Python, something like this.
import os
def arg_max_args(args):
"""
Split up the list in `args` into a list of lists
where each list contains fewer than ARG_MAX bytes
(including room for a terminating null byte for each
entry)
"""
arg_max = os.sysconf("SC_ARG_MAX")
result = []
sublist = []
count = 0
for arg in args:
argl = len(arg) + 1
if count + argl > arg_max:
result.append(sublist)
sublist = [arg]
count = argl
else:
sublist.append(arg)
count += argl
if sublist:
result.append(sublist)
return result
You'd run a separate subprocess on each sublist returned by this function.
A proper implementation should raise an error if any one argument is larger than ARG_MAX
but this is just a quick demo.