Dynamic communication between main and subprocess in Python

You want to make a Popen object with subprocess.PIPE for standard input and output and use its file objects to communicate—rather than using one of the cantrips like run (and the older, more specific ones like check_output). The challenge is avoiding deadlock: it’s easy to land in a situation where each process is trying to write, the pipe buffers fill (because no one is reading from them), and everything hangs. You also have to remember to flush in both processes, to avoid having a request or response stuck in a file object’s buffer.

Popen.communicate is provided to avoid these issues, but it supports only a single string (rather than an ongoing conversation). The traditional solution is select, but it also works to use separate threads to send requests and read results. (This is one of the reasons to use CPython threads in spite of the GIL: each exists to run while the other is blocked, so there’s very little contention.) Of course, synchronization is then an issue, and you may need to do some work to make the multithreaded client act like a simple, synchronous function call on the outside.

Note that both processes need to flush, but it’s enough if either implements such non-blocking I/O; one normally does that job in the process that starts the other because that’s where it’s known to be necessary (and such programs are the exception).

It seems like pipe might be a suitable choice for your use case. Beware though that under normal circumstance both reading and writing ends expect data to be written or read, respectively. Also make sure you do not get surprised by buffering (nothing comes through because buffers would not get automatically flushed except on an expected boundary, unless set accordingly).

A basic example of how two pipes (they are unidirectional) can be used between two processes:

import os

def child():
    """This function is executed in a child process."""
    infile = os.fdopen(r1)
    outfile = os.fdopen(w2, 'w', buffering=1)
    for line in infile:
        if line.rstrip() == 'quit':
            break
        print(line.upper(), end='', file=outfile)

def parent():
    """This function is executed in a parent process."""
    outfile = os.fdopen(w1, 'w', buffering=1)
    infile = os.fdopen(r2)
    print('Foo', file=outfile)
    print(infile.readline(), end='')
    print('bar', file=outfile)
    print(infile.readline(), end='')
    print('quit', file=outfile)

(r1, w1) = os.pipe()  # for parent -> child writes
(r2, w2) = os.pipe()  # for child -> parent writes
pid = os.fork()
if pid == 0:
    child()  # child code runs here
elif pid > 0:
    parent()  # parent code runs here
    os.waitpid(pid, 0)  # wait for child
else:
    raise RuntimeError("This should not have happened.")

Indeed it would be easier and more practical to use subprocess, and you likely want to run another program. The former would require to be told not to close the pipe file descriptors and the latter would require the pipe file descriptors to be inheritable (not have O_CLOEXEC flag set).

Child program:

import os
import sys

infile = os.fdopen(int(sys.argv[1]))
outfile = os.fdopen(int(sys.argv[2]), 'w', buffering=1)    
for line in infile:
    if line.rstrip() == 'quit':
        break
    print(line.upper(), end='', file=outfile)

Parent program:

import os
import subprocess

(r1, w1) = os.pipe2(0)  # for parent -> child writes
(r2, w2) = os.pipe2(0)  # for child -> parent writes    
child = subprocess.Popen(['./child.py', str(r1), str(w2)], pass_fds=(r1, w2))
outfile = os.fdopen(w1, 'w', buffering=1)
infile = os.fdopen(r2)
print('Foo', file=outfile)
print(infile.readline(), end='')
print('bar', file=outfile)
print(infile.readline(), end='')
print('quit', file=outfile)
child.wait()

If the child program does not need standard input nor standard output, they could be used to get information respectively in and out of the child program. This would even be simpler.

Child program:

import sys

for line in sys.stdin:
    if line.rstrip() == 'quit':
        break
    print(line.upper(), end='', flush=True)

Parent program:

import os
import subprocess

(r1, w1) = os.pipe2(0)  # for parent -> child writes
(r2, w2) = os.pipe2(0)  # for child -> parent writes
child = subprocess.Popen(['./child.py'], stdin=r1, stdout=w2)
outfile = os.fdopen(w1, 'w', buffering=1)
infile = os.fdopen(r2)
print('Foo', file=outfile)
print(infile.readline(), end='')
print('bar', file=outfile)
print(infile.readline(), end='')
print('quit', file=outfile)
child.wait()

As stated, it is not really Python specific and these are just rough hints on how pipes as one option could be used.