Description
Bug report
Bug description:
This came up in #130849 (comment)
The problem is that popen_fork.Popen (and popen_spawn.Popen and popen_forkserver.Popen) are not thread-safe:
cpython/Lib/multiprocessing/popen_fork.py
Lines 25 to 35 in 02de9cb
The first successful call to os.waitpid() may reap the pid so that subsequent calls raise an OSError. I've only seen this on macOS (not Linux). We may not yet however have set self.returncode -- that happens a few statements later, so poll() can return None if:
- The process has finished
- Another thread called poll(), but hasn't yet set
self.returncode
And then is_alive() can return True:
cpython/Lib/multiprocessing/process.py
Lines 153 to 170 in 02de9cb
Note that some classes like concurrent.futures.ProcessPoolExecutor use threads internally, so the user may not even know that threads are involved.
Repro:
repro.py
import os
import multiprocessing as mp
import threading
import time
import sys
original_excepthook = threading.excepthook
def on_except(args):
original_excepthook(args)
os._exit(1)
threading.excepthook = on_except
def p1():
pass
def thread1(p):
while p.is_alive():
time.sleep(0.00001)
pass
def test():
for i in range(1000):
print(i)
p = mp.Process(target=p1)
p.start()
t = threading.Thread(target=thread1, args=(p,))
t.start()
p.join()
assert not p.is_alive()
t.join()
def main():
threads = [threading.Thread(target=test) for _ in range(10)]
for t in threads:
t.start()
for t in threads:
t.join()
if __name__ == "__main__":
main()NOTE:
- This is unrelated to free threading
popen_fork.Popen(and subclasses) are distinct fromsubprocess.Popen
CPython versions tested on:
CPython main branch
Operating systems tested on:
macOS