Skip to content

ProcessPoolExecutor.map may hang forever when the submitted iterable is far greater than max_tasks_per_child #111498

Open
@alirzaev

Description

@alirzaev

Bug report

Bug description:

Summary

ProcessPoolExecutor.map may hang forever when the submitted iterable is far greater than max_tasks_per_child

Internally ProcessPoolExecutor.map submits to the executor all items from the iterable (via submit). The worker process exits when the first max_tasks_per_child work items are processed but a new worker process isn't created because ProcessPoolExecutor._adjust_process_count can be invoked only in submit

Reproducible example

import time
from concurrent.futures import ProcessPoolExecutor


def work(i):
    time.sleep(1)

    return i * i


def main():
    with ProcessPoolExecutor(max_workers=1) as executor:
        _ = list(executor.map(work, range(100)))
        # completes

    with ProcessPoolExecutor(max_workers=1, max_tasks_per_child=5) as executor:
        _ = list(executor.map(work, range(100)))
        # hangs forever


if __name__ == '__main__':
    main()

Expected behavior

Program exits

Actual behavor

Program hangs forever

CPython versions tested on:

3.11

Operating systems tested on:

Windows

Metadata

Metadata

Assignees

No one assigned

    Labels

    stdlibPython modules in the Lib dirtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions