Allow multiprocessing finalizers to run on a separate thread

[pitrou]

Feature or enhancement

Proposal:

The multiprocessing library allows finalizers to be attached to specific objects, so as to release system resources. These finalizers may have non-trivial code that rely on and alter global state.

Unfortunately, since mainline code can be interrupted at any time by the GC, there can be rare reentrancy issues with such non-trivial finalizers, such as in gh-109593.

Ideally, we would run such finalizers in a separate thread rather than from inside the GC.

Note this is related to https://peps.python.org/pep-0556/, but much more focussed and therefore with a higher chance of success.

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

Linked PRs

• GH-109978: Allow multiprocessing finalizers to run on a separate thread #110510

[pitrou]

@vstinner FYI

[pitrou]

@iritkatriel @pablogsal IIRC you tried to work on PEP 556 before.

[pitrou]

Ok, I gave it a try... but, of course, the problem is that fork wreaks havoc with threads, and of course it's difficult to test multiprocessing (or concurrent.futures) without making a lot of calls to fork.

:-D