TemporaryDirectory clean-up fails with unsearchable directories

[lilydjwg]

BPO	35144
Nosy	@lilydjwg, @serhiy-storchaka, @eryksun, @vidartf, @tirkarthi
PRs	bpo-26660, bpo-35144: Fix permission errors in TemporaryDirectory cleanup. #10320

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = None
created_at = <Date 2018-11-02.08:40:08.637>
labels = ['3.8', 'type-bug', 'library', '3.9', '3.10']
title = 'TemporaryDirectory clean-up fails with unsearchable directories'
updated_at = <Date 2021-03-28.07:31:28.296>
user = 'https://github.com/lilydjwg'

bugs.python.org fields:

activity = <Date 2021-03-28.07:31:28.296>
actor = 'eryksun'
assignee = 'serhiy.storchaka'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2018-11-02.08:40:08.637>
creator = 'lilydjwg'
dependencies = []
files = []
hgrepos = []
issue_num = 35144
keywords = ['patch']
message_count = 9.0
messages = ['329116', '329127', '329184', '329185', '344036', '377309', '377316', '377329', '377358']
nosy_count = 5.0
nosy_names = ['lilydjwg', 'serhiy.storchaka', 'eryksun', 'vidartf', 'xtreak']
pr_nums = ['10320']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue35144'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

Linked PRs

• gh-79325: Fix recursion error in TemporaryDirectory cleanup on Windows #112762
• [3.12] gh-79325: Fix recursion error in TemporaryDirectory cleanup on Windows (GH-112762) #112847
• [3.11] gh-79325: Fix recursion error in TemporaryDirectory cleanup on Windows (GH-112762) #112848

[lilydjwg]

If the title doesn't explain clearly, here's a demo program that will fail:

import tempfile
import pathlib

def test():
  with tempfile.TemporaryDirectory(prefix='test-bad-') as tmpdir:
    tmpdir = pathlib.Path(tmpdir)
    subdir = tmpdir / 'sub'
    subdir.mkdir()
    with open(subdir / 'file', 'w'):
      pass
    subdir.chmod(0o600)

if __name__ == '__main__':
  test()

I didn't expect this, and I didn't find an easy way to handle this except not using TemporaryDirectory at all:

import tempfile
import pathlib
import shutil
import os

def rmtree_error(func, path, excinfo):
  if isinstance(excinfo[1], PermissionError):
    os.chmod(os.path.dirname(path), 0o700)
    os.unlink(path)
  print(func, path, excinfo)

def test():
  tmpdir = tempfile.mkdtemp(prefix='test-good-')
  try:
    tmpdir = pathlib.Path(tmpdir)
    subdir = tmpdir / 'sub'
    subdir.mkdir()
    with open(subdir / 'file', 'w'):
      pass
    subdir.chmod(0o600)
  finally:
    shutil.rmtree(tmpdir, onerror=rmtree_error)

if __name__ == '__main__':
  test()

This works around the issue, but the dirfd is missing in the onerror callback.

I have this issue because my program extracts tarballs to a temporary directory for examination. I expected that TemporaryDirectory cleaned up things when it could.

What do you think? rm -rf can't remove such a directory either but this is annoying and I think Python can do better.

[tirkarthi]

Seems like a related issue : bpo-26660 . Maybe TemporaryDirectory can allow an onerror argument that is passed internally to rmtree during cleanup and state the same in the documentation that TemporaryDirectory can't cleanup read-only files?

[lilydjwg]

Yes bpo-26660 is similar but not the same. On Windows it seems a read-only file cannot be deleted while on Linux a file resident in a non-searchable directory cannot be deleted.

An onerror for TemporaryDirectory will work. Also I'd like to add an optional dir_fd argument for onerror.

[serhiy-storchaka]

I am working on this. Left to test on Windows and analyze possible security issues.

[serhiy-storchaka]

New changeset e9b51c0 by Serhiy Storchaka in branch 'master':
bpo-26660, bpo-35144: Fix permission errors in TemporaryDirectory cleanup. (GH-10320)
e9b51c0

[vidartf]

On Python 3.8.5 on Windows using the code from the above patch I recently got a stack overflow:

Thread 0x00002054 (most recent call first):
File "...\lib\concurrent\futures\thread.py", line 78 in _worker
File "...\lib\threading.py", line 870 in run
File "...\lib\threading.py", line 932 in _bootstrap_inner
File "...\lib\threading.py", line 890 in _bootstrap

Thread 0x00000de4 (most recent call first):
File "...\lib\concurrent\futures\thread.py", line 78 in _worker
File "...\lib\threading.py", line 870 in run
File "...\lib\threading.py", line 932 in _bootstrap_inner
File "...\lib\threading.py", line 890 in _bootstrap

Current thread 0x00004700 (most recent call first):
File "...\lib\tempfile.py", line 803 in onerror
File "...\lib\shutil.py", line 619 in _rmtree_unsafe
File "...\lib\shutil.py", line 737 in rmtree
File "...\lib\tempfile.py", line 814 in _rmtree
File "...\lib\tempfile.py", line 806 in onerror
File "...\lib\shutil.py", line 619 in _rmtree_unsafe
File "...\lib\shutil.py", line 737 in rmtree
... repeating

-------------------------------------------

In my case, the outer exc_info from rmtree is:

PermissionError(13, 'The process cannot access the file because it is being used by another process')

And the inner exception from _os.unlink(path) is:

PermissionError(13, 'Access is denied')

I would say that expected behavior in this case would be to let the 'file is in use' error raise, instead of killing the process with an SO.

[serhiy-storchaka]

Thank you Vidar! I wasn't sure about Windows, but was not able to reproduce possible failures. Your report gives a clue.

[vidartf]

A somewhat easy repro:

Create the temporary directory, add a subdir (not sure if subdir truly necessary at this point), use os.chdir() to set the cwd to that subdir. Clean up the temp dir. The cwd should prevent the deletion because it will be "in use".

[eryksun]

It seems to me that if path == name, then resetperms(path) and possibly a recursive call are only needed on the first call. In subsequent calls, if path == name, then we know that resetperms(path) was already called, so it shouldn't handle PermissionError. If resetperms was ineffective (e.g. in Windows, a sharing violation or custom discretionary/mandatory permissions), or if something else changed the permissions in the mean time, just give up instead of risking a RecursionError or stack overflow. For example:

    @classmethod
    def _rmtree(cls, name, first_call=True):
        resetperms_funcs = (_os.unlink, _os.rmdir, _os.scandir, _os.open)

        def resetperms(path):
            try:
                _os.chflags(path, 0)
            except AttributeError:
                pass
            _os.chmod(path, 0o700)

        def onerror(func, path, exc_info):
            if (issubclass(exc_info[0], PermissionError) and
                  func in resetperms_funcs and (first_call or path != name)):
                try:
                    if path != name:
                        resetperms(_os.path.dirname(path))
                    resetperms(path)
                    try:
                        _os.unlink(path)
                    # PermissionError is raised on FreeBSD for directories
                    except (IsADirectoryError, PermissionError):
                        cls._rmtree(path, first_call=False)
                except FileNotFoundError:
                    pass
            elif issubclass(exc_info[0], FileNotFoundError):
                pass
            else:
                raise

        _shutil.rmtree(name, onerror=onerror)