BlockingIOError: [Errno 11] Resource temporarily unavailable: on GPFS.

[pconesamingo]

BPO	43743
Nosy	@gpshead, @giampaolo, @alexeicolin, @pmrv
PRs	bpo-43743: add comment stating _USE_CP_SENDFILE should not be removed #26024
Files	sendfile.py shutil.patch: Fix for BlockingIOError in shutil.copy

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-05-06.19:04:38.663>
created_at = <Date 2021-04-06.08:21:31.814>
labels = ['type-bug', '3.8', '3.9', '3.10', 'expert-IO', 'invalid']
title = 'BlockingIOError: [Errno 11] Resource temporarily unavailable: on GPFS.'
updated_at = <Date 2022-03-13.15:22:25.341>
user = 'https://bugs.python.org/pconesamingo'

bugs.python.org fields:

activity = <Date 2022-03-13.15:22:25.341>
actor = 'pmrv'
assignee = 'none'
closed = True
closed_date = <Date 2021-05-06.19:04:38.663>
closer = 'gregory.p.smith'
components = ['IO']
creation = <Date 2021-04-06.08:21:31.814>
creator = 'p.conesa.mingo'
dependencies = []
files = ['50671', '50672']
hgrepos = []
issue_num = 43743
keywords = ['patch']
message_count = 9.0
messages = ['390297', '391649', '392985', '393134', '393364', '393419', '393429', '415037', '415038']
nosy_count = 6.0
nosy_names = ['gregory.p.smith', 'giampaolo.rodola', 'alexeicolin', 'p.conesa.mingo', 'PEAR', 'pmrv']
pr_nums = ['26024']
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue43743'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

Linked PRs

• gh-87909: avoid a 0 bytes copy call to sendfile() #101231

[pconesamingo]

Hi, one of our users is reporting this starting to happen in a GPFS. All has been working fine for NTFS so far for many years.

I had a look at my shutil code, and I can see the try/except code trying to fall back to the "slower" copyfileobj(fsrc, fdst).

But it seems, by the stacktrace bellow that the "catch" is not happening.

Any idea how to fix this?

I guess something like:

import shutil
shutil._USE_CP_SENDFILE = False

should avoid the fast_copy attempt.

> Traceback (most recent call last):
>   File "/opt/pxsoft/scipion/v3/ubuntu20.04/scipion-em-esrf/esrf/workflow/esrf_launch_workflow.py", line 432, in <module>
>     project.scheduleProtocol(prot)
>   File "/opt/pxsoft/scipion/v3/ubuntu20.04/anaconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/project/project.py", line 633, in scheduleProtocol
>     pwutils.path.copyFile(self.dbPath, protocol.getDbPath())
>   File "/opt/px/scipion/v3/ubuntu20.04/anaconda3/envs/.scipion3env/lib/python3.8/site-packages/pyworkflow/utils/path.py", line 247, in copyFile
>     shutil.copy(source, dest)
>   File "/opt/pxsoft/scipion/v3/ubuntu20.04/anaconda3/envs/.scipion3env/lib/python3.8/shutil.py", line 415, in copy
>     copyfile(src, dst, follow_symlinks=follow_symlinks)
>   File "/opt/pxsoft/scipion/v3/ubuntu20.04/anaconda3/envs/.scipion3env/lib/python3.8/shutil.py", line 272, in copyfile
>     _fastcopy_sendfile(fsrc, fdst)
>   File "/opt/pxsoft/scipion/v3/ubuntu20.04/anaconda3/envs/.scipion3env/lib/python3.8/shutil.py", line 169, in _fastcopy_sendfile
>     raise err
>   File "/opt/pxsoft/scipion/v3/ubuntu20.04/anaconda3/envs/.scipion3env/lib/python3.8/shutil.py", line 149, in _fastcopy_sendfile
>     sent = os.sendfile(outfd, infd, offset, blocksize)
> BlockingIOError: [Errno 11] Resource temporarily unavailable: 'project.sqlite' -> 'Runs/000002_ProtImportMovies/logs/run.db'

[alexeicolin]

Can confirm that this BlockingIOError happens on GPFS (alpine) on Summit supercomputer, tested with Python 3.8 and 3.10a7.

I found that it happens only for file sizes above 65536. Minimal example:

This filesize works:

$ rm -f srcfile dstfile && truncate --size 65535 srcfile && python3.10 -c "import shutil; shutil.copyfile(b'srcfile', b'dstfile')"

This file size (and larger) does not work:

$ rm -f srcfile dstfile && truncate --size 65536 srcfile && python3.10 -c "import shutil; shutil.copyfile(b'srcfile', b'dstfile')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/.../usr/lib/python3.10/shutil.py", line 265, in copyfile
    _fastcopy_sendfile(fsrc, fdst)
  File "/.../usr/lib/python3.10/shutil.py", line 162, in _fastcopy_sendfile
    raise err
  File "/.../usr/lib/python3.10/shutil.py", line 142, in _fastcopy_sendfile
    sent = os.sendfile(outfd, infd, offset, blocksize)
BlockingIOError: [Errno 11] Resource temporarily unavailable: b'srcfile' -> b'dstfile'

I tried patching shutil.py to retry the the call on this EAGAIN, but subsequent attempts fail with EAGAIN again indefinitely.

I also use OP's workaround: set _USE_CP_SENDFILE = False in shutil.py

[PEAR]

Most probably related: https://www.ibm.com/support/pages/apar/IJ28891

[gpshead]

I don't believe CPython should be working around a bug in specific Linux kernel versions in the standard library unless they are extremely pernicious and not considered to be a bug and thus ever be fixed in the OS kernel.

As the sendfile system call appears to infinitely return one of EAGAIN, EALREADY, EWOULDBLOCK, or EINPROGRESS in this case, there isn't anything CPython could do. A retry/backoff loop won't help.

This should be worked around at the application level by whatever means are appropriate.