stdlib wrongly uses len() for bytes-like object

[animalize]

BPO	44439
Nosy	@pitrou, @serhiy-storchaka, @animalize, @miss-islington, @iritkatriel
PRs	bpo-44439: BZ2File.write() / LZMAFile.write() handle buffer protocol correctly #26764 [3.10] bpo-44439: BZ2File.write() / LZMAFile.write() handle buffer protocol correctly (GH-26764) #26845 [3.9] bpo-44439: BZ2File.write()/LZMAFile.write() handle length correctly #26846 bpo-44439: _ZipWriteFile.write() handle buffer protocol correctly #29468 [3.9] bpo-44439: _ZipWriteFile.write() handle buffer protocol correctly (GH-29468) #31755 [3.10] bpo-44439: _ZipWriteFile.write() handle buffer protocol correctly (GH-29468) #31756

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-03-19.13:19:59.142>
created_at = <Date 2021-06-17.05:05:24.162>
labels = ['type-bug', 'library', '3.9', '3.10', '3.11']
title = 'stdlib wrongly uses len() for bytes-like object'
updated_at = <Date 2022-03-19.13:19:59.141>
user = 'https://github.com/animalize'

bugs.python.org fields:

activity = <Date 2022-03-19.13:19:59.141>
actor = 'malin'
assignee = 'none'
closed = True
closed_date = <Date 2022-03-19.13:19:59.142>
closer = 'malin'
components = ['Library (Lib)']
creation = <Date 2021-06-17.05:05:24.162>
creator = 'malin'
dependencies = []
files = []
hgrepos = []
issue_num = 44439
keywords = ['patch']
message_count = 14.0
messages = ['395971', '395973', '395976', '396305', '396309', '396334', '396335', '396336', '405948', '414737', '414742', '414743', '415528', '415549']
nosy_count = 6.0
nosy_names = ['pitrou', 'nadeem.vawda', 'serhiy.storchaka', 'malin', 'miss-islington', 'iritkatriel']
pr_nums = ['26764', '26845', '26846', '29468', '31755', '31756']
priority = 'normal'
resolution = 'later'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue44439'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[animalize]

If run this code, it will raise an exception:

    import pickle
    import lzma
    import pandas as pd
    with lzma.open("test.xz", "wb") as file:
        pickle.dump(pd.DataFrame(range(1_000_000)), file, protocol=5)

The exception:

    Traceback (most recent call last):
      File "E:\testlen.py", line 7, in <module>
        pickle.dump(pd.DataFrame(range(1_000_000)), file, protocol=5)
      File "D:\Python39\lib\lzma.py", line 234, in write
        self._pos += len(data)
    TypeError: object of type 'pickle.PickleBuffer' has no len()
    
The exception is raised in lzma.LZMAFile.write() method:
https://github.com/python/cpython/blob/v3.10.0b2/Lib/lzma.py#L238
        
PickleBuffer doesn't have .__len__ method, is it intended?

[serhiy-storchaka]

Oh, LZMAFile.write() should not use len() directly on input data because it does not always work correctly with memoryview and other objects supporting the buffer protocol. It should use memoryview(data).nbytes or data = memoryview(data).cast('B') if other byte-oriented operations (indexing, slicing) are used. See for example Lib/gzip.py, Lib/_pyio.py, Lib/_compression.py, Lib/ssl.py, Lib/socketserver.py, Lib/wave.py.

[animalize]

Ok, I'm working on a PR.

[animalize]

I am checking all the .py files in Lib folder.
hmac.py has two len() bugs:
https://github.com/python/cpython/blob/v3.10.0b3/Lib/hmac.py#L212
https://github.com/python/cpython/blob/v3.10.0b3/Lib/hmac.py#L214

I think PR 26764 is prepared, it fixes the len() bugs in bz2.py/lzma.py files.

[serhiy-storchaka]

New changeset bc6c12c by Ma Lin in branch 'main':
bpo-44439: BZ2File.write() / LZMAFile.write() handle buffer protocol correctly (GH-26764)
bc6c12c

[serhiy-storchaka]

New changeset 8bc26d8 by Ma Lin in branch '3.9':
bpo-44439: BZ2File.write()/LZMAFile.write() handle length correctly (GH-26846)
8bc26d8

[serhiy-storchaka]

Thank you for your contribution Ma Lin.