Manually setting module to `main` for callables causes errors starting in 0.3.5 #482

dwhswenson · 2022-05-21T22:26:23Z

Context: Our code uses dill to serialize callables only under certain circumstances, including when the callable's module is __main__. We want to be able to test this at an integration level, using dill on a function where func.__module__ == __main__.

Prior to the release of 0.3.5, we did this by manually setting the __module__ attribute. However, our tests have been failing since the release of 0.3.5.

I'm not sure if this is supposed to be an allowed use case. But is there supposed to be another way that I make it look like my functions were defined in __main__ (for the part our code manages) such that dill can also handle it, when really I'm running under pytest or another testing framework?

MCVE: In an environment with pytest and dill installed:

# test_dill.py

import dill

def setup_module():
    print(f"\nDill version: {dill.__version__}")


def external_func_pretending_to_be_main():
    return


def test_dill_lambda():
    print()  # cleaner output
    func = lambda x: x
    func.__module__ = "__main__"
    dill.dumps(func)


def test_dill_func():
    print() # cleaner output
    func = external_func_pretending_to_be_main
    func.__module__ = "__main__"
    dill.dumps(func)

pytest -vs test_dill.py

Output in 0.3.4:

test_dill.py::test_dill_lambda
Dill version: 0.3.4

PASSED
test_dill.py::test_dill_func
PASSED

Output in 0.3.5.1

test_dill.py::test_dill_lambda
Dill version: 0.3.5.1

FAILED
test_dill.py::test_dill_func
FAILED

Tracebacks, as reported by pytest:

test_dill_lambda

../../mambaforge/envs/dill/lib/python3.10/site-packages/dill/_dill.py:364: in dumps
    dump(obj, file, protocol, byref, fmode, recurse, **kwds)#, strictio)
../../mambaforge/envs/dill/lib/python3.10/site-packages/dill/_dill.py:336: in dump
    Pickler(file, protocol, **_kwds).dump(obj)
../../mambaforge/envs/dill/lib/python3.10/site-packages/dill/_dill.py:620: in dump
    StockPickler.dump(self, obj)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:487: in dump
    self.save(obj)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
../../mambaforge/envs/dill/lib/python3.10/site-packages/dill/_dill.py:1963: in save_function
    _save_with_postproc(pickler, (_create_function, (
../../mambaforge/envs/dill/lib/python3.10/site-packages/dill/_dill.py:1154: in _save_with_postproc
    pickler._batch_setitems(iter(source.items()))
../../mambaforge/envs/dill/lib/python3.10/pickle.py:998: in _batch_setitems
    save(v)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:717: in save_reduce
    save(state)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
../../mambaforge/envs/dill/lib/python3.10/site-packages/dill/_dill.py:1251: in save_module_dict
    StockPickler.save_dict(pickler, obj)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:972: in save_dict
    self._batch_setitems(obj.items())
../../mambaforge/envs/dill/lib/python3.10/pickle.py:998: in _batch_setitems
    save(v)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:717: in save_reduce
    save(state)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
../../mambaforge/envs/dill/lib/python3.10/site-packages/dill/_dill.py:1251: in save_module_dict
    StockPickler.save_dict(pickler, obj)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:972: in save_dict
    self._batch_setitems(obj.items())
../../mambaforge/envs/dill/lib/python3.10/pickle.py:998: in _batch_setitems
    save(v)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:687: in save_reduce
    save(cls)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
../../mambaforge/envs/dill/lib/python3.10/site-packages/dill/_dill.py:1838: in save_type
    _save_with_postproc(pickler, (_create_type, (
../../mambaforge/envs/dill/lib/python3.10/site-packages/dill/_dill.py:1140: in _save_with_postproc
    pickler.save_reduce(*reduction, obj=obj)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:692: in save_reduce
    save(args)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
../../mambaforge/envs/dill/lib/python3.10/pickle.py:902: in save_tuple
    save(element)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
../../mambaforge/envs/dill/lib/python3.10/site-packages/dill/_dill.py:1251: in save_module_dict
    StockPickler.save_dict(pickler, obj)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:972: in save_dict
    self._batch_setitems(obj.items())
../../mambaforge/envs/dill/lib/python3.10/pickle.py:998: in _batch_setitems
    save(v)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
../../mambaforge/envs/dill/lib/python3.10/site-packages/dill/_dill.py:1963: in save_function
    _save_with_postproc(pickler, (_create_function, (
../../mambaforge/envs/dill/lib/python3.10/site-packages/dill/_dill.py:1154: in _save_with_postproc
    pickler._batch_setitems(iter(source.items()))
../../mambaforge/envs/dill/lib/python3.10/pickle.py:998: in _batch_setitems
    save(v)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
../../mambaforge/envs/dill/lib/python3.10/site-packages/dill/_dill.py:2004: in save_function
    StockPickler.save_global(pickler, obj, name=name)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <dill._dill.Pickler object at 0x1082f2e90>
obj = <function make_set_closure_cell.<locals>.set_closure_cell at 0x1053c6290>
name = 'make_set_closure_cell.<locals>.set_closure_cell'

    def save_global(self, obj, name=None):
        write = self.write
        memo = self.memo

        if name is None:
            name = getattr(obj, '__qualname__', None)
        if name is None:
            name = obj.__name__

        module_name = whichmodule(obj, name)
        try:
            __import__(module_name, level=0)
            module = sys.modules[module_name]
            obj2, parent = _getattribute(module, name)
        except (ImportError, KeyError, AttributeError):
>           raise PicklingError(
                "Can't pickle %r: it's not found as %s.%s" %
                (obj, module_name, name)) from None
E           _pickle.PicklingError: Can't pickle <function make_set_closure_cell.<locals>.set_closure_cell at 0x1053c6290>: it's not found as attr._compat.make_set_closure_cell.<locals>.set_closure_cell

../../mambaforge/envs/dill/lib/python3.10/pickle.py:1071: PicklingError

test_dill_func

../../mambaforge/envs/dill/lib/python3.10/site-packages/dill/_dill.py:364: in dumps
    dump(obj, file, protocol, byref, fmode, recurse, **kwds)#, strictio)
../../mambaforge/envs/dill/lib/python3.10/site-packages/dill/_dill.py:336: in dump
    Pickler(file, protocol, **_kwds).dump(obj)
../../mambaforge/envs/dill/lib/python3.10/site-packages/dill/_dill.py:620: in dump
    StockPickler.dump(self, obj)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:487: in dump
    self.save(obj)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
../../mambaforge/envs/dill/lib/python3.10/site-packages/dill/_dill.py:1963: in save_function
    _save_with_postproc(pickler, (_create_function, (
../../mambaforge/envs/dill/lib/python3.10/site-packages/dill/_dill.py:1154: in _save_with_postproc
    pickler._batch_setitems(iter(source.items()))
../../mambaforge/envs/dill/lib/python3.10/pickle.py:998: in _batch_setitems
    save(v)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:717: in save_reduce
    save(state)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
../../mambaforge/envs/dill/lib/python3.10/site-packages/dill/_dill.py:1251: in save_module_dict
    StockPickler.save_dict(pickler, obj)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:972: in save_dict
    self._batch_setitems(obj.items())
../../mambaforge/envs/dill/lib/python3.10/pickle.py:998: in _batch_setitems
    save(v)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:717: in save_reduce
    save(state)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
../../mambaforge/envs/dill/lib/python3.10/site-packages/dill/_dill.py:1251: in save_module_dict
    StockPickler.save_dict(pickler, obj)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:972: in save_dict
    self._batch_setitems(obj.items())
../../mambaforge/envs/dill/lib/python3.10/pickle.py:998: in _batch_setitems
    save(v)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:687: in save_reduce
    save(cls)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
../../mambaforge/envs/dill/lib/python3.10/site-packages/dill/_dill.py:1838: in save_type
    _save_with_postproc(pickler, (_create_type, (
../../mambaforge/envs/dill/lib/python3.10/site-packages/dill/_dill.py:1140: in _save_with_postproc
    pickler.save_reduce(*reduction, obj=obj)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:692: in save_reduce
    save(args)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
../../mambaforge/envs/dill/lib/python3.10/pickle.py:902: in save_tuple
    save(element)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
../../mambaforge/envs/dill/lib/python3.10/site-packages/dill/_dill.py:1251: in save_module_dict
    StockPickler.save_dict(pickler, obj)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:972: in save_dict
    self._batch_setitems(obj.items())
../../mambaforge/envs/dill/lib/python3.10/pickle.py:998: in _batch_setitems
    save(v)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
../../mambaforge/envs/dill/lib/python3.10/site-packages/dill/_dill.py:1963: in save_function
    _save_with_postproc(pickler, (_create_function, (
../../mambaforge/envs/dill/lib/python3.10/site-packages/dill/_dill.py:1154: in _save_with_postproc
    pickler._batch_setitems(iter(source.items()))
../../mambaforge/envs/dill/lib/python3.10/pickle.py:998: in _batch_setitems
    save(v)
../../mambaforge/envs/dill/lib/python3.10/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
../../mambaforge/envs/dill/lib/python3.10/site-packages/dill/_dill.py:2004: in save_function
    StockPickler.save_global(pickler, obj, name=name)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <dill._dill.Pickler object at 0x10849f910>
obj = <function make_set_closure_cell.<locals>.set_closure_cell at 0x1053c6290>
name = 'make_set_closure_cell.<locals>.set_closure_cell'

    def save_global(self, obj, name=None):
        write = self.write
        memo = self.memo

        if name is None:
            name = getattr(obj, '__qualname__', None)
        if name is None:
            name = obj.__name__

        module_name = whichmodule(obj, name)
        try:
            __import__(module_name, level=0)
            module = sys.modules[module_name]
            obj2, parent = _getattribute(module, name)
        except (ImportError, KeyError, AttributeError):
>           raise PicklingError(
                "Can't pickle %r: it's not found as %s.%s" %
                (obj, module_name, name)) from None
E           _pickle.PicklingError: Can't pickle <function make_set_closure_cell.<locals>.set_closure_cell at 0x1053c6290>: it's not found as attr._compat.make_set_closure_cell.<locals>.set_closure_cell

../../mambaforge/envs/dill/lib/python3.10/pickle.py:1071: PicklingError

The text was updated successfully, but these errors were encountered:

anivegesana · 2022-05-21T23:00:17Z

That is very odd. Based on the error message, it seems like the issue is with pickling something in the attrs package as opposed to your code, but when I run

import dill
x = attr._compat.make_set_closure_cell()
dill.dumps(x)

it pickles just fine. What is clear right now is that dill is delegating a function to the default Python pickler when it shouldn't.

anivegesana · 2022-05-22T00:40:56Z

Ah. I found this issue. I guess this was a very old cryptic bug that was present since the Python 2/3 transition that was never fixed. You stumbled across the bug after I had made modifications to the function pickling code, which now used the code that wasn't transitioned when it didn't before.

The problem is that dill is looking for set_closure_cell, a local of make_set_closure_cell in attr._compat. In Python 2, inner functions had the same namespace as their outer function. But Python 3 introduced __qualname__, which allows for the function to be more descriptive and unique: make_set_closure_cell.<locals>.set_closure_cell. Instead, currently, dill tries to find set_closure_cell directly in attr._compat, which may not be the same object. In your test case, it was not the same object and it crashed.

This is the code that is outdated:

dill/dill/_dill.py

Lines 1133 to 1139 in 78e3d55

    
           def _locate_function(obj, pickler=None): 
        
               if obj.__module__ in ['__main__', None] or \ 
        
                       pickler and is_dill(pickler, child=False) and pickler._session and obj.__module__ == pickler._main.__name__: 
        
                   return False 
        
               found = _import_module(obj.__module__ + '.' + obj.__name__, safe=True) 
        
               return found is obj

This is the corrected Python 3 code:
https://github.com/uqfoundation/dill/pull/450/files#diff-bdd40cc870892c410adce8c1b155daf5e6c5b5e7192e751f171e6dae6a578dc1R1115

@mmckerns I assumed that this code made its way into the repo, but I guess I had added it to #450 directly because I needed this feature immediately and had forgotten to open a separate PR for it. Opening one now.

anivegesana · 2022-05-26T20:41:47Z

I am fairly convinced that this is the problem. I will open a PR to add this to dill 0.3.5.2 (if this is still coming out.) Also, dill 0.3.5.* fixes a bug that made it so that dill didn't recurse all of the way down all functions, stopping early in a couple of cases that it shouldn't have. To make this code work in both dill 0.3.5 and earlier versions, use

dill.dumps(func, recurse=True)

or set the setting globally

dill.settings['recurse'] = True

I know that it is a little bizarre that to limit the amount of recursion that dill is doing to pickle the function, we need to turn on a setting called recurse, but that is because the setting actually recurses over the global dictionary and finds the smallest subset that the function needs to run, which will limit the number of objects that dill needs to include in the pickle.

with recurse=True, objects referred to in the global dictionary are recursively traced and pickled, instead of the default behavior of attempting to store the entire global dictionary.

Without this setting, dill will save the entire __main__.__dict__ dictionary into the pickle so that it is available to the function, which includes some objects in PyTest that cannot be pickled.

@dwhswenson Does this fix your test suite for dill 0.3.5?

dwhswenson · 2022-05-30T17:17:37Z

Thanks for working on this @anivegesana (and sorry for slow replies -- the affected project is passion project; day job gets in the way).

Using recurse=True seems to be a workaround (on master or on #486). Without recurse=True, #486 (at c81dfe5) leads to a different error in the same spot, both in my actual tests in the MCVE above. I'm getting:

TypeError: cannot pickle 'EncodedFile' object

Full error (from pytest, just showing test_dill_func from above)

____________________________________ test_dill_func ____________________________________

    def test_dill_func():
        print() # cleaner output
        func = external_func_pretending_to_be_main
        func.__module__ = "__main__"
>       dill.dumps(func)

test_dill.py:24:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
dill/dill/_dill.py:364: in dumps
    dump(obj, file, protocol, byref, fmode, recurse, **kwds)#, strictio)
dill/dill/_dill.py:336: in dump
    Pickler(file, protocol, **_kwds).dump(obj)
dill/dill/_dill.py:620: in dump
    StockPickler.dump(self, obj)
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:487: in dump
    self.save(obj)
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
dill/dill/_dill.py:2024: in save_function
    _save_with_postproc(pickler, (_create_function, (
dill/dill/_dill.py:1206: in _save_with_postproc
    pickler._batch_setitems(iter(source.items()))
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
dill/dill/_dill.py:1312: in save_module_dict
    StockPickler.save_dict(pickler, obj)
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
dill/dill/_dill.py:1312: in save_module_dict
    StockPickler.save_dict(pickler, obj)
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
dill/dill/_dill.py:1312: in save_module_dict
    StockPickler.save_dict(pickler, obj)
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
dill/dill/_dill.py:1312: in save_module_dict
    StockPickler.save_dict(pickler, obj)
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
dill/dill/_dill.py:1312: in save_module_dict
    StockPickler.save_dict(pickler, obj)
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
dill/dill/_dill.py:1312: in save_module_dict
    StockPickler.save_dict(pickler, obj)
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:603: in save
    self.save_reduce(obj=obj, *rv)
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:717: in save_reduce
    save(state)
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:560: in save
    f(self, obj)  # Call unbound method with explicit self
dill/dill/_dill.py:1312: in save_module_dict
    StockPickler.save_dict(pickler, obj)
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:971: in save_dict
    self._batch_setitems(obj.items())
../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:997: in _batch_setitems
    save(v)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <dill._dill.Pickler object at 0x10e67f820>
obj = <_io.TextIOWrapper name="<_io.FileIO name=7 mode='rb+' closefd=True>" mode='r+' encoding='utf-8'>
save_persistent_id = True

    def save(self, obj, save_persistent_id=True):
        self.framer.commit_frame()

        # Check for persistent id (defined by a subclass)
        pid = self.persistent_id(obj)
        if pid is not None and save_persistent_id:
            self.save_pers(pid)
            return

        # Check the memo
        x = self.memo.get(id(obj))
        if x is not None:
            self.write(self.get(x[0]))
            return

        rv = NotImplemented
        reduce = getattr(self, "reducer_override", None)
        if reduce is not None:
            rv = reduce(obj)

        if rv is NotImplemented:
            # Check the type dispatch table
            t = type(obj)
            f = self.dispatch.get(t)
            if f is not None:
                f(self, obj)  # Call unbound method with explicit self
                return

            # Check private dispatch table if any, or else
            # copyreg.dispatch_table
            reduce = getattr(self, 'dispatch_table', dispatch_table).get(t)
            if reduce is not None:
                rv = reduce(obj)
            else:
                # Check for a class with a custom metaclass; treat as regular
                # class
                if issubclass(t, type):
                    self.save_global(obj)
                    return

                # Check for a __reduce_ex__ method, fall back to __reduce__
                reduce = getattr(obj, "__reduce_ex__", None)
                if reduce is not None:
>                   rv = reduce(self.proto)
E                   TypeError: cannot pickle 'EncodedFile' object

../../mambaforge/envs/dill-check/lib/python3.9/pickle.py:578: TypeError

anivegesana · 2022-05-30T23:40:12Z

In your particular case, it makes sense to use recurse=True. Without it, dill will save unnecessary globals created by PyTest even if are not used by your code, so you can treat this as a permanent fix. The fact that EncodedFile is not able to be pickled might entail that something else is broken, so I'll look into that, but it shouldn't effect your code.

dwhswenson mentioned this issue May 21, 2022

Dill 0.3.5 causes problems in test suite openpathsampling/openpathsampling#1112

Closed

This was referenced May 26, 2022

Lookup __qualname__ instead of __name__ in Python 3 #486

Merged

regression from 0.3.4 to 0.3.5.x - class decorators #483

Closed

mmckerns added this to the dill-0.3.6 milestone May 29, 2022

dwhswenson mentioned this issue May 30, 2022

(Possible temporary) Fix dill in testing openpathsampling/openpathsampling#1114

Merged

mmckerns closed this as completed in #486 Jun 6, 2022

mmckerns added the bug label Jun 6, 2022

tjni mentioned this issue Nov 13, 2022

Fix compatibility with dill 0.3.6. apache/beam#24143

Closed

4 tasks

anivegesana mentioned this issue Apr 13, 2023

[Discuss] Recent breaking changes in new dill releases #589

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Manually setting module to `main` for callables causes errors starting in 0.3.5 #482

Manually setting module to `main` for callables causes errors starting in 0.3.5 #482

dwhswenson commented May 21, 2022

anivegesana commented May 21, 2022 •

edited

Loading

anivegesana commented May 22, 2022

anivegesana commented May 26, 2022 •

edited

Loading

dwhswenson commented May 30, 2022

anivegesana commented May 30, 2022

Manually setting module to __main__ for callables causes errors starting in 0.3.5 #482

Manually setting module to __main__ for callables causes errors starting in 0.3.5 #482

Comments

dwhswenson commented May 21, 2022

anivegesana commented May 21, 2022 • edited Loading

anivegesana commented May 22, 2022

anivegesana commented May 26, 2022 • edited Loading

dwhswenson commented May 30, 2022

anivegesana commented May 30, 2022

Manually setting module to `main` for callables causes errors starting in 0.3.5 #482

Manually setting module to `main` for callables causes errors starting in 0.3.5 #482

anivegesana commented May 21, 2022 •

edited

Loading

anivegesana commented May 26, 2022 •

edited

Loading