Skip to content

Honor return type of __new__ even if not a subclass #15182

Open
@oscarbenjamin

Description

@oscarbenjamin

This follows gh-1020 which was closed by gh-7188. This arises because I am adding type hints to sympy (sympy/sympy#25103).

In the following code we have a subclass whose __new__ method is only guaranteed to return objects that are types of the superclass:

from __future__ import annotations

class A:
    def __new__(cls) -> A:
        return super().__new__(cls)

class B(A):
    def __new__(cls) -> A:
        return super().__new__(cls)

reveal_type(B())

With mypy this is rejected and the type of B() is inferred incorrectly but pyright accepts this and gets the inference for B() correct:

$ mypy q.py
q.py:8: error: Incompatible return type for "__new__" (returns "A", but must return a subtype of "B")  [misc]
q.py:11: note: Revealed type is "q.B"
Found 1 error in 1 file (checked 1 source file)
$ pyright q.py
...
  ./q.py:11:13 - information: Type of "B()" is "A"
0 errors, 0 warnings, 1 information 
Completed in 0.955sec

The fix for __new__ in gh-7188 was:

  • If the return type is Any, ignore that and keep the class type as
    the return type
  • Otherwise respect __new__'s return type
  • Produce an error if the return type is not a subtype of the class.

The issue here is about mixing the last two points. Here mypy produces an error but then does not respect __new__'s return type. The return type is not respected since mypy infers that it is of type B unlike pyright which infers type A as specified by the hint. I don't mind mypy reporting an error here but in context -> A is the accurate type hint and mypy should respect that because anything else is just not correct. I can add a type ignore to silence the mypy error but the inference will still be wrong in all downstream/user code and is also inconsistent with pyright.

Activity

oscarbenjamin

oscarbenjamin commented on May 4, 2023

@oscarbenjamin
Author

This is handled here where there is an explicit check to ignore the type given in __new__ if it is not a subtype of the class and also in a bunch of other cases:

mypy/mypy/typeops.py

Lines 184 to 196 in 13f35ad

if (
isinstance(explicit_type, (Instance, TupleType, UninhabitedType))
# We have to skip protocols, because it can be a subtype of a return type
# by accident. Like `Hashable` is a subtype of `object`. See #11799
and isinstance(default_ret_type, Instance)
and not default_ret_type.type.is_protocol
# Only use the declared return type from __new__ or declared self in __init__
# if it is actually returning a subtype of what we would return otherwise.
and is_subtype(explicit_type, default_ret_type, ignore_type_params=True)
):
ret_type: Type = explicit_type
else:
ret_type = default_ret_type

I'm not sure where all of those conditions come from but I don't see how any of them can override the hint given in __new__ because __new__ can return anything (this is generally the reason for using __new__ rather than __init__).

Viicos

Viicos commented on May 4, 2023

@Viicos
Contributor

Hi @oscarbenjamin, this seems to be a duplicate of #8330. I've opened #14471 since then but I'm kind of stuck on it. Hopefully if someone with the required knowledge could see what's wrong in the PR we could get it merged one day!

oscarbenjamin

oscarbenjamin commented on May 4, 2023

@oscarbenjamin
Author

this seems to be a duplicate of #8330

Yes, I saw that issue. I thought this was different but now that I've seen the code that causes this I can see that it causes both issues.

To be honest having looked at the code I don't really understand what any of it is doing. It doesn't bear any resemblance to Python's runtime semantics for type.__call__, __new__ and __init__. For example the code here chooses between taking a hint from __new__ or __init__ based on which comes earlier in the MRO:

mypy/mypy/checkmember.py

Lines 1198 to 1199 in 13f35ad

# We take the type from whichever of __init__ and __new__ is first
# in the MRO, preferring __init__ if there is a tie.

That makes no sense to me:

  • __new__ and __init__ do not share method resolution so it does not matter which comes "first" in the MRO: __new__ is always called first and gets to decide whether __init__ is ever called at all.
  • The concrete type returned is determined solely by __new__ (leaving aside metaclasses overriding type.__call__) and importantly __new__ can return anything.
  • __init__ is irrelevant for determining the type of self: it is only ever called with an instance of the class in which it is defined and will not be called at all if __new__ returns something else.

I would expect the inference to work something like this. We have a class:

class A(B, metaclass=Meta):
    def __new__(cls, *args, **kwargs):
        ...
    def __init__(self, *args, **kwargs):
       ....

When we call A(*args, **kwargs) it will return Meta.__call__(A, *args, **kwargs) so we should determine what type that would return. By default Meta is type whose call method looks like:

class type:
    def __call__(A, *args, **kwargs):
        # Use __new__ to create the object
        if A.__new__ is object.__new__:
            obj = object.__new__(A)
        else:
            obj = A.__new__(A, *args, **kwargs)

        # Call __init__ with the created object (only if is an instance of A):
        if isinstance(obj, A):
            if type(obj).__init__ is not object.__init__:
                obj.__init__(*args, **kwargs)

        # Return the object obtained from __new__
        return obj

The final piece to know is that object.__new__(A) returns an object of type A. The call to A.__new__(A, ...) needs further analysis but usually it will eventually come down to object.__new__(A) (via calls like B.__new__(A) along the way).

We see here that the type of the object returned is just the return type of A.__new__(). If A.__new__() returns an object that is not an instance of A then A.__init__() will never be called meaning any hints in A.__init__() are irrelevant. Even if A.__init__() is called it does not change the type of obj (unless it reassigns obj.__class__ but that is very obscure).

NeilGirdhar

NeilGirdhar commented on Dec 19, 2023

@NeilGirdhar
Contributor

Respecting the metaclass' __call__ return type is required to type check nnx, which makes extensive use of this feature here.

Viicos

Viicos commented on Dec 20, 2023

@Viicos
Contributor

Respecting the metaclass' __call__ return type is required to type check nnx, which makes extensive use of this feature here.

As a workaround, I'm using the __new__ method instead (see #16020 (comment), where I'm hitting the same use case as you)

jorenham

jorenham commented on Mar 25, 2024

@jorenham
Contributor

This currently makes it impossible to correctly type builtins.reversed. See python/typeshed#11645 and python/typeshed#11646

jorenham

jorenham commented on Mar 25, 2024

@jorenham
Contributor

Like @oscarbenjamin noted, this fails in the contravariant case, i.e. when __new__ returns an instance of its supertype.

But when __new__ is annotated to return something else entirely, mypy will simply ignore it, even if annotated explicitly:

class Other:
    def __new__(cls, arg: T, /) -> T:  # error: "__new__" must return a class instance (got "T")  [misc]
        return arg

assert_type(Other(42), 'int')  # error: Expression is of type "Other", not "int"  [assert-type]

For details, see https://mypy-play.net/?mypy=latest&python=3.12&flags=strict&gist=336602d390b5e6566e9fca93d7fa48a6

Viicos

Viicos commented on Mar 25, 2024

@Viicos
Contributor

The PR fixing this can be found here: #16020

jorenham

jorenham commented on Jul 5, 2024

@jorenham
Contributor

@Viicos #16020 does not fix this issue. In the PR description it explicit states:

We avoid fixing #15182 for now.

jorenham

jorenham commented on Jul 5, 2024

@jorenham
Contributor

The official typing specs now include a section on the __new__ method:

For most classes, the return type for the __new__ method is typically Self, but other types are also allowed. For example, the __new__ method may return an instance of a subclass or an instance of some completely unrelated class.

So this confirms that this is, in fact, a bug (and IMO a rather big one).

5 remaining items

jorenham

jorenham commented on Feb 28, 2025

@jorenham
Contributor

This is also a blocker for correctly annotating the numpy.object_ constructor, as well as the constructors of (all) other scalar types in case of "multidimensional" input:

>>> import numpy as np
>>> type(np.object_(42))
<class 'int'>
>>> np.float16([3.14, 6.28])
array([3.14, 6.28], dtype=float16)

See #18343 (comment) for details.

Avasam

Avasam commented on May 25, 2025

@Avasam
SponsorContributor

In the meantime, could Incompatible return type for "__new__" (returns "...", but must return a subtype of "...") and "__new__" must return a class instance (got <some union>) be made into a separate error code than misc ? I can't really ignore them without ignoring the entirety of misc atm.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugmypy got something wrong

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @NeilGirdhar@oscarbenjamin@Avasam@jorenham@Viicos

        Issue actions

          Honor return type of `__new__` even if not a subclass · Issue #15182 · python/mypy