Description
This follows gh-1020 which was closed by gh-7188. This arises because I am adding type hints to sympy (sympy/sympy#25103).
In the following code we have a subclass whose __new__
method is only guaranteed to return objects that are types of the superclass:
from __future__ import annotations
class A:
def __new__(cls) -> A:
return super().__new__(cls)
class B(A):
def __new__(cls) -> A:
return super().__new__(cls)
reveal_type(B())
With mypy this is rejected and the type of B()
is inferred incorrectly but pyright accepts this and gets the inference for B()
correct:
$ mypy q.py
q.py:8: error: Incompatible return type for "__new__" (returns "A", but must return a subtype of "B") [misc]
q.py:11: note: Revealed type is "q.B"
Found 1 error in 1 file (checked 1 source file)
$ pyright q.py
...
./q.py:11:13 - information: Type of "B()" is "A"
0 errors, 0 warnings, 1 information
Completed in 0.955sec
The fix for __new__
in gh-7188 was:
- If the return type is Any, ignore that and keep the class type as
the return type - Otherwise respect
__new__
's return type - Produce an error if the return type is not a subtype of the class.
The issue here is about mixing the last two points. Here mypy produces an error but then does not respect __new__
's return type. The return type is not respected since mypy infers that it is of type B
unlike pyright which infers type A
as specified by the hint. I don't mind mypy reporting an error here but in context -> A
is the accurate type hint and mypy should respect that because anything else is just not correct. I can add a type ignore to silence the mypy error but the inference will still be wrong in all downstream/user code and is also inconsistent with pyright.
Activity
oscarbenjamin commentedon May 4, 2023
This is handled here where there is an explicit check to ignore the type given in
__new__
if it is not a subtype of the class and also in a bunch of other cases:mypy/mypy/typeops.py
Lines 184 to 196 in 13f35ad
I'm not sure where all of those conditions come from but I don't see how any of them can override the hint given in
__new__
because__new__
can return anything (this is generally the reason for using__new__
rather than__init__
).Viicos commentedon May 4, 2023
Hi @oscarbenjamin, this seems to be a duplicate of #8330. I've opened #14471 since then but I'm kind of stuck on it. Hopefully if someone with the required knowledge could see what's wrong in the PR we could get it merged one day!
oscarbenjamin commentedon May 4, 2023
Yes, I saw that issue. I thought this was different but now that I've seen the code that causes this I can see that it causes both issues.
To be honest having looked at the code I don't really understand what any of it is doing. It doesn't bear any resemblance to Python's runtime semantics for
type.__call__
,__new__
and__init__
. For example the code here chooses between taking a hint from__new__
or__init__
based on which comes earlier in the MRO:mypy/mypy/checkmember.py
Lines 1198 to 1199 in 13f35ad
That makes no sense to me:
__new__
and__init__
do not share method resolution so it does not matter which comes "first" in the MRO:__new__
is always called first and gets to decide whether__init__
is ever called at all.__new__
(leaving aside metaclasses overridingtype.__call__
) and importantly__new__
can return anything.__init__
is irrelevant for determining the type of self: it is only ever called with an instance of the class in which it is defined and will not be called at all if__new__
returns something else.I would expect the inference to work something like this. We have a class:
When we call
A(*args, **kwargs)
it will returnMeta.__call__(A, *args, **kwargs)
so we should determine what type that would return. By defaultMeta
istype
whose call method looks like:The final piece to know is that
object.__new__(A)
returns an object of typeA
. The call toA.__new__(A, ...)
needs further analysis but usually it will eventually come down toobject.__new__(A)
(via calls likeB.__new__(A)
along the way).We see here that the type of the object returned is just the return type of
A.__new__()
. IfA.__new__()
returns an object that is not an instance ofA
thenA.__init__()
will never be called meaning any hints inA.__init__()
are irrelevant. Even ifA.__init__()
is called it does not change the type ofobj
(unless it reassignsobj.__class__
but that is very obscure).NeilGirdhar commentedon Dec 19, 2023
Respecting the metaclass'
__call__
return type is required to type check nnx, which makes extensive use of this feature here.Viicos commentedon Dec 20, 2023
As a workaround, I'm using the
__new__
method instead (see #16020 (comment), where I'm hitting the same use case as you)jorenham commentedon Mar 25, 2024
This currently makes it impossible to correctly type
builtins.reversed
. See python/typeshed#11645 and python/typeshed#11646jorenham commentedon Mar 25, 2024
Like @oscarbenjamin noted, this fails in the contravariant case, i.e. when
__new__
returns an instance of its supertype.But when
__new__
is annotated to return something else entirely, mypy will simply ignore it, even if annotated explicitly:For details, see https://mypy-play.net/?mypy=latest&python=3.12&flags=strict&gist=336602d390b5e6566e9fca93d7fa48a6
builtins.reversed.__new__
python/typeshed#11646Viicos commentedon Mar 25, 2024
The PR fixing this can be found here: #16020
jorenham commentedon Jul 5, 2024
@Viicos #16020 does not fix this issue. In the PR description it explicit states:
jorenham commentedon Jul 5, 2024
The official typing specs now include a section on the
__new__
method:So this confirms that this is, in fact, a bug (and IMO a rather big one).
5 remaining items
numpy.dtypes
numpy/numpy#27008reversed
python/typeshed#13269jorenham commentedon Feb 28, 2025
This is also a blocker for correctly annotating the
numpy.object_
constructor, as well as the constructors of (all) other scalar types in case of "multidimensional" input:See #18343 (comment) for details.
__new__
#1020np.object_
generic for better object array type hinting numpy/numpy#25351logical_{not,and,or,xor}
numpy/numtype#324Avasam commentedon May 25, 2025
In the meantime, could
Incompatible return type for "__new__" (returns "...", but must return a subtype of "...")
and"__new__" must return a class instance (got <some union>)
be made into a separate error code thanmisc
? I can't really ignore them without ignoring the entirety ofmisc
atm.import-untyped
and mostmisc
issues microsoft/python-type-stubs#378