Use get_type_hints() to resolve type annotations that are encoded as strings

[tmke8]

Closes #303

When using from __future__ import annotations (which will become default behavior in Python 3.10), type annotations are stored as strings. The standard library provides typing.get_type_hints() to convert these strings to types.

For the tests, I simulated this behavior by wrapping the type annotations in quotes:

@dataclass
class Foo:
    a: "int"

This has the same effect and also works on Python 3.6. (The future import was introduced in Python 3.7.)

I was a bit unsure about the tests. Let me know if I can improve them.

[tmke8]

I don't think the pre-commit.ci failure is related?

[omry]

Looks good.

1. Please add a news fragment in the news directory (similar to what is described here).
2. pre-commit.ci error is unrelated, don't worry about it.
3. Can you point me to some docs saying using forward (quoted) types is simulating the issue?

[tmke8]

1. Done.
2. 👍
3. Quoted types might not be 100% identical to postponed annotations. This is what I found:

The relevant PEP says:

This PEP proposes changing function annotations and variable annotations so that they are no longer evaluated at function definition time. Instead, they are preserved in __annotations__ in string form.

and

This PEP is meant to solve the problem of forward references in type annotations.

So, postponed annotations certainly remove the need for quoted forward references. The PEP also makes clear that using get_type_hints() solves the problem:

For code that uses type hints, the typing.get_type_hints(obj, globalns=None, localns=None) function correctly evaluates expressions back from its string form. Note that all valid code currently using __annotations__ should already be doing that since a type annotation can be expressed as a string literal.

Additionally, the mypy documentation states:

If you are running Python 3.7+ you can use from __future__ import annotations as a (nicer) alternative to string quotes, read more in PEP 563.

If this doesn't convince you, I would need to add a test that's only run on Python 3.7+ (where the future import is available). Tell me what you think!

[omry]

I am not sure exactly how from __future__ import annotations works to enable future compatibility.
does it have side effects or do you need to use it directly for the import to have the effect?

One possibility is to have testing that depends on the Python version, but in the presence of side effects from that import (as I suspect is the case), I am not sure that this is a good idea.

Related:
What would be the implications of from __future__ import annotations on the execution path of OmegaConf import?
Will there be behavior changing side effects beyond the scope of OmegaConf itself (that may cause issues to user's code?)

[tmke8]

In the announcement of Python 3.7 (where the future import was introduced), it says:

Instead of compiling code which executes expressions in annotations at their definition time, the compiler stores the annotation in a string form equivalent to the AST of the expression in question. If needed, annotations can be resolved at runtime using typing.get_type_hints(). In the common case where this is not required, the annotations are cheaper to store (since short strings are interned by the interpreter) and make startup time faster.

So this does seem to affect some deeper level of the compiler.

---

Will there be behavior changing side effects beyond the scope of OmegaConf itself (that may cause issues to user's code?)

I don't think so. In my experience, it seems to work on a per-module level. Meaning that it doesn't matter if you import a future activated module from one that is not activated, or vice-versa. I think this is also how the from __future__ import print_function worked in Python 2 (which affects the parser).

If you have two modules a.py and b.py and they look like this:

a.py

def f(a: unknown_name):
    pass

b.py

from __future__ import annotations
from a import f

and you run b.py, it throws an error:

Traceback (most recent call last):
  File "b.py", line 2, in <module>
    from a import f
  File "/Users/tk324/dev/python/a.py", line 3, in <module>
    def f(a: unknown_name):
NameError: name 'unknown_name' is not defined

but if you change a.py to be

from __future__ import annotations
def f(a: unknown_name):
    pass

it works. So the effect of the future import seems to be locally restricted.

I really tried finding more detailed documentation for this, but wasn't much I could find.

In general though, I think OmegaConf should use typing.get_type_hints() in any case. Even if it doesn't solve #303 .

[omry]

Cool, thanks for testing.
This opens up the option of creating a specific test file module to test the behavior with future annotations directly instead of (or in addition to) simulating it the way you did.

Can you add such a test? you can condition it con Python 3.7+ before the import.

[tmke8]

I changed the test. Is this what you had in mind? A major difficulty is that from __future__ import annotations has to be the first code in a module, otherwise it throws a syntax error.

[omry]

Awesome.
One nit with the news file.

[tmke8]

Done!

[omry]

Awesome, thanks!