[syntax-errors] Tuple unpacking in return and yield before Python 3.8

[ntBre]

Summary

Checks for tuple unpacking in return and yield statements before Python 3.8, as described here.

Test Plan

Inline tests.

[github-actions]

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.

[MichaReiser]

Python also knows yield expressions. Is this something we have to distinguish / consider in this PR?

[ntBre]

I'll have to test this out. The CPython PR said statements, but maybe I took it too literally.

[ntBre]

It looks like this also applies to expressions:

Python 3.7:

>>> def i(): yield 1, (yield 2, *rest), 3
  File "<stdin>", line 1
    def i(): yield 1, (yield 2, *rest), 3
                                ^
SyntaxError: invalid syntax

[MichaReiser]

Does this also handle nested yield expressions a = yield x
Could this code be moved into parse_yield_expression?

[dhruvmanila]

Can we add test cases using single unpack element like return *rest or yield *rest?

[dhruvmanila]

What about yield 1, (*rest), yield 1, (yield 2, *rest), 3 (yield can be used as an expression)? I'm just throwing out random examples :)

[ntBre]

Thanks for these! The single cases are actually still invalid on my system Python (3.13):

>>> def foo(): return *rest
  File "<python-input-0>", line 1
    def foo(): return *rest
                      ^^^^^
SyntaxError: can't use starred expression here
>>> def foo(): yield *rest
  File "<python-input-1>", line 1
    def foo(): yield *rest
                     ^^^^^
SyntaxError: can't use starred expression here

But these aren't flagged as ParseErrors by ruff.

The second case is very interesting. It's actually accepted on both 3.7 and 3.8, but not on my 3.13 system Python (SyntaxError: cannot use starred expression here). That might be another change not currently tracked in #6591.

I added your third case as an example of a yield expression and moved the check into parse_yield_expression as @MichaReiser suggested.

[dhruvmanila]

But these aren't flagged as ParseErrors by ruff.

That's because the grammar says it's allowed.

return_stmt:
    | 'return' [star_expressions] 

star_expressions:
    | star_expression (',' star_expression )+ [','] 
    | star_expression ',' 
    | star_expression

star_expression:
    | '*' bitwise_or 
    | expression

I think it's caught by the CPython compiler and not the parser. You can see that python -m ast test.py wouldn't raise a syntax error here.

$ python -m ast _.py       
Module(
   body=[
      FunctionDef(
         name='foo',
         args=arguments(
            posonlyargs=[],
            args=[
               arg(arg='args')],
            kwonlyargs=[],
            kw_defaults=[],
            defaults=[]),
         body=[
            Return(
               value=Starred(
                  value=Name(id='args', ctx=Load()),
                  ctx=Load()))],
         decorator_list=[],
         type_params=[])],
   type_ignores=[])

[dhruvmanila]

Oops, missed the issue #16520

[ntBre]

I added more test cases (thanks for the suggestions!). Two of them became TODOs because I think they aren't exactly related to this PR:

• return|yield *rest is invalid on every Python version I tried and should be flagged as a parse error, in my opinion
• return|yield 1, (*rest) is valid on 3.7 and 3.8 but not 3.13, so I think this is a separate change from the 3.8 unpacking (and a second Change::Removed case to use the enum from the other PR)

I also checked on yield from and it looks like our parse_yield_from method already reports an error for any unparenthesized tuple expression, so I think that should be covered.

Finally, I changed the message for the yield statement to yield expression. I think in an ideal case we'd have a third variant of StarTupleKind and differentiate YieldExpr from YieldStmt but I didn't see an easy way to do that with the new detection in parse_yield_expression. Since I had to pick one, I thought "expression" was slightly preferable.

[MichaReiser]

return|yield *rest is invalid on every Python version I tried and should be flagged as a parse error, in my opinion

Yes, that seems correct. Can you create an issue for it?