Skip to content

Compiler loses location information before calling macros (sometimes) #43081

Closed
@alexcrichton

Description

@alexcrichton
Member

Updated description

First implemented in #43230 the compiler can now tokenize a few AST nodes when necessary losslessly from the original token stream. Currently, however, this is somewhat buggy:

  • It only happens for items in certain situations (aka no inner attributes). The consequence of this is that more and more attributed items will have invalid span information when sent to procedural macros.
  • We don't invalidate the cache when later changing the item. The consequence of this is that procedural macros will receive a buggy view of what the AST node actually represents.

The "real bug" here is that we haven't actually implemented converting an AST to a token tree. The initial implementation in #43230 was effectively just a heuristic, and not even a great one! As a result we still need the ability basically to losslessly tokenize an AST node back to its original set of tokens.

Some bugs that arise from this are:

Original Description

There's an associated FIXME in the code right now, and to fix this we'll need to implement tokenization of an AST node. Right now the thinking of how to implement this is to save all TokenStream instances adjacent to an AST node, and use that instead of converting back into a token stream

cc @dtolnay, @nrc, @jseyfried

Activity

added a commit that references this issue on Jul 28, 2017

Auto merge of #43230 - alexcrichton:more-tokenstream, r=nrc,jseyfried

253 remaining items

Aaron1011

Aaron1011 commented on Jan 19, 2021

@Aaron1011
Member

Progress update: The pretty-print/retokenize check was removed in #79472. We now only lose spans when we explicitly remove the original TokenStream. This happens in two cases:

  1. The input to a derive proc-macro has cfg or cfg_attr attributes (except when a #[cfg] attribute is kept due to its predicate evaluating to true)
  2. Inner attributes are present

PR #80689 completely resolves this issue. However, it is quite large, so I'm in the process of splitting it into independent pieces for easier review.

Aaron1011

Aaron1011 commented on Apr 11, 2021

@Aaron1011
Member

PR #82608 has been merged! We now preserve spans for all stable uses of macros (e.g. custom derives), and nearly all unstable usages (e.g. inner attrs).

If anyone encounters a loss of location information, please open a new issue for the specific problem you're having.

added a commit that references this issue on Feb 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-decl-macros-1-2Area: Declarative macros 1.2A-decl-macros-2-0Area: Declarative macros 2.0 (#39412)A-diagnosticsArea: Messages for errors, warnings, and lintsC-bugCategory: This is a bug.P-mediumMedium priorityT-compilerRelevant to the compiler team, which will review and decide on the PR/issue.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @alexcrichton@eddyb@nikomatsakis@shepmaster@korken89

        Issue actions

          Compiler loses location information before calling macros (sometimes) · Issue #43081 · rust-lang/rust