Description
Updated description
First implemented in #43230 the compiler can now tokenize a few AST nodes when necessary losslessly from the original token stream. Currently, however, this is somewhat buggy:
- It only happens for items in certain situations (aka no inner attributes). The consequence of this is that more and more attributed items will have invalid span information when sent to procedural macros.
- We don't invalidate the cache when later changing the item. The consequence of this is that procedural macros will receive a buggy view of what the AST node actually represents.
The "real bug" here is that we haven't actually implemented converting an AST to a token tree. The initial implementation in #43230 was effectively just a heuristic, and not even a great one! As a result we still need the ability basically to losslessly tokenize an AST node back to its original set of tokens.
Some bugs that arise from this are:
- Usage of
#[cfg]
modifies the AST but doesn't invalidate the cache -- Macros 2.0: #[cfg_attr] makes .to_string() and TokenStream disagree #48644 - Modules with attributes disagree on what their internal tokens are -- Inconsistency between Display and IntoIterator for a TokenStream containing a module #47627
macro_rules!
and procedural macros don't play well together -- Inconsistent tokenization of procedural macros with macro_rules #49846Invoking a macro with brackets loses span information -- Procedural macros stringify inputs with macro invocations too aggressively #50840
Original Description
There's an associated FIXME in the code right now, and to fix this we'll need to implement tokenization of an AST node. Right now the thinking of how to implement this is to save all TokenStream
instances adjacent to an AST node, and use that instead of converting back into a token stream
cc @dtolnay, @nrc, @jseyfried
Activity
proc_macro: Use an item's tokens if available
Auto merge of #43230 - alexcrichton:more-tokenstream, r=nrc,jseyfried
253 remaining items
Aaron1011 commentedon Jan 19, 2021
Progress update: The pretty-print/retokenize check was removed in #79472. We now only lose spans when we explicitly remove the original
TokenStream
. This happens in two cases:cfg
orcfg_attr
attributes (except when a#[cfg]
attribute is kept due to its predicate evaluating totrue
)PR #80689 completely resolves this issue. However, it is quite large, so I'm in the process of splitting it into independent pieces for easier review.
Aaron1011 commentedon Apr 11, 2021
PR #82608 has been merged! We now preserve spans for all stable uses of macros (e.g. custom derives), and nearly all unstable usages (e.g. inner attrs).
If anyone encounters a loss of location information, please open a new issue for the specific problem you're having.
join_spans
dtolnay/quote#222Merge #537