token module documentation is incomplete

[encukou]

Documentation

The token docs use a generated list of tokens, with short blurbs for the OP tokens but nothing for the other (more interesting) ones.
Some tokens (COMMENT, NL, etc.) have secondary, hand-written definitions. However, these are excluded from the index; links go to the empty entries.

I propose adding hand-written prose for the non-OP tokens, and changing the automation to check the entries rather than generate them.
For the OP tokens, generating the docs is the correct thing to do, but the list can be more compact since they're all the same.

Linked PRs

• gh-130587: Add hand-written docs for non-OP tokens #130588
• [3.13] gh-130587: Add hand-written docs for non-OP tokens (GH-130588) #131465
• gh-130587: Invoke regen-token rst with rstfile as an argument #132304
• [3.13] gh-130587: Invoke regen-token rst with rstfile as an argument (GH-132304) #132327

[StanFromIreland]

Also another issue in the file:

The '/' are not escaped.

Edit: I must have put this under the wrong issue! Apologies

[befeleme]

In this issue, the make_rst function was extended to accept another argument, rstfile. Makefile.pre.in was not updated accordingly, causing the regen directive to fail in our Fedora environment where we can't deal with relative paths (as the default value of rstfile is defined).