Fix findRightmostValidToken

[gabritto]

Turns out, for findPrecedingToken/findRightmostValidToken, we have to look at every token that is not part of the AST and that occurs between "empty" AST nodes.

A simplified illustration of the problem present in the test I added:

variable.|

function foo() { ... }

With position at |, we should return the dot token, because it is the rightmost valid (non-empty) token that precedes the position.
But to find it, we have to look at tokens that occur between the identifier node variable, and the empty identifier node that the parser adds after the dot to construct a valid property access expression.

Before this PR, we were only looking at tokens after the last visited node, i.e. after the empty identifier node.
Now, we look at every token that occurs in between visited nodes, after the rightmost non-empty valid node:

| Rightmost valid node | ... tokens ... | Invalid node 1 | ... more tokens ... | Invalid node n | ... more tokens ...

What this means for the example is that the rightmost non-empty valid node is identifier variable, and to the right of that we have the empty identifier node, and we look in between those two nodes for tokens:

| Identifier 'variable' | ... dot token ... | Identifier '' | ...

[Copilot]

Pull Request Overview

This PR fixes the behavior of findRightmostValidToken by expanding the token search to include tokens between valid and invalid AST nodes.

• In tokens_test.go, the positions slice is extended to account for an optional EOF token.
• In tokens.go, the logic for collecting rightmost visited tokens is refactored to correctly capture trailing tokens between nodes.

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
internal/astnav/tokens_test.go	Adjusted positions slice length to include an extra EOF token.
internal/astnav/tokens.go	Refactored token collection using rightmostVisitedNodes for clarity and correctness.

Comments suppressed due to low confidence (2)

internal/astnav/tokens_test.go:86

• Consider adding test cases that explicitly verify the handling of the extra EOF token when includeEOF is true to ensure that the positions slice indexing is correct.

positions := make([]int, len(fileText)+core.IfElse(includeEOF, 1, 0))

internal/astnav/tokens.go:476

• It would be beneficial to add tests covering scenarios with multiple invalid nodes between valid tokens so that the behavior of rightmostVisitedNodes and trailing token collection is rigorously validated.

for i := validIndex + 1; i < index; i++ {

[gabritto]

Drive-by fix, I forgot to do this in my previous PR.

[jakebailey]

Is this difference just a problem with the ported code, or was this some bug in the old code too? I assume the former?

[gabritto]

Is this difference just a problem with the ported code, or was this some bug in the old code too? I assume the former?

Yep, a problem with the ported code. The old code doesn't have this problem because it uses node.getChildren(), which automatically includes tokens that are not part of the AST, so it can just iterate through children that way.

The added complexity of the new implementation is because it has to simulate the old findPrecedingToken while also doing the extra work of considering tokens that are not part of the AST, since we are avoiding a port of node.getChildren() for now, because it was inefficient.