Extending comment scanning to resolve #24

[anthonygam]

Hello @nickng, this project has been very useful for me with formatting my honours thesis, so thank you for maintaining it!

I have been looking into #24. For now, I've just written a unit test reproducing the issue, which fails on my machine. I also tried logging s.String() and it appears to be completely empty.

I would like to try to solve the bug over the next few days. Please let me know if this is welcome, if I've made any mistakes with my test, or if you have any suggestions about how to approach fixing the bug.

[anthonygam]

So far, I have a very simple fix that does not modify the existing grammar. Please let me know if you would prefer a different approach.

[nickng]

I think this looks good! Just a small comment on the variable name.
I start to think whether it's worth support parsing of comments too down the line, so we don't just strip out comments but can also put them back. 🤔

[nickng]

I wonder if it's better to call this "outsideEntry" or something like that to be clear that we're not switching into a different context mode (comment mode is when we're scanning comments, not affected by the grammar), but actually not inside an entry so we can ignore everything.

[anthonygam]

I think that's a great idea, definitely a much clearer name. I've updated it in a new commit.

[anthonygam]

I think this looks good! Just a small comment on the variable name. I start to think whether it's worth support parsing of comments too down the line, so we don't just strip out comments but can also put them back. 🤔

Oh that's an interesting idea! Are you thinking of doing this with @comment{......@ text, or with all the text in between entries? If it's the latter, then maybe this PR would require a refactor right?