Skip to content

createNodeBuilder, declaration emit, and associated utility port #791

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 34 commits into
base: main
Choose a base branch
from

Conversation

weswigham
Copy link
Member

@weswigham weswigham commented Apr 11, 2025

This branch to replaces the current typeToString implementation with the ported node-builder backed implementation, which is the top-down goal of this PR, ultimately. It also enables declaration emit and tests for it. CI on it should not pass until the node builder is mostly (if not totally) completely ported, due to interdependencies between many of its' subsystems. Baseline diffs are seperately in weswigham#1 because if they were merged into this PR, it would become impossible to view in the github web UI - so we'll only do that once reviewers are done with it.

Structurally, nodebuilder.go is the inner contents of createNodeBuilder's closure environment, lifted into members of a struct. All context parameters have been removed and replaced with lookups of context on the struct itself, since we're OO now, but that's about the only refactor. nodebuilderapi.go is the wrapper returned by createNodeBuilder which maps all the internal closed over methods to the internal node builder API shape (which is recorded as the NodeBuilderInterface... interface)- basically it's the logic that handles setting up context objects for each request. Some of that might get renamed to reduce confusion eventually, but the structure seems sound. nodebuilderscopes.go may or may not go away - NodeBuilder.enterNewScope was pretty big but isolated (used by mapped type and signature construction), so felt like it could stand alone, and it has some utilities only it uses. symbolaccessibility.go is for the checker's symbol accessibility functionality - these are also mostly self-contained, though do depend on one-another and some common utilities (though I only have stubs here right now - my previous attempts to optimize them as I ported them have broken them, so we're just gonna port them straight as we can for now).

This is already a bit of a beast to review, size-wise, and I'd say there's still a fair bit left for a full port - but if we add some extra unit tests, some subsystems, like the specifier generation and maybe accessibility, can reasonably stand alone as changes. Those things just aren't currently unit tested outside of their integrations into the builder in strada, though, so those tests'd all be additional greenfield work.

The remaining features to port (from the TODOs left in the code), which may or may not make this PR or followups depending on reviewer satisfaction, are:

  • Expando function declaration emit (is this fully implemented in checker now?)
  • Late bound index signature declaration emit (checker implementation here is currently partial, and missing the parts required for accurate declaration emit)
  • JS declaration emit support (will need to be substantively rewritten given different upfront parsing and checking of JSDoc structures - likely for the better)
  • isolatedDeclarations support and associated node reuse logic (this is a large amount of error checking code for very little practical payoff)
  • support for attaching type arguments to identifiers in the node builder for quickinfo
  • support attaching synthetic comments to nodes for better truncated output in some modes
  • extract most nodebuilder logic from the checker package into the nodebuilder package with an interface indirection over the checker (likely requires renaming a lot of things on checker to make them "public" and reworking how they're accessed)
  • clean up layering of emitContext/emitResolver/nodebuilderapi layers to automatically pass through more bits so things like EmitContexts don't need to be arguments to functions on the nodebuilderapi
  • bundled emit support? seems mostly cut for js output, but we're the only provider of upfront-bundled declaration output
  • support preserving input quote style in declaration emit (not currently supported on the AST itself)
  • Memoize printers used for diagnostics and node builder in the checker (is this even actually needed? spawning new printers on demand seems like a pretty cheap allocation)
  • Support for the OmitTrailingSemicolon and NeverAsciiEscape printer options (only affects some diagnostic output minorly - also warrants investigating if createSemicolonDeferringWriter actually needs to be ported or if it overlaps with the printer flag of the same)
  • project references support in module specifier generation (currently stubbed, since project references don't exist)
  • cached getOutputPathsFor on the emit host (or decide if this is cheap enough a cache isn't worth bothering with)
  • stripInternal support? Unsure if we planned to support this. It's not bad in the declaration emitter so long as jsdoc tag parsing is in place...
  • Flatten immediately nested module objects? We now parse module a.b {} into module a { export module b {} }, but AFAIK the AST doesn't even support representing the former anymore, which makes printing it back that way for declaration emit difficult!
  • resolutionMode is not currently varied with usage declarations - the helper for calculating it is missing and unused at module lookup sites. This is a more general issue across the compiler presently, but persists into this declaration emit logic.
  • Symlink cache support in module specifier generation to support generating specifiers for modules not directly imported in a project (the loader doesn't keep a symlink cache around anymore as far as I can tell, so that's required before it can be reused in specifier generation)
  • output path remapping in specifier generation for import maps (output path calculation helpers are currently in weird places and need some refactoring to be reused in the modulespecifiers package)
  • underlineing type baseliner to resume measuring node reuse, likely after isolated declarations logic is ported

Sorry, something went wrong.

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham
…ogic excluding ID, JSX, and symbol names
Copy link
Member

@jakebailey jakebailey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll be interested to see if we can move any of this out of the checker package; it's getting pretty big in there.

(I'm not sure how much you want eyes on the contents of the PR yet but I'm happy to look if you do.)

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham
This needs more pulled in from stashed old work to fill it out,
(part of the way through the declaration transform itself and resolver)
and it's useless without the nodebuilder itself done, but has the full
declaration emit pipeline hooked in and tests enabled - another thing
showing just how not complete it is! If the PR was already massive,
this won't help~

For real, this is all going to have to be un-integrated to be at all
mergable in the end, but all up it will be able to signal completion
with green CI in the end, so long as nobody blind baseline-accepts...
¯\_(ツ)_/¯

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham
…atures

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham
…to builder-backed since thats now

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham
…ans type parameter smuggling and module specifier generation

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham
…/stub some missing functionality

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham
…oad printback

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham
…bol printback behavior

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham
…nd host impl

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham
…lity and currently unsupported features

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham
…Host

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham
…of module blocks

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham
…uses

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham
@weswigham weswigham marked this pull request as ready for review April 30, 2025 22:06

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham
… chain logic
@weswigham
Copy link
Member Author

I have weswigham#1 separately open with the baseline changes, because if I merge those into this PR, this change goes from "practically unreviewable" to "actually impossible to navigate in the github web UI", so it's probably best if reviews of this are done before any baseline updates get merged in.

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham
@weswigham
Copy link
Member Author

I'm not sure how much you want eyes on the contents of the PR yet but I'm happy to look if you do

At this point, eyes are welcome, since we could probably merge this any time we wanted given the completeness bar for corsa right now - full stack declaration emit is functional and non-panicing on our test suite, despite some missing features (js support, for example). I've already worked through some of the more egregious bugs and differences from the initial version, too. If you wanted to look at this + baselines, there's weswigham#1, but I can't recommend opening that in the github web UI.


if len(candidateChains) > 0 {
// pick first, shortest
slices.SortStableFunc(candidateChains, ch.compareSymbolChains)
Copy link
Member Author

@weswigham weswigham May 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel obligated to point out that this logic - exhaustively traversing symbol tables looking for a way to name a symbol - is the most expensive part of declaration emit, and here in the go port, it is always worst case performance if anything is exported through a renaming alias and thus dodges the direct lookup fast path - we can't early bail on finding a matching symbol because of the need to sort the results for stability.

So this logic is incredibly slow. You will notice how much slower tests are with this PR (though to some degree that was guaranteed!). This stuff has been a candidate for a rewrite for approximately forever - it dates all the way back to the old string-based declaration emitter logic, and has just grown in weird preference-encoding and edge-case-handling ever since. Making a more efficient equivalent that behaves the same way is almost impossible (I've tried), but the randomly iterated and-then-sorted symbol tables are already potentially going to break strict equivalence with old output (since the symbols we select will first be sorted by declaration location and alphabetically according to the new symbol sort algorithm, and not by creation time). Given that, this whole suite of functions (and wrappers in the node builder) is ripe for a rewrite at some point, while we're here breaking things.

@weswigham weswigham changed the title createNodeBuilder and associated utility port createNodeBuilder, declaration emit, and associated utility port May 1, 2025
@weswigham
Copy link
Member Author

The baselines at weswigham#1 have something like 2026 deleted .types.diff and .js.diff files, so there's certainly some amount of progress here~

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham
…on emit diagnostics

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham

Verified

This commit was signed with the committer’s verified signature.
weswigham Wesley Wigham
…lines aside from diagnostic output)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants