spike: e2e tests with openapi client

[hashicc]

Description

Spike on using a generated openapi client

• Source definition of api fixture wrapping generated api clients
• Example of api fixture usage
• Script to generate api client

Notes

• Based generation using package and techniques used elsewhere in hashicorp
• Package to generate openapi spec requires java
• Playwright "just works" with typescript so generating typescript-based open api client makes sense
• Using an api fixture allows centralized setup for all the clients with common configuration
• attributes in payloads are untyped and would need to rely on documentation and lookup
• Generated client has decent error handling and the generated code while a bit verbose isn't that complicated and supports a good set of default configuration
• Generated client uses the openapi specs and can be re-generated on demand. It currently points at HEAD of the main branch on the boundary repo.
• ❓ Should the openapi clients be used directly from the fixture or should there be a wrapper to provide niceties like specifying a name automatically with Target-${nanoid()} or arguments for untyped attributes (see above)
  • A fixture could be created that extends and is based on the api fixture, but the api fixture could be the first step
• ❓ Should the openapi generated client be checked into source control?
  • I think so, since the openapi schema lives outside the repo and has to be fetched
• ❓ Is there a preference for how resources are created? cli, rest api, ui (ui clicks, Page Objects)
  • Right now we're using a combination of each and have some overlap. I think when it's the thing being tested (the "subject") it should use the UI to create the resource where possible. Otherwise, I think using a rest api first makes sense since it's less layers of abstraction and might have better error messages, although I'm not sure. Most of our UIs will end up using the rest api so it might make sense to prefer this method. That leaves the cli as a method of last resort.
• ❓ Right now the tooling is set to pull the swagger/openapi docs from head of the main branch of the boundary repo. Is there any concern of this during active development? (eg: work against changes on a branch). The tooling could be take a commit but that would still rely on that commit being available on github.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
boundary-ui	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 4, 2025 6:30pm
boundary-ui-desktop	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 4, 2025 6:30pm

[hashicc]

Can add any of the generated clients here as needed

[hashicc]

Need to look into the generator to see if the targetService prefix can be removed because it's redundant and noisy, it'd be nicer if this read as boundaryApi.target.createTarget

[hashicc]

This is based on the spec "operationId": "TargetService_CreateTarget", so it might be best to keep the verbosity to have it match more clearly the open api spec

[ZedLi]

❓ Should the openapi clients be used directly from the fixture or should there be a wrapper to provide niceties like specifying a name automatically with Target-${nanoid()} or arguments for untyped attributes (see above)

When you say from the fixture do you mean in the test directly or in the fixture which is already a layer abstracted from the test? I think from the POV of writing the actual tests themselves, we could totally have wrappers to at least provide some convenience since I imagine there could be a decent amount of boiler-plate otherwise, though it might reduce the convenience of having auto generated clients if we do that for everything (besides having the types).

❓ Should the openapi generated client be checked into source control?

I think so, I would imagine even if the schema was checked in here and didn't need to be fetched, I would still check in the client so that way the history is clear and to know what tests were running against.

❓ Is there a preference for how resources are created? cli, rest api, ui (ui clicks, Page Objects)

Yea, I agree with what your line of thinking as well and is similar to what I was thinking here in this PR #2594. Unless it's under test, we most likely we want to stick with using the HTTP client rather than using the CLI except in scenarios where we need it.

❓ Right now the tooling is set to pull the swagger/openapi docs from head of the main branch of the boundary repo. Is there any concern of this during active development? (eg: work against changes on a branch). The tooling could be take a commit but that would still rely on that commit being available on github.

I think this is fine as we're also checking in the client itself so we'd be aware of any changes that are being made to the clients.

[hashicc]

@ZedLi

When you say from the fixture do you mean in the test directly or in the fixture which is already a layer abstracted from the test? I think from the POV of writing the actual tests themselves, we could totally have wrappers to at least provide some convenience since I imagine there could be a decent amount of boiler-plate otherwise

That's right, an additional fixture that is based on the api fixture (using the word fixture still feels odd to me but its what playwright calls them). Thinking ahead one step it would be really nice for these domain wrapper fixtures to be class based so that they could also easily track and teardown what they create likely automatically. A target fixture based on an api fixture could look like:

class TargetSandbox {
    created = new Set();

    constructor(private api) {}
    // ... nice things like filling in `default_port`
    create(args: Args & {port?: number}) {
        const result = this.api.createTarget({
            ...args,
            attributes: {
                args.port
            }
        })

        this.created.add(result);
        return result;
    }

    delete(id) {
        const result = this.api.deleteTarget(id);
        if (this.created.has(id)) {
            this.created.remove(id);
        }

        return result;
    }

    destroy() {
        for (const item in this.created) {
            try {
                this.delete(item.id);
            } catch (e) {
                console.error(`Unable to cleanup item`)
            }
        }
    }
}

export const test = base.extend({
    targetOptions: [{ shouldDestroy: true }, { option: true }],

    target: async ({ api, targetOptions }, use) => {
        const target = new TargetSandbox(api);
        await use(target);

        if (!target.isDestroyed && targetOptions.shouldDestroy) {
            target.destroy();
        }
    }
});

---

Yea, I agree with what your line of thinking as well and is similar to what I was thinking here in this PR #2594. Unless it's under test, we most likely we want to stick with using the HTTP client rather than using the CLI except in scenarios where we need it.

💯 on everything you mentioned in that pull request

[ZedLi]

Very cool POC!! 🚀

[ZedLi]

I have two questions regarding this file:

1. How come you had to use commonJS instead of just ESM for this file?
2. Could we have just had this entirely in the package.json script? It doesn't look it needs to do much besides call the openapi generator cli? Or do you forsee more we'll need to add to the script?

[hashicc]

I started off with just a yarn run and then moved it to a script. We could return it to that, I think that could make sense. I ended up copying the script from another repo to try and have a common approach as them. That's why it ended up being cjs for the spike, I didn't really want to refactor it, you're right it could totally be esm. I trimmed the script for parts we didn't need which ended up making it as simple as package.json script.

TLDR: It could totally be a package.json script, and we could return to a script later if needed

[ZedLi]

Could you have just used node for this? fs.rmSync(apiClientsDirectory, { recursive: true, force: true })

[hashicc]

Definitely, that's a lot clearer

[ZedLi]

We likely want to move these out of the globalSetup as these are really just shared fixtures at this point

[hashicc]

💯