Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(#330): Adds e2e tests for default geopoint question type #313

Open
wants to merge 44 commits into
base: main
Choose a base branch
from

Conversation

latin-panda
Copy link
Collaborator

@latin-panda latin-panda commented Feb 25, 2025

Closes #330
Part of #249

I have verified this PR works in these browsers (latest versions):

  • Chrome
  • Firefox
  • Safari (macOS)
  • Safari (iOS)
  • Not applicable

What else has been done to verify that this works as intended?

Running tests locally and in CI

Why is this the best possible solution? Were any other approaches considered?

Initially, I tried adding tests in Vue, but our Web Form client is primarily a presentation layer, and Vue doesn't support tests requiring browser permissions. Therefore, I chose Playwright for automation (already a project dependency), following a standard folder structure and the use of page objects, as I’ve done with other test frameworks (Protractor, WebDriverIO). This is compatible with most frameworks if we ever switch in the future. Additionally, I included a minimal README file to guide contributors in adding new tests.

How does this change affect users? Describe intentional changes to behavior and behavior that could have accidentally been affected by code changes. In other words, what are the regression risks?

N/A

Do we need any specific form for testing your changes? If so, please attach one.

N/A

What's changed

This is not perfect, and more can be improved, but it is a good start that we can iterate as we add more tests.

  • Adds minimal README file to guide contributors when adding new tests
  • Adds a new folder structure
  • Moves the content of wf-preview.test.ts and vue.test.ts to /e2e/specs/build/preview-form-render.test.ts. The original name wasn't descriptive.
    • We can consider moving the /e2e/specs/build/preview-form-render.test.ts to /e2e/specs/preview-form-render.test.ts
  • Adds new basic tests for geopoint and note
  • Adds new page objects for form-page, preview-page and geopoint-control

Copy link

changeset-bot bot commented Feb 25, 2025

⚠️ No Changeset found

Latest commit: 459f2d7

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@latin-panda latin-panda changed the title chore(#249): adds e2e tests for default geopoint question type chore(#249): Adds e2e tests for default geopoint question type Feb 25, 2025

```
e2e/
├── fixtures/ # Sample forms and data for testing.
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At the moment, this folder is empty, but next time, it'd be good to take the XML forms from here. I added a ToDo comment for this improvement in the PreviewPage.ts

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We moved the fixtures into @getodk/common because they're shared across:

  • @getodk/scenario tests
  • @getodk/web-forms:
    • E2E tests
    • Vitest tests
    • Dev/demo
    • Web Forms Preview

In theory they could also be used in @getodk/xpath and @getodk/xforms-engine as well. We just haven't added tests that would take advantage of that, since this shared solution was established. A fair bit of effort has gone into making sure that sharing works for a wide variety of conditions (including not just form fixtures themselves, but external resources they reference by jr: URL) and use cases (including common structural organization, and associating XForms XML fixtures with their source XLSForms).

All of that context may not be obvious without elaborating on it, but I want to make sure it's clear before asking: what would be the advantage of bypassing the shared fixture solution here? Is there something the E2E use case needs that isn't already solved? I'd prefer to try to address it in the shared context first if that's the case, so we can continue to benefit from solutions being broadly available anywhere fixtures are referenced. If that doesn't seem reasonable or tenable I'm open to reconsidering! But I definitely want to understand the intent before we start splitting up something that's working well as a project-wide resource for everything else.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the context!

I'm still unsure if the fixtures in @getodk/common will fully meet the E2E testing needs, but if a special case arises, e.g., verify UX based on special appearance that modifies the display, I don’t think there will be many, and we can add it to@getodk/common.

This was intended to keep it contained from other types of testing, but I don't have a strong preference. This approach has existed for a while without producing any issues with other test types. I'm open to keeping it that way.

Do you agree to remove the fixture from these e2e tests and keep using the one in @getodk/common?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file name is confusing. I moved the test to /e2e/specs/build/preview-form-render.test.ts

We can move the /e2e/specs/build/preview-form-render.test.ts to /e2e/specs/preview-form-render.test.ts

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file now contains the tests of vue.test.ts and wf-preview.test.ts since both verify the demo forms.
I didn't change the tests; I just added the PreviewPage to navigate to the demo page. Everything else is as original

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could move this file out from the build folder, from e2e/specs/build/style.test.ts to e2e/specs/style.test.ts

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same, this file name is not descriptive, for now I moved the content to e2e/specs/build/preview-form-render.test.ts

@latin-panda latin-panda marked this pull request as ready for review February 25, 2025 22:09
@latin-panda
Copy link
Collaborator Author

Hi @eyelidlessness, can you please review? Thank you for your time!

Quick question, we don't need a changeset for this PR, or do we?

@latin-panda latin-panda force-pushed the e2e-default-geopoint-question-type branch from 5b1547f to 825318e Compare February 27, 2025 04:39
Copy link
Member

@eyelidlessness eyelidlessness left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lots of feedback, I know! But I don't want that to detract from saying...

THIS IS AWESOME! Thank you so much for taking this on.


```
e2e/
├── fixtures/ # Sample forms and data for testing.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We moved the fixtures into @getodk/common because they're shared across:

  • @getodk/scenario tests
  • @getodk/web-forms:
    • E2E tests
    • Vitest tests
    • Dev/demo
    • Web Forms Preview

In theory they could also be used in @getodk/xpath and @getodk/xforms-engine as well. We just haven't added tests that would take advantage of that, since this shared solution was established. A fair bit of effort has gone into making sure that sharing works for a wide variety of conditions (including not just form fixtures themselves, but external resources they reference by jr: URL) and use cases (including common structural organization, and associating XForms XML fixtures with their source XLSForms).

All of that context may not be obvious without elaborating on it, but I want to make sure it's clear before asking: what would be the advantage of bypassing the shared fixture solution here? Is there something the E2E use case needs that isn't already solved? I'd prefer to try to address it in the shared context first if that's the case, so we can continue to benefit from solutions being broadly available anywhere fixtures are referenced. If that doesn't seem reasonable or tenable I'm open to reconsidering! But I definitely want to understand the intent before we start splitting up something that's working well as a project-wide resource for everything else.

};

test.describe('Preview Form Render', () => {
test('demo forms load', async ({ context, page }) => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The origin of this test was #265. I probably could have written up more detail about why that PR was created. It was a bit of a rush to fix a regression in the Web Forms Preview, much like #320 is.

The original name and file system structure was descriptive, though you may need additional context for that to be clear! The test is exercising:

  • the build (hence its presence in a build subdirectory)
  • the Web Forms Preview build product, specifically

I also think it's important to note that up to now, Web Forms Preview is currently our only production deployment. It's also our most immediate, concrete way to show users the state of the Web Forms project. (For additional context: this is also why I dropped everything, even extended my work day yesterday, to prioritize #320!)

It's good that the build aspect is still captured in the filesystem structure. I would like to restore both:

  • Specific mention of Web Forms Preview in the test description
  • A distinct, clearly named module/suite for tests exercising Web Forms Preview specifically

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure!

  • The file will remain in the build directory of e2e tests: packages/web-forms/e2e/specs/build/
  • Changing the test description to test.describe('Web Forms Preview Demo Forms', () => {
  • Changing the test filename to web-forms-preview-forms.ts or preview-demo-forms.ts

Did I capture correctly?

Comment on lines 42 to 129
test('all forms are rendered and there is no console error', async ({ page, browserName }) => {
let consoleErrors = 0;
page.on('console', (msg) => {
if (msg.type() === 'error' || msg.type() === 'warning') {
consoleErrors++;
}
});

const previewPage = new PreviewPage(page);
await previewPage.goToPage();

// this ensures that Vue application is loaded before proceeding forward.
await expect(page.getByText('Demo Forms')).toBeVisible();

// Let's expand all categories, so that Form list is visible
await expandAllCategories(page);

const forms = await page.locator('ul.form-list li a').all();

expect(forms.length).toBeGreaterThan(0);

for (const form of forms) {
await form.click();

// Traverse the form element by element
// if focused element is an editable textbox then fill it
// Exit the loop when focus is on the Send button
while (true) {
const onSendButton = await page.evaluate(() => {
const activeElement = document.activeElement;
return activeElement?.tagName === 'BUTTON' && activeElement.textContent === 'Send';
});

if (onSendButton) {
break;
}

await page.keyboard.press(browserName == 'webkit' ? 'Alt+Tab' : 'Tab');

const inputType = await page.evaluate(() => {
const isInputElement = (
activeElement: Element | null
): activeElement is HTMLInputElement => {
return activeElement?.tagName === 'INPUT';
};

const activeElement = document.activeElement;

if (
!isInputElement(activeElement) ||
activeElement.hasAttribute('readonly') ||
activeElement.hasAttribute('disabled')
) {
return null;
}

return activeElement.type;
});

if (inputType === 'text') {
await page.keyboard.type(faker.internet.displayName());
}
// Select the next option, if the last option is selected by default
// then browser selects the first one.
else if (inputType === 'radio') {
await page.keyboard.press('ArrowDown');
}
// Tab behaviour for checkboxes is different, each Tab press moves the focus
// to the next option. Here we are toggling every checkbox option.
else if (inputType === 'checkbox') {
// Toggle the option
await page.keyboard.press('Space');
}
}

const langChanger = page.locator('.larger-screens .language-changer');

if ((await langChanger.count()) > 0) {
await langChanger.click();
await page.locator('.language-dd-label').last().click();
}

await page.goBack();
await expandAllCategories(page);
}

expect(consoleErrors).toBe(0);
});
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I mentioned this in chat... I'm not sure how much value this test (originally from vue.test.ts) provides. In its current form, I think it's probably at least as much of a liability.1

I agree the original module/suite name wasn't descriptive. I'm not sure the test description (copied here unmodified I believe) is very helpful either. In my opinion, pretty much everything about this test screams: if there's anything worth keeping in place of this test, it might be a TODO. (And in a module/suite separate from the more clearly valuable test of the Web Forms Preview build product.)

Maybe something like this?

Suggested change
test('all forms are rendered and there is no console error', async ({ page, browserName }) => {
let consoleErrors = 0;
page.on('console', (msg) => {
if (msg.type() === 'error' || msg.type() === 'warning') {
consoleErrors++;
}
});
const previewPage = new PreviewPage(page);
await previewPage.goToPage();
// this ensures that Vue application is loaded before proceeding forward.
await expect(page.getByText('Demo Forms')).toBeVisible();
// Let's expand all categories, so that Form list is visible
await expandAllCategories(page);
const forms = await page.locator('ul.form-list li a').all();
expect(forms.length).toBeGreaterThan(0);
for (const form of forms) {
await form.click();
// Traverse the form element by element
// if focused element is an editable textbox then fill it
// Exit the loop when focus is on the Send button
while (true) {
const onSendButton = await page.evaluate(() => {
const activeElement = document.activeElement;
return activeElement?.tagName === 'BUTTON' && activeElement.textContent === 'Send';
});
if (onSendButton) {
break;
}
await page.keyboard.press(browserName == 'webkit' ? 'Alt+Tab' : 'Tab');
const inputType = await page.evaluate(() => {
const isInputElement = (
activeElement: Element | null
): activeElement is HTMLInputElement => {
return activeElement?.tagName === 'INPUT';
};
const activeElement = document.activeElement;
if (
!isInputElement(activeElement) ||
activeElement.hasAttribute('readonly') ||
activeElement.hasAttribute('disabled')
) {
return null;
}
return activeElement.type;
});
if (inputType === 'text') {
await page.keyboard.type(faker.internet.displayName());
}
// Select the next option, if the last option is selected by default
// then browser selects the first one.
else if (inputType === 'radio') {
await page.keyboard.press('ArrowDown');
}
// Tab behaviour for checkboxes is different, each Tab press moves the focus
// to the next option. Here we are toggling every checkbox option.
else if (inputType === 'checkbox') {
// Toggle the option
await page.keyboard.press('Space');
}
}
const langChanger = page.locator('.larger-screens .language-changer');
if ((await langChanger.count()) > 0) {
await langChanger.click();
await page.locator('.language-dd-label').last().click();
}
await page.goBack();
await expandAllCategories(page);
}
expect(consoleErrors).toBe(0);
});
test.skip('All forms are rendered without producing unexpected errors', () => {
/**
* There was previously a test with a similar description which:
*
* 1. Loads the dev/demo fixtures list
* 2. Iterates through each fixture, and for each:
* 1. Loads that fixture in dev/demo mode
* 2. [Attempts to] progress through all of the form's controls, and for
* each:
* - If it is a text field: fill a value
* - If it is a field with options: select each of its options
* - [The test was not maintained, so no other controls/interaction
* methods have been exercised]
* 3. Attempts to invoke the dev/demo form submission behavior
* 4. Returns to the dev/demo fixtures list for the next iteration
*
* Currently we see little (if any) value in this test _as it was written_. We
* may revisit it in the future. **If** (and insofar as) there would be value
* in such a generalized testing approach, revisiting it would likely involve:
*
* - Splitting out individual tests per fixture, which might utilize:
* - [Playwright Fixtures](https://playwright.dev/docs/api/class-fixtures)
* (note: despite the name, this is a different concept from what we call
* fixtures!)
* - Code generation
* - [Some other, hopefully maintainable, way to apply test logic across a
* frequently changing set of fixtures with a wide variety of use cases]
*
* - A more thoughtful approach to covering the ever-growing set of form
* controls and interaction modes they provide/require
*
* - A more thoughtful approach to checking for errors, including at least:
* - A more intentional way to identify whether errors occur
* - A more intentional way to identify whether errors that do occur are
* expected (i.e. messaging to user) or not (i.e. uncaught or unhandled
* in myriad ways)
* - Some way of designating fixtures which _are expected_ to produce
errors…
* - on load
* - due to a specific interaction
*
* - Possibly some set of additional assertions and/or test cases which would
* apply globally to any form (or any clear subset of forms)
*/
});

Footnotes

  1. And I think the amount of work that's already gone into keeping it wobbling along is a testament to that!

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eyelidlessness, I would prefer to delete this test, create a new ticket with this detail, and link this PR. So it can be assigned as a proper task.

Would that be okay?

├── GeopointComponent.ts # Example of a reusable control for the geopoint question type.
├── pages/ # Full page representations.
├── FormPage.ts # Example of a full page representation for a form.
├── specs/ # Test specification files.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We generally haven't used the "specs" naming convention for tests, throughout the WF project. We have either:

  • test (in all other packages)
  • tests (Vitest tests in @getodk/web-forms)1

As I understand it, the "spec" terminology comes from the Behavior Driven Development philosophy: tests act as a specification of behavior for the functionality under test. I like a lot about the BDD philosophy, especially its approach to describing tests. In a vacuum, I even like the "spec" concept/naming convention!

But it's potentially a major source of confusion where we as a team already refer to several things as "specs":

  • Actual specifications named as such—e.g. ODK XForms Specification (and the underlying W3C XForms, XML, XPath specifications)
  • Documentation—e.g. of XLSForms, Collect, etc
  • In some cases, observable behavior—e.g. aligning Web Forms behavior to match what we know of the equivalent behavior in Collect/JavaRosa has at times been called "the spec"

I originally started writing this comment as a "nit", but honestly as I've written this out I gotta be honest: I find the term is already way overloaded, which has already caused a fair bit of confusion. I'm really reluctant to overload it more!

Footnotes

  1. IIRC this was inherited from a Vue project template. In hindsight, I wish we'd made it consistent with the other packages then! I'd prefer to do so soon/now, if we agree!

Copy link
Collaborator Author

@latin-panda latin-panda Mar 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was actually expecting to find the e2e tests inside the /test folder, any type of test actually, for example:

|─ tests/
|─|─ e2e/
|─|─ |─ page-objects/
|─|─ |─ specs/
|─|─ unit/
|─|─|─ components/
|─|─ integration etc....

Would it be possible for us to do this? Starting with Web Forms client. What do you think?


But it's potentially a major source of confusion where we as a team already refer to several things as "specs"

Good point! What do you think about changing specs to any of these options:

  • cases/ or test-cases/ -- simple and clear
  • assertions/ -- not super fan, it feels like utilities asserting values, but still clear
  • journeys/ -- like user journeys as e2e tests is simulating user interactions


## Contributing

- Keep tests focused: one feature per file.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per file? What is a "feature" in this context? Depending on the latter, I don't think we're doing the former (and I wouldn't necessarily want to).

There's a fair bit of "philosophy of testing" that goes into a suggestion like this, which is prone to a lot of debate. Maybe we're better off being way more vague for now?

Suggested change
- Keep tests focused: one feature per file.
- Keep tests focused

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's good

altitude: 0,
});

await formPage.expectLabel(`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a concern now/for these tests, but I have a lurking feeling that the "at index"/unindexed dichotomy is going to be a source of confusion, and a potential risk factor for e.g. flaky tests (things moving around, for reasons unrelated to the test) or false positives (things which are expected at a specific point in form filling progress, but happen to appear somewhere else for some unrelated reason).

Some conceptual prior art we can think about:

  • In @getodk/scenario: PositionalEvent, the notion of a form as a linear sequence of positioned questions, and stateful progression through that (which also adapts to important form concepts like changing relevant state, adding/removing repeats, etc).

    Notably, this concept is reflected in Scenario methods like next (which advances positional state through a form, and its prev counterpart), answer (which sets state at the current positional state).

    Also notably, this whole set of concepts is very well battle tested, as it's all inherited from JavaRosa (albeit implemented rather differently!).

  • We may eventually (maybe even soon!) want to generalize something for this. Probably either in the engine, or at some layer between engine/client, so the same logic (especially stateful aspects of it) can be shared as appropriate by scenario, web-forms UI (e.g. paginated presentation of forms, progress indicator), E2E tests.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't want to add these at index functions, but I didn't get a better idea fast enough to test these notes automatically (It was also a bit of an experiment to see how to address these cases)

Screenshot 2025-03-04 at 8 32 04 PM

Comment on lines 13 to 14
├── controls/ # Reusable controls such as form fields, UI components, etc.
├── GeopointComponent.ts # Example of a reusable control for the geopoint question type.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At the E2E level, I think it might be best to avoid "component" terminology here entirely. Since it's become a technical term of art, I worry that something like GeopointComponent might drift from association with a specific UI component.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure! I changed it to control for these things: GeopointControl

├── geopoint.test.ts # Example of a test file for the geopoint question type.
```

### Key Components
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same commentary on "component" but different reasoning. Maybe...

Suggested change
### Key Components
### Key concepts

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

Comment on lines 27 to 52
## Getting Started

1. **Build the project**
In the root folder run:

```bash
yarn build
```

2. **Run tests**
Execute all E2E tests:

```bash
yarn workspace @getodk/web-forms test:e2e
```

Or run specific tests:

```bash
yarn workspace @getodk/web-forms test:e2e <filepath, e.g. e2e/specs/geopoint.test.ts>
```

3. **Add new tests**
- Create fixtures in `fixtures/` if needed.
- Update or add Page Objects in `page-objects/` for new UI elements.
- Write tests in `specs/` with descriptive names.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe remove this section for now?

  1. Building dependent packages is a prerequisite for everything, and should be covered by package and/or project READMEs. Building the @getodk/web-forms package is actually handled by Playwright config.
  2. Would probably fit better at the package-level README, in a section covering all package-specific testing.
  3. Is pretty much redundant to information in other sections.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, removed

### Key Components

- **Fixtures**: Reusable test data (e.g., sample XForms) to simulate real-world use cases. Add new forms here as needed.
- **Page Objects**: Implements the [Page Object Model (POM)](https://playwright.dev/docs/pom) pattern to encapsulate UI interactions, enhancing test readability and maintainability.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Not our fault/problem, but I can't help call this out...)

What an unfortunate name collision! Microsoft should have known better.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought the same thing!

Base automatically changed from adds-default-geopoint-question-type to main March 3, 2025 16:39
…when readonly mode, and has white background
Improves test cases.
Exposes GeopointNoteValue and GeopointInputValue.
Renames component from InputGeopointReadonly to GeopointFormattedValue. And moves it one folder up.
Adds more variants to demo geopoint xml form
@latin-panda latin-panda force-pushed the e2e-default-geopoint-question-type branch from 825318e to be52263 Compare March 3, 2025 17:28
@latin-panda latin-panda changed the title chore(#249): Adds e2e tests for default geopoint question type chore(#330): Adds e2e tests for default geopoint question type Mar 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add e2e tests for default geopoint question type
2 participants