-
Notifications
You must be signed in to change notification settings - Fork 13
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore(#330): Adds e2e tests for default geopoint question type #313
base: main
Are you sure you want to change the base?
Conversation
|
|
||
``` | ||
e2e/ | ||
├── fixtures/ # Sample forms and data for testing. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
At the moment, this folder is empty, but next time, it'd be good to take the XML forms from here. I added a ToDo comment for this improvement in the PreviewPage.ts
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We moved the fixtures into @getodk/common
because they're shared across:
@getodk/scenario
tests@getodk/web-forms
:- E2E tests
- Vitest tests
- Dev/demo
- Web Forms Preview
In theory they could also be used in @getodk/xpath
and @getodk/xforms-engine
as well. We just haven't added tests that would take advantage of that, since this shared solution was established. A fair bit of effort has gone into making sure that sharing works for a wide variety of conditions (including not just form fixtures themselves, but external resources they reference by jr:
URL) and use cases (including common structural organization, and associating XForms XML fixtures with their source XLSForms).
All of that context may not be obvious without elaborating on it, but I want to make sure it's clear before asking: what would be the advantage of bypassing the shared fixture solution here? Is there something the E2E use case needs that isn't already solved? I'd prefer to try to address it in the shared context first if that's the case, so we can continue to benefit from solutions being broadly available anywhere fixtures are referenced. If that doesn't seem reasonable or tenable I'm open to reconsidering! But I definitely want to understand the intent before we start splitting up something that's working well as a project-wide resource for everything else.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the context!
I'm still unsure if the fixtures in @getodk/common
will fully meet the E2E testing needs, but if a special case arises, e.g., verify UX based on special appearance that modifies the display, I don’t think there will be many, and we can add it to@getodk/common
.
This was intended to keep it contained from other types of testing, but I don't have a strong preference. This approach has existed for a while without producing any issues with other test types. I'm open to keeping it that way.
Do you agree to remove the fixture from these e2e tests and keep using the one in @getodk/common
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This file name is confusing. I moved the test to /e2e/specs/build/preview-form-render.test.ts
We can move the /e2e/specs/build/preview-form-render.test.ts
to /e2e/specs/preview-form-render.test.ts
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This file now contains the tests of vue.test.ts
and wf-preview.test.ts
since both verify the demo forms.
I didn't change the tests; I just added the PreviewPage to navigate to the demo page. Everything else is as original
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could move this file out from the build folder, from e2e/specs/build/style.test.ts
to e2e/specs/style.test.ts
packages/web-forms/e2e/vue.test.ts
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same, this file name is not descriptive, for now I moved the content to e2e/specs/build/preview-form-render.test.ts
Hi @eyelidlessness, can you please review? Thank you for your time! Quick question, we don't need a changeset for this PR, or do we? |
5b1547f
to
825318e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lots of feedback, I know! But I don't want that to detract from saying...
THIS IS AWESOME! Thank you so much for taking this on.
|
||
``` | ||
e2e/ | ||
├── fixtures/ # Sample forms and data for testing. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We moved the fixtures into @getodk/common
because they're shared across:
@getodk/scenario
tests@getodk/web-forms
:- E2E tests
- Vitest tests
- Dev/demo
- Web Forms Preview
In theory they could also be used in @getodk/xpath
and @getodk/xforms-engine
as well. We just haven't added tests that would take advantage of that, since this shared solution was established. A fair bit of effort has gone into making sure that sharing works for a wide variety of conditions (including not just form fixtures themselves, but external resources they reference by jr:
URL) and use cases (including common structural organization, and associating XForms XML fixtures with their source XLSForms).
All of that context may not be obvious without elaborating on it, but I want to make sure it's clear before asking: what would be the advantage of bypassing the shared fixture solution here? Is there something the E2E use case needs that isn't already solved? I'd prefer to try to address it in the shared context first if that's the case, so we can continue to benefit from solutions being broadly available anywhere fixtures are referenced. If that doesn't seem reasonable or tenable I'm open to reconsidering! But I definitely want to understand the intent before we start splitting up something that's working well as a project-wide resource for everything else.
}; | ||
|
||
test.describe('Preview Form Render', () => { | ||
test('demo forms load', async ({ context, page }) => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The origin of this test was #265. I probably could have written up more detail about why that PR was created. It was a bit of a rush to fix a regression in the Web Forms Preview, much like #320 is.
The original name and file system structure was descriptive, though you may need additional context for that to be clear! The test is exercising:
- the build (hence its presence in a
build
subdirectory) - the Web Forms Preview build product, specifically
I also think it's important to note that up to now, Web Forms Preview is currently our only production deployment. It's also our most immediate, concrete way to show users the state of the Web Forms project. (For additional context: this is also why I dropped everything, even extended my work day yesterday, to prioritize #320!)
It's good that the build
aspect is still captured in the filesystem structure. I would like to restore both:
- Specific mention of Web Forms Preview in the test description
- A distinct, clearly named module/suite for tests exercising Web Forms Preview specifically
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure!
- The file will remain in the build directory of e2e tests:
packages/web-forms/e2e/specs/build/
- Changing the test description to
test.describe('Web Forms Preview Demo Forms', () => {
- Changing the test filename to
web-forms-preview-forms.ts
orpreview-demo-forms.ts
Did I capture correctly?
test('all forms are rendered and there is no console error', async ({ page, browserName }) => { | ||
let consoleErrors = 0; | ||
page.on('console', (msg) => { | ||
if (msg.type() === 'error' || msg.type() === 'warning') { | ||
consoleErrors++; | ||
} | ||
}); | ||
|
||
const previewPage = new PreviewPage(page); | ||
await previewPage.goToPage(); | ||
|
||
// this ensures that Vue application is loaded before proceeding forward. | ||
await expect(page.getByText('Demo Forms')).toBeVisible(); | ||
|
||
// Let's expand all categories, so that Form list is visible | ||
await expandAllCategories(page); | ||
|
||
const forms = await page.locator('ul.form-list li a').all(); | ||
|
||
expect(forms.length).toBeGreaterThan(0); | ||
|
||
for (const form of forms) { | ||
await form.click(); | ||
|
||
// Traverse the form element by element | ||
// if focused element is an editable textbox then fill it | ||
// Exit the loop when focus is on the Send button | ||
while (true) { | ||
const onSendButton = await page.evaluate(() => { | ||
const activeElement = document.activeElement; | ||
return activeElement?.tagName === 'BUTTON' && activeElement.textContent === 'Send'; | ||
}); | ||
|
||
if (onSendButton) { | ||
break; | ||
} | ||
|
||
await page.keyboard.press(browserName == 'webkit' ? 'Alt+Tab' : 'Tab'); | ||
|
||
const inputType = await page.evaluate(() => { | ||
const isInputElement = ( | ||
activeElement: Element | null | ||
): activeElement is HTMLInputElement => { | ||
return activeElement?.tagName === 'INPUT'; | ||
}; | ||
|
||
const activeElement = document.activeElement; | ||
|
||
if ( | ||
!isInputElement(activeElement) || | ||
activeElement.hasAttribute('readonly') || | ||
activeElement.hasAttribute('disabled') | ||
) { | ||
return null; | ||
} | ||
|
||
return activeElement.type; | ||
}); | ||
|
||
if (inputType === 'text') { | ||
await page.keyboard.type(faker.internet.displayName()); | ||
} | ||
// Select the next option, if the last option is selected by default | ||
// then browser selects the first one. | ||
else if (inputType === 'radio') { | ||
await page.keyboard.press('ArrowDown'); | ||
} | ||
// Tab behaviour for checkboxes is different, each Tab press moves the focus | ||
// to the next option. Here we are toggling every checkbox option. | ||
else if (inputType === 'checkbox') { | ||
// Toggle the option | ||
await page.keyboard.press('Space'); | ||
} | ||
} | ||
|
||
const langChanger = page.locator('.larger-screens .language-changer'); | ||
|
||
if ((await langChanger.count()) > 0) { | ||
await langChanger.click(); | ||
await page.locator('.language-dd-label').last().click(); | ||
} | ||
|
||
await page.goBack(); | ||
await expandAllCategories(page); | ||
} | ||
|
||
expect(consoleErrors).toBe(0); | ||
}); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I mentioned this in chat... I'm not sure how much value this test (originally from vue.test.ts
) provides. In its current form, I think it's probably at least as much of a liability.1
I agree the original module/suite name wasn't descriptive. I'm not sure the test description (copied here unmodified I believe) is very helpful either. In my opinion, pretty much everything about this test screams: if there's anything worth keeping in place of this test, it might be a TODO. (And in a module/suite separate from the more clearly valuable test of the Web Forms Preview build product.)
Maybe something like this?
test('all forms are rendered and there is no console error', async ({ page, browserName }) => { | |
let consoleErrors = 0; | |
page.on('console', (msg) => { | |
if (msg.type() === 'error' || msg.type() === 'warning') { | |
consoleErrors++; | |
} | |
}); | |
const previewPage = new PreviewPage(page); | |
await previewPage.goToPage(); | |
// this ensures that Vue application is loaded before proceeding forward. | |
await expect(page.getByText('Demo Forms')).toBeVisible(); | |
// Let's expand all categories, so that Form list is visible | |
await expandAllCategories(page); | |
const forms = await page.locator('ul.form-list li a').all(); | |
expect(forms.length).toBeGreaterThan(0); | |
for (const form of forms) { | |
await form.click(); | |
// Traverse the form element by element | |
// if focused element is an editable textbox then fill it | |
// Exit the loop when focus is on the Send button | |
while (true) { | |
const onSendButton = await page.evaluate(() => { | |
const activeElement = document.activeElement; | |
return activeElement?.tagName === 'BUTTON' && activeElement.textContent === 'Send'; | |
}); | |
if (onSendButton) { | |
break; | |
} | |
await page.keyboard.press(browserName == 'webkit' ? 'Alt+Tab' : 'Tab'); | |
const inputType = await page.evaluate(() => { | |
const isInputElement = ( | |
activeElement: Element | null | |
): activeElement is HTMLInputElement => { | |
return activeElement?.tagName === 'INPUT'; | |
}; | |
const activeElement = document.activeElement; | |
if ( | |
!isInputElement(activeElement) || | |
activeElement.hasAttribute('readonly') || | |
activeElement.hasAttribute('disabled') | |
) { | |
return null; | |
} | |
return activeElement.type; | |
}); | |
if (inputType === 'text') { | |
await page.keyboard.type(faker.internet.displayName()); | |
} | |
// Select the next option, if the last option is selected by default | |
// then browser selects the first one. | |
else if (inputType === 'radio') { | |
await page.keyboard.press('ArrowDown'); | |
} | |
// Tab behaviour for checkboxes is different, each Tab press moves the focus | |
// to the next option. Here we are toggling every checkbox option. | |
else if (inputType === 'checkbox') { | |
// Toggle the option | |
await page.keyboard.press('Space'); | |
} | |
} | |
const langChanger = page.locator('.larger-screens .language-changer'); | |
if ((await langChanger.count()) > 0) { | |
await langChanger.click(); | |
await page.locator('.language-dd-label').last().click(); | |
} | |
await page.goBack(); | |
await expandAllCategories(page); | |
} | |
expect(consoleErrors).toBe(0); | |
}); | |
test.skip('All forms are rendered without producing unexpected errors', () => { | |
/** | |
* There was previously a test with a similar description which: | |
* | |
* 1. Loads the dev/demo fixtures list | |
* 2. Iterates through each fixture, and for each: | |
* 1. Loads that fixture in dev/demo mode | |
* 2. [Attempts to] progress through all of the form's controls, and for | |
* each: | |
* - If it is a text field: fill a value | |
* - If it is a field with options: select each of its options | |
* - [The test was not maintained, so no other controls/interaction | |
* methods have been exercised] | |
* 3. Attempts to invoke the dev/demo form submission behavior | |
* 4. Returns to the dev/demo fixtures list for the next iteration | |
* | |
* Currently we see little (if any) value in this test _as it was written_. We | |
* may revisit it in the future. **If** (and insofar as) there would be value | |
* in such a generalized testing approach, revisiting it would likely involve: | |
* | |
* - Splitting out individual tests per fixture, which might utilize: | |
* - [Playwright Fixtures](https://playwright.dev/docs/api/class-fixtures) | |
* (note: despite the name, this is a different concept from what we call | |
* fixtures!) | |
* - Code generation | |
* - [Some other, hopefully maintainable, way to apply test logic across a | |
* frequently changing set of fixtures with a wide variety of use cases] | |
* | |
* - A more thoughtful approach to covering the ever-growing set of form | |
* controls and interaction modes they provide/require | |
* | |
* - A more thoughtful approach to checking for errors, including at least: | |
* - A more intentional way to identify whether errors occur | |
* - A more intentional way to identify whether errors that do occur are | |
* expected (i.e. messaging to user) or not (i.e. uncaught or unhandled | |
* in myriad ways) | |
* - Some way of designating fixtures which _are expected_ to produce | |
errors… | |
* - on load | |
* - due to a specific interaction | |
* | |
* - Possibly some set of additional assertions and/or test cases which would | |
* apply globally to any form (or any clear subset of forms) | |
*/ | |
}); |
Footnotes
-
And I think the amount of work that's already gone into keeping it wobbling along is a testament to that! ↩
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@eyelidlessness, I would prefer to delete this test, create a new ticket with this detail, and link this PR. So it can be assigned as a proper task.
Would that be okay?
├── GeopointComponent.ts # Example of a reusable control for the geopoint question type. | ||
├── pages/ # Full page representations. | ||
├── FormPage.ts # Example of a full page representation for a form. | ||
├── specs/ # Test specification files. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We generally haven't used the "specs" naming convention for tests, throughout the WF project. We have either:
test
(in all other packages)tests
(Vitest tests in@getodk/web-forms
)1
As I understand it, the "spec" terminology comes from the Behavior Driven Development philosophy: tests act as a specification of behavior for the functionality under test. I like a lot about the BDD philosophy, especially its approach to describing tests. In a vacuum, I even like the "spec" concept/naming convention!
But it's potentially a major source of confusion where we as a team already refer to several things as "specs":
- Actual specifications named as such—e.g. ODK XForms Specification (and the underlying W3C XForms, XML, XPath specifications)
- Documentation—e.g. of XLSForms, Collect, etc
- In some cases, observable behavior—e.g. aligning Web Forms behavior to match what we know of the equivalent behavior in Collect/JavaRosa has at times been called "the spec"
I originally started writing this comment as a "nit", but honestly as I've written this out I gotta be honest: I find the term is already way overloaded, which has already caused a fair bit of confusion. I'm really reluctant to overload it more!
Footnotes
-
IIRC this was inherited from a Vue project template. In hindsight, I wish we'd made it consistent with the other packages then! I'd prefer to do so soon/now, if we agree! ↩
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was actually expecting to find the e2e tests inside the /test
folder, any type of test actually, for example:
|─ tests/
|─|─ e2e/
|─|─ |─ page-objects/
|─|─ |─ specs/
|─|─ unit/
|─|─|─ components/
|─|─ integration etc....
Would it be possible for us to do this? Starting with Web Forms client. What do you think?
But it's potentially a major source of confusion where we as a team already refer to several things as "specs"
Good point! What do you think about changing specs
to any of these options:
cases/
ortest-cases/
-- simple and clearassertions/
-- not super fan, it feels like utilities asserting values, but still clearjourneys/
-- like user journeys as e2e tests is simulating user interactions
packages/web-forms/e2e/README.md
Outdated
|
||
## Contributing | ||
|
||
- Keep tests focused: one feature per file. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Per file? What is a "feature" in this context? Depending on the latter, I don't think we're doing the former (and I wouldn't necessarily want to).
There's a fair bit of "philosophy of testing" that goes into a suggestion like this, which is prone to a lot of debate. Maybe we're better off being way more vague for now?
- Keep tests focused: one feature per file. | |
- Keep tests focused |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, that's good
altitude: 0, | ||
}); | ||
|
||
await formPage.expectLabel(` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not a concern now/for these tests, but I have a lurking feeling that the "at index"/unindexed dichotomy is going to be a source of confusion, and a potential risk factor for e.g. flaky tests (things moving around, for reasons unrelated to the test) or false positives (things which are expected at a specific point in form filling progress, but happen to appear somewhere else for some unrelated reason).
Some conceptual prior art we can think about:
-
In
@getodk/scenario
:PositionalEvent
, the notion of a form as a linear sequence of positioned questions, and stateful progression through that (which also adapts to important form concepts like changingrelevant
state, adding/removing repeats, etc).Notably, this concept is reflected in
Scenario
methods likenext
(which advances positional state through a form, and itsprev
counterpart),answer
(which sets state at the current positional state).Also notably, this whole set of concepts is very well battle tested, as it's all inherited from JavaRosa (albeit implemented rather differently!).
-
We may eventually (maybe even soon!) want to generalize something for this. Probably either in the engine, or at some layer between engine/client, so the same logic (especially stateful aspects of it) can be shared as appropriate by
scenario
,web-forms
UI (e.g. paginated presentation of forms, progress indicator), E2E tests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
packages/web-forms/e2e/README.md
Outdated
├── controls/ # Reusable controls such as form fields, UI components, etc. | ||
├── GeopointComponent.ts # Example of a reusable control for the geopoint question type. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
At the E2E level, I think it might be best to avoid "component" terminology here entirely. Since it's become a technical term of art, I worry that something like GeopointComponent
might drift from association with a specific UI component.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure! I changed it to control for these things: GeopointControl
packages/web-forms/e2e/README.md
Outdated
├── geopoint.test.ts # Example of a test file for the geopoint question type. | ||
``` | ||
|
||
### Key Components |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same commentary on "component" but different reasoning. Maybe...
### Key Components | |
### Key concepts |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed
packages/web-forms/e2e/README.md
Outdated
## Getting Started | ||
|
||
1. **Build the project** | ||
In the root folder run: | ||
|
||
```bash | ||
yarn build | ||
``` | ||
|
||
2. **Run tests** | ||
Execute all E2E tests: | ||
|
||
```bash | ||
yarn workspace @getodk/web-forms test:e2e | ||
``` | ||
|
||
Or run specific tests: | ||
|
||
```bash | ||
yarn workspace @getodk/web-forms test:e2e <filepath, e.g. e2e/specs/geopoint.test.ts> | ||
``` | ||
|
||
3. **Add new tests** | ||
- Create fixtures in `fixtures/` if needed. | ||
- Update or add Page Objects in `page-objects/` for new UI elements. | ||
- Write tests in `specs/` with descriptive names. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe remove this section for now?
- Building dependent packages is a prerequisite for everything, and should be covered by package and/or project READMEs. Building the
@getodk/web-forms
package is actually handled by Playwright config. - Would probably fit better at the package-level README, in a section covering all package-specific testing.
- Is pretty much redundant to information in other sections.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agree, removed
### Key Components | ||
|
||
- **Fixtures**: Reusable test data (e.g., sample XForms) to simulate real-world use cases. Add new forms here as needed. | ||
- **Page Objects**: Implements the [Page Object Model (POM)](https://playwright.dev/docs/pom) pattern to encapsulate UI interactions, enhancing test readability and maintainability. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(Not our fault/problem, but I can't help call this out...)
What an unfortunate name collision! Microsoft should have known better.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought the same thing!
This reverts commit f447f13.
…when readonly mode, and has white background
…berParsers.ts. Moving it to the lib folder
825318e
to
be52263
Compare
Closes #330
Part of #249
I have verified this PR works in these browsers (latest versions):
What else has been done to verify that this works as intended?
Running tests locally and in CI
Why is this the best possible solution? Were any other approaches considered?
Initially, I tried adding tests in Vue, but our Web Form client is primarily a presentation layer, and Vue doesn't support tests requiring browser permissions. Therefore, I chose Playwright for automation (already a project dependency), following a standard folder structure and the use of page objects, as I’ve done with other test frameworks (Protractor, WebDriverIO). This is compatible with most frameworks if we ever switch in the future. Additionally, I included a minimal README file to guide contributors in adding new tests.
How does this change affect users? Describe intentional changes to behavior and behavior that could have accidentally been affected by code changes. In other words, what are the regression risks?
N/A
Do we need any specific form for testing your changes? If so, please attach one.
N/A
What's changed
This is not perfect, and more can be improved, but it is a good start that we can iterate as we add more tests.
wf-preview.test.ts
andvue.test.ts
to/e2e/specs/build/preview-form-render.test.ts
. The original name wasn't descriptive.