[test] split up ess and beats serverless integration tests

[pkoutsovasilis]

What does this PR do?

This PR introduces a CI pipeline update that splits the existing ECH-oriented integration tests into dedicated packages for improved modularity and execution flexibility. Specifically, it:

• Introduces new test packages:
  • testing/integration/ess: contains ESS-targeting integration tests
  • testing/integration/serverless_beats: contains Beats-specific serverless integration tests
• Updates the mage targets (integration:Test, integration:Matrix, integration:Single and integration:TestBeatServerless) to run the corresponding test packages.

Why is it important?

• Improved modularity: Isolates tests specific to ECH and serverless flavors into their own packages, reducing the noise and tight coupling in the shared integration test package.
• Better maintainability: Allows more focused test ownership and easier local test development.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

No user-facing disruption; internal test structure change only.

How to test this PR locally

for Beats serverless you need to have an already compiled filebeat

mage integration:auth
STACK_PROVISIONER=serverless mage integration:testBeatServerless filebeat

STACK_PROVISIONER=stateful TEST_PLATFORMS="windows/amd64" mage integration:single TestStandaloneUpgrade

Related issues

• Relates Split out integration tests into different packages #8087
• Closes Split up ECH integration tests into dedicated package #8411

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[pchila]

I left a small nitpick (not blocking for merging) but code LGTM overall

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: e3a4138

History

• 💚 Build #22769 succeeded d1bf164
• 💔 Build #22717 failed db1dd0c
• 💔 Build #22662 failed 2eb68d3
• 💚 Build #22578 succeeded cb5a016

cc @pkoutsovasilis

[swiatekm]

LGTM, thanks for doing this!

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube