Open
Description
Failing test case
TestUpgradeAgentWithTamperProtectedEndpoint_RPM
Error message
Condition never satisfied
Build
OS
Linux
Stacktrace and notes
=== Failed
=== FAIL: testing/integration TestUpgradeAgentWithTamperProtectedEndpoint_RPM/Install_same_version_over_the_installed_agent (386.38s)
fetcher_artifact.go:222: Downloading artifact from https://snapshots.elastic.co/9.1.0-3292dfc0/downloads/beats/elastic-agent/elastic-agent-9.1.0-SNAPSHOT-x86_64.rpm
fetcher_artifact.go:301: Downloading artifact progress 14.96%
fetcher_artifact.go:301: Downloading artifact progress 27.05%
fetcher_artifact.go:301: Downloading artifact progress 37.41%
fetcher_artifact.go:301: Downloading artifact progress 50.11%
fetcher_artifact.go:301: Downloading artifact progress 61.73%
fetcher_artifact.go:301: Downloading artifact progress 72.79%
fetcher_artifact.go:301: Downloading artifact progress 82.30%
fetcher_artifact.go:301: Downloading artifact progress 91.66%
fetcher_artifact.go:301: Downloading artifact progress 100.00%
fetcher_artifact.go:301: Downloading artifact progress 100.00%
fetcher_artifact.go:268: Completed downloading artifact from https://snapshots.elastic.co/9.1.0-3292dfc0/downloads/beats/elastic-agent/elastic-agent-9.1.0-SNAPSHOT-x86_64.rpm
fetcher_artifact.go:222: Downloading artifact from https://snapshots.elastic.co/9.1.0-3292dfc0/downloads/beats/elastic-agent/elastic-agent-9.1.0-SNAPSHOT-x86_64.rpm.sha512
fetcher_artifact.go:268: Completed downloading artifact from https://snapshots.elastic.co/9.1.0-3292dfc0/downloads/beats/elastic-agent/elastic-agent-9.1.0-SNAPSHOT-x86_64.rpm.sha512
fetcher_artifact.go:222: Downloading artifact from https://snapshots.elastic.co/9.1.0-3292dfc0/downloads/beats/elastic-agent/elastic-agent-9.1.0-SNAPSHOT-x86_64.rpm.asc
fetcher_artifact.go:268: Completed downloading artifact from https://snapshots.elastic.co/9.1.0-3292dfc0/downloads/beats/elastic-agent/elastic-agent-9.1.0-SNAPSHOT-x86_64.rpm.asc
fixture.go:311: Extracting artifact elastic-agent-9.1.0-SNAPSHOT-x86_64.rpm to /tmp/TestUpgradeAgentWithTamperProtectedEndpoint_RPM-Install_same_version_over_the_installed_agent1824277952
fixture.go:329: Completed extraction of artifact elastic-agent-9.1.0-SNAPSHOT-x86_64.rpm to /tmp/TestUpgradeAgentWithTamperProtectedEndpoint_RPM-Install_same_version_over_the_installed_agent1824277952
fixture.go:1011: Components were not modified from the fetched artifact
endpoint_security_test.go:206: Creating a generic policy and enrollment token
log_level_test.go:385: Creating Agent policy...
log_level_test.go:389: Creating Agent enrollment API key...
endpoint_security_test.go:210: Install elastic defend
endpoint_security_test.go:211: Templating endpoint package policy request
endpoint_security_test.go:211: POST /api/fleet/package_policies
endpoint_security_test.go:220: Updating the policy to set "is_protected" to true
endpoint_security_test.go:223: Get the policy uninstall token
endpoint_security_test.go:228: Install and enroll the first agent
endpoint_security_test.go:229: Creating enrollment API key...
fixture_install.go:200: [test TestUpgradeAgentWithTamperProtectedEndpoint_RPM/Install_same_version_over_the_installed_agent] Inside fixture install function
fixture_install.go:564: [test TestUpgradeAgentWithTamperProtectedEndpoint_RPM/Install_same_version_over_the_installed_agent] Inside fixture installRpm function
endpoint_security_test.go:229: >>> Enroll succeeded. Output:
fixture.go:684: >> running binary with: [/usr/bin/elastic-agent status --output json]
endpoint_security_test.go:229: >>> Enrolled Agent ID: 0449a30e-09ad-41d4-9449-91ad03b72475
endpoint_security_test.go:244: The initial installation of both the agent and endpoint are healthy
endpoint_security_test.go:247: The initial endpoint version is 9.1.0-SNAPSHOT
endpoint_security_test.go:334: The initial endpoint version is 9.1.0-SNAPSHOT
[ ===] Failed to uninstall service [0s] failed to uninstall component "endpoint-default": error uninstalling service: 2025-06-21 01:26:48: error: InstallLib.cpp:1265 Invalid uninstall token: exit status 28
[====] Failed to uninstall agent [0s] Error uninstalling. Printing logs
2025-06-21T01:26:47.930Z DEBUG [uninstall.state_migration] not attempting to migrate from action store: state store already exists
2025-06-21T01:26:48.260Z DEBUG [uninstall.state_migration] not attempting to migrate from action store: state store already exists
2025-06-21T01:26:48.282Z DEBUG [uninstall.composable] Starting controller for composable inputs
2025-06-21T01:26:48.282Z DEBUG [uninstall.composable] Started controller for composable inputs
2025-06-21T01:26:48.282Z DEBUG [uninstall.composable] Computing new variable state for composable inputs
2025-06-21T01:26:48.282Z DEBUG [uninstall.composable] Stopping controller for composable inputs
2025-06-21T01:26:48.282Z DEBUG [uninstall.composable] Stopped controller for composable inputs
2025-06-21T01:26:48.285Z INFO [uninstall] Capabilities file not found in /etc/elastic-agent/capabilities.yml
2025-06-21T01:26:48.285Z DEBUG [uninstall] uninstall endpoint-security service
2025-06-21T01:26:48.295Z ERROR [uninstall] 2025-06-21 01:26:48: debug: ProcFile.cpp:855 Found 12 cgroups for pid(83271) {"context": "command output"}
2025-06-21T01:26:48.295Z ERROR [uninstall] 2025-06-21 01:26:48: debug: ProcFile.cpp:861 cgroup: id=10 type=pids path=/user.slice/user-1003.slice/session-c4.scope {"context": "command output"}
2025-06-21T01:26:48.295Z ERROR [uninstall] 2025-06-21 01:26:48: info: MainPosix.cpp:263 Executing uninstall {"context": "command output"}
2025-06-21T01:26:48.295Z ERROR [uninstall] 2025-06-21 01:26:48: debug: VaultLib.cpp:207 Vault initialized with existing seed file {"context": "command output"}
2025-06-21T01:26:48.305Z ERROR [uninstall] 2025-06-21 01:26:48: debug: VaultLib.cpp:614 Successfully read vault key: config {"context": "command output"}
2025-06-21T01:26:48.305Z ERROR [uninstall] 2025-06-21 01:26:48: debug: ECSUtilities.cpp:497 Tamper protection enabled {"context": "command output"}
2025-06-21T01:26:48.305Z ERROR [uninstall] 2025-06-21 01:26:48: info: InstallLib.cpp:977 Checking installed uninstall protection artifacts {"context": "command output"}
2025-06-21T01:26:48.312Z ERROR [uninstall] 2025-06-21 01:26:48: debug: VaultLib.cpp:614 Successfully read vault key: config {"context": "command output"}
2025-06-21T01:26:48.313Z ERROR [uninstall] 2025-06-21 01:26:48: info: InstallLib.cpp:736 No custom public key detected in Endpoint config {"context": "command output"}
2025-06-21T01:26:48.313Z ERROR [uninstall] 2025-06-21 01:26:48: debug: CryptoLib_Cpp.cpp:1403 RSA signature verified {"context": "command output"}
2025-06-21T01:26:48.313Z ERROR [uninstall] 2025-06-21 01:26:48: info: InstallLib.cpp:909 Failed to read os section of tamper-protection-config, continuing {"context": "command output"}
2025-06-21T01:26:48.313Z ERROR [uninstall] 2025-06-21 01:26:48: info: InstallLib.cpp:998 Finished checking installed uninstall protection artifacts with result deny {"context": "command output"}
2025-06-21T01:26:48.313Z ERROR [uninstall] 2025-06-21 01:26:48: info: InstallLib.cpp:1070 Finished checking command line provided uninstall resource result deny {"context": "command output"}
2025-06-21T01:26:48.313Z ERROR [uninstall] 2025-06-21 01:26:48: error: InstallLib.cpp:1265 Invalid uninstall token {"context": "command output"}
Error: error uninstalling agent: error uninstalling components: error uninstalling component: error uninstalling service: 2025-06-21 01:26:48: error: InstallLib.cpp:1265 Invalid uninstall token: exit status 28
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/9.1/fleet-troubleshooting.html
endpoint_security_test.go:348: Tamper protection for the initial installation of the agent is enabled
endpoint_security_test.go:357: Getting source package
endpoint_security_test.go:361: Installing the second agent, upgrading from the older version
endpoint_security_test.go:366: warning: /opt/buildkite-agent/builds/bk-agent-prod-gcp-1750468221167740260/elastic/elastic-agent/.agent-testing/artifact/elastic-agent-9.1.0-SNAPSHOT-x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Verifying... ########################################
Preparing... ########################################
ElasticEndpoint is installed
ElasticEndpoint is running, stopping it
ElasticEndpoint is tamper protected
Removing ElasticEndpoint vault
previous installation directory /var/lib/elastic-agent/data/elastic-agent-9.1.0-SNAPSHOT-e0d2d2
Updating / installing...
elastic-agent-9.1.0-1 ########################################
found symlink /usr/share/elastic-agent/bin/elastic-agent, unlink
Checking if ElasticEndpoint is installed
ElasticEndpoint is installed at version version: 9.1.0-SNAPSHOT, compiled: Fri Jun 20 06:00:00 2025, branch: HEAD, commit: e9d5ab261a1ad8a0df1ff96a88594099755cf7e2
ElasticEndpoint is installed but not running
New endpoint and installed endpoint versions are the same: "version: 9.1.0-SNAPSHOT, compiled: Fri Jun 20 06:00:00 2025, branch: HEAD, commit: e9d5ab261a1ad8a0df1ff96a88594099755cf7e2"
Starting ElasticEndpoint
create symlink /usr/share/elastic-agent/bin/elastic-agent to /var/lib/elastic-agent/data/elastic-agent-9.1.0-SNAPSHOT-e0d2d2/elastic-agent
systemd enable/restart elastic-agent
endpoint_security_test.go:376: Error getting agent state: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/lib/elastic-agent/elastic-agent.sock: connect: no such file or directory"
endpoint_security_test.go:376: endpoint component is not Healthy: current state: {ID:endpoint-default Name:endpoint State:STARTING Message:Starting: endpoint service runtime Units:[{UnitID:endpoint-default UnitType:OUTPUT State:STARTING Message:Starting: endpoint service runtime Payload:map[]} {UnitID:endpoint-default-2475a0b0-cbb0-4bc2-aa60-c9689bd6cc16 UnitType:INPUT State:STARTING Message:Starting: endpoint service runtime Payload:map[]}] VersionInfo:{Name: Meta:map[]}}
endpoint_security_test.go:376: endpoint component is not Healthy: current state: {ID:endpoint-default Name:endpoint State:STARTING Message:Starting: endpoint service runtime Units:[{UnitID:endpoint-default UnitType:OUTPUT State:STARTING Message:Starting: endpoint service runtime Payload:map[]} {UnitID:endpoint-default-2475a0b0-cbb0-4bc2-aa60-c9689bd6cc16 UnitType:INPUT State:STARTING Message:Starting: endpoint service runtime Payload:map[]}] VersionInfo:{Name: Meta:map[]}}
endpoint_security_test.go:376: endpoint component is not Healthy: current state: {ID:endpoint-default Name:endpoint State:STARTING Message:Starting: endpoint service runtime Units:[{UnitID:endpoint-default UnitType:OUTPUT State:STARTING Message:Starting: endpoint service runtime Payload:map[]} {UnitID:endpoint-default-2475a0b0-cbb0-4bc2-aa60-c9689bd6cc16 UnitType:INPUT State:STARTING Message:Starting: endpoint service runtime Payload:map[]}] VersionInfo:{Name: Meta:map[]}}
endpoint_security_test.go:376: endpoint component is not Healthy: current state: {ID:endpoint-default Name:endpoint State:STARTING Message:Starting: endpoint service runtime Units:[{UnitID:endpoint-default UnitType:OUTPUT State:STARTING Message:Starting: endpoint service runtime Payload:map[]} {UnitID:endpoint-default-2475a0b0-cbb0-4bc2-aa60-c9689bd6cc16 UnitType:INPUT State:STARTING Message:Starting: endpoint service runtime Payload:map[]}] VersionInfo:{Name: Meta:map[]}}
endpoint_security_test.go:376: endpoint component is not Healthy: current state: {ID:endpoint-default Name:endpoint State:STARTING Message:Starting: endpoint service runtime Units:[{UnitID:endpoint-default UnitType:OUTPUT State:STARTING Message:Starting: endpoint service runtime Payload:map[]} {UnitID:endpoint-default-2475a0b0-cbb0-4bc2-aa60-c9689bd6cc16 UnitType:INPUT State:STARTING Message:Starting: endpoint service runtime Payload:map[]}] VersionInfo:{Name: Meta:map[]}}
endpoint_security_test.go:376: endpoint component is not Healthy: current state: {ID:endpoint-default Name:endpoint State:STARTING Message:Starting: endpoint service runtime Units:[{UnitID:endpoint-default UnitType:OUTPUT State:STARTING Message:Starting: endpoint service runtime Payload:map[]} {UnitID:endpoint-default-2475a0b0-cbb0-4bc2-aa60-c9689bd6cc16 UnitType:INPUT State:STARTING Message:Starting: endpoint service runtime Payload:map[]}] VersionInfo:{Name: Meta:map[]}}
endpoint_security_test.go:376: endpoint component is not Healthy: current state: {ID:endpoint-default Name:endpoint State:STARTING Message:Starting: endpoint service runtime Units:[{UnitID:endpoint-default UnitType:OUTPUT State:STARTING Message:Starting: endpoint service runtime Payload:map[]} {UnitID:endpoint-default-2475a0b0-cbb0-4bc2-aa60-c9689bd6cc16 UnitType:INPUT State:STARTING Message:Starting: endpoint service runtime Payload:map[]}] VersionInfo:{Name: Meta:map[]}}
endpoint_security_test.go:376: endpoint component is not Healthy: current state: {ID:endpoint-default Name:endpoint State:STARTING Message:Starting: endpoint service runtime Units:[{UnitID:endpoint-default-2475a0b0-cbb0-4bc2-aa60-c9689bd6cc16 UnitType:INPUT State:STARTING Message:Starting: endpoint service runtime Payload:map[]} {UnitID:endpoint-default UnitType:OUTPUT State:STARTING Message:Starting: endpoint service runtime Payload:map[]}] VersionInfo:{Name: Meta:map[]}}
endpoint_security_test.go:376: endpoint component is not Healthy: current state: {ID:endpoint-default Name:endpoint State:STARTING Message:Starting: endpoint service runtime Units:[{UnitID:endpoint-default UnitType:OUTPUT State:STARTING Message:Starting: endpoint service runtime Payload:map[]} {UnitID:endpoint-default-2475a0b0-cbb0-4bc2-aa60-c9689bd6cc16 UnitType:INPUT State:STARTING Message:Starting: endpoint service runtime Payload:map[]}] VersionInfo:{Name: Meta:map[]}}
endpoint_security_test.go:376: endpoint component is not Healthy: current state: {ID:endpoint-default Name:endpoint State:STARTING Message:Starting: endpoint service runtime Units:[{UnitID:endpoint-default UnitType:OUTPUT State:STARTING Message:Starting: endpoint service runtime Payload:map[]} {UnitID:endpoint-default-2475a0b0-cbb0-4bc2-aa60-c9689bd6cc16 UnitType:INPUT State:STARTING Message:Starting: endpoint service runtime Payload:map[]}] VersionInfo:{Name: Meta:map[]}}
endpoint_security_test.go:376: endpoint component is not Healthy: current state: {ID:endpoint-default Name:endpoint State:STARTING Message:Starting: endpoint service runtime Units:[{UnitID:endpoint-default UnitType:OUTPUT State:STARTING Message:Starting: endpoint service runtime Payload:map[]} {UnitID:endpoint-default-2475a0b0-cbb0-4bc2-aa60-c9689bd6cc16 UnitType:INPUT State:STARTING Message:Starting: endpoint service runtime Payload:map[]}] VersionInfo:{Name: Meta:map[]}}
endpoint_security_test.go:376: endpoint component is not Healthy: current state: {ID:endpoint-default Name:endpoint State:STARTING Message:Starting: endpoint service runtime Units:[{UnitID:endpoint-default UnitType:OUTPUT State:STARTING Message:Starting: endpoint service runtime Payload:map[]} {UnitID:endpoint-default-2475a0b0-cbb0-4bc2-aa60-c9689bd6cc16 UnitType:INPUT State:STARTING Message:Starting: endpoint service runtime Payload:map[]}] VersionInfo:{Name: Meta:map[]}}
endpoint_security_test.go:376: endpoint component is not Healthy: current state: {ID:endpoint-default Name:endpoint State:STARTING Message:Starting: endpoint service runtime Units:[{UnitID:endpoint-default UnitType:OUTPUT State:STARTING Message:Starting: endpoint service runtime Payload:map[]} {UnitID:endpoint-default-2475a0b0-cbb0-4bc2-aa60-c9689bd6cc16 UnitType:INPUT State:STARTING Message:Starting: endpoint service runtime Payload:map[]}] VersionInfo:{Name: Meta:map[]}}
endpoint_security_test.go:376: endpoint component is not Healthy: current state: {ID:endpoint-default Name:endpoint State:STARTING Message:Starting: endpoint service runtime Units:[{UnitID:endpoint-default UnitType:OUTPUT State:STARTING Message:Starting: endpoint service runtime Payload:map[]} {UnitID:endpoint-default-2475a0b0-cbb0-4bc2-aa60-c9689bd6cc16 UnitType:INPUT State:STARTING Message:Starting: endpoint service runtime Payload:map[]}] VersionInfo:{Name: Meta:map[]}}
...
endpoint_security_test.go:375:
Error Trace: /opt/buildkite-agent/builds/bk-agent-prod-gcp-1750468221167740260/elastic/elastic-agent/testing/integration/endpoint_security_test.go:375
/opt/buildkite-agent/builds/bk-agent-prod-gcp-1750468221167740260/elastic/elastic-agent/testing/integration/endpoint_security_test.go:118
Error: Condition never satisfied
Test: TestUpgradeAgentWithTamperProtectedEndpoint_RPM/Install_same_version_over_the_installed_agent
Messages: Endpoint component or units are not healthy after the upgrade.
endpoint_security_test.go:376: Error getting agent state: rpc error: code = Canceled desc = context canceled
endpoint_security_test.go:186: Uninstalling endpoint with the following uinstall token: 631b8e60dff444d08defe6e9477a5135
endpoint_security_test.go:192: Endpoint is successfully uninstalled by the cleanup function
fixture_install.go:582: [test TestUpgradeAgentWithTamperProtectedEndpoint_RPM/Install_same_version_over_the_installed_agent] Inside fixture installRpm cleanup function
fixture_install.go:587: running 'sudo systemctl stop elastic-agent'
fixture_install.go:593: running 'sudo rpm -e elastic-agent'
fixture_install.go:600: removing installed agent files
fixture.go:1427: Temporary directory "/tmp/TestUpgradeAgentWithTamperProtectedEndpoint_RPM-Install_same_version_over_the_installed_agent1824277952" preserved for investigation/debugging
--- FAIL: TestUpgradeAgentWithTamperProtectedEndpoint_RPM/Install_same_version_over_the_installed_agent (386.38s)
=== FAIL: testing/integration TestUpgradeAgentWithTamperProtectedEndpoint_RPM (784.66s)