tools: use docker compose and quiet for pulling when using docker or docker compose #8594

v1v · 2025-06-19T20:25:13Z

What does this PR do?

Use docker compose
Use docker compose --progress=plain (see https://docs.docker.com/reference/cli/docker/compose/)
Use docker run --quiet (see https://docs.docker.com/reference/cli/docker/container/run)
Use docker build --progress=plain (see https://docs.docker.com/reference/cli/docker/buildx/build/#options)

Why is it important?

Docker-compose reached EoL in 2023, see https://www.docker.com/blog/announcing-compose-v2-general-availability/#proposed-compose-v1-end-of-lie-timeline.
Avoid too much verbose output in the logs when pulling docker images.

See

vs

Checklist

I have read and understood the pull request guidelines of this project.
My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in ./changelog/fragments using the changelog tool
I have added an integration test or an E2E test

Disruptive User Impact

How to test this PR locally

CI

Related issues

Relates to Enable go integrations tests on Windows beats#44324

Questions to ask yourself

How are we going to support this in production?
How are we going to measure its adoption?
How are we going to debug this?
What are the metrics I should take care of?
...

mergify · 2025-06-19T20:25:56Z

This pull request does not have a backport label. Could you fix it @v1v? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
backport-active-all is the label that automatically backports to all active branches.
backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

v1v · 2025-06-19T20:30:26Z

dev-tools/mage/common.go

@@ -209,16 +209,6 @@ func dockerInfo() (*DockerInfo, error) {
 	return &info, nil
 }

-// HaveDockerCompose returns an error if docker-compose is not found on the


Assuming docker compose is available, so by validating docker I think we can remove this condition. It's not available, it will fail regardless.

dev-tools/mage/integtest_docker.go

…ithub.com/v1v/elastic-agent into feature/remove-verbose-pulling-containers * 'feature/remove-verbose-pulling-containers' of https://github.com/v1v/elastic-agent: Update dev-tools/mage/integtest_docker.go

elasticmachine · 2025-06-20T07:04:45Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

pkoutsovasilis

In general I am not in favour of this change as I don't wanna lose the info of which images I pull to build my packages especially since it involves golangcross-build, which has many different variants 🙂

pchila · 2025-06-20T08:01:04Z

@v1v
docker-compose is not really used in CI and we can use docker compose when we need it so thank you for removing references to obsolete tools.

I am not so sure about reducing the docker logs though, since those have been useful to debug build/packaging issues in the past; removing part of those logs means that in order to investigate issues, those must be reproduced and analyzed off buildkite or with a dedicated PR.
What is the impact of those docker logs that we are trying to mitigate/remove?

dliappis

I am skeptical about the use of --quiet; while I understand the motivation to reduce verbosity related to the individual layers, by making all stdout quiet we are losing previous information about the docker image (e.g. golang crossbuild) that got pulled, thus making troubleshooting unnecessarily harder.

v1v · 2025-06-20T09:22:53Z

My concern is that those details are too verbose and unstructured; however, I think using --progress=plain mitigates the former.

I understand your view, probably, we could say if a retry then enable the prior approach, otherwise use --quiet by default. Then, we know it's easy to debug.

However, maybe I'm overthinking, I feel you prefer to do a step back and only make the changes for:

docker compose --progress=plain
docker build --progres=plain

That's something I took from elastic/golang-crossbuild#528

If no objections, I'll remove the --quiet

v1v · 2025-06-20T09:24:47Z

dev-tools/mage/crossbuild.go

@@ -294,7 +294,7 @@ func (b GolangCrossBuilder) Build() error {
 	// container. So we ensure that all path separators are `/`.
 	buildCmd = filepath.ToSlash(buildCmd)

-	dockerRun := sh.RunCmd("docker", "run")
+	dockerRun := sh.RunCmd("docker", "run", "--quiet")


According to the docs:

$ docker run --help ... --pull string Pull image before running ("always", "missing", "never") (default "missing") -q, --quiet Suppress the pull output

See https://docs.docker.com/reference/cli/docker/container/run/

dev-tools/mage/integtest_docker.go

v1v · 2025-06-20T09:25:54Z

dev-tools/mage/pkgtypes.go

@@ -785,7 +785,7 @@ func runFPM(spec PackageSpec, packageType PackageType) error {
 	}
 	spec.OutputFile = packageType.AddFileExtension(filepath.Join(distributionsDir, outputFile))

-	dockerRun := sh.RunCmd("docker", "run")
+	dockerRun := sh.RunCmd("docker", "run", "--quiet")


See https://github.com/elastic/elastic-agent/pull/8594/files#r2158465644

dev-tools/mage/integtest_docker.go

pkoutsovasilis · 2025-06-20T09:30:47Z

My concern is that those details are too verbose and unstructured; however, I think using --progress=plain mitigates the former.

--progress=plain makes sense and most probably makes the output more readable which is something good IMO

I understand your view, probably, we could say if a retry then enable the prior approach, otherwise use --quiet by default. Then, we know it's easy to debug.

Again sorry but I don't agree, here you seem to care only for failures, but we would also like to be in the position of looking the docker build for green steps.

However, maybe I'm overthinking, I feel you prefer to do a step back and only make the changes for:

docker compose --progress=plain

docker build --progres=plain

That's something I took from elastic/golang-crossbuild#528

Again to understand the purpose of this PR, I am gonna echo the same question of @pchila; What is the impact of those docker logs that we are trying to mitigate/remove? 🙂

elastic-sonarqube · 2025-06-20T10:47:45Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

elasticmachine · 2025-06-20T11:06:52Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 512ba55

Failed CI Steps

Unit tests - macOS 15 ARM

cc @v1v

v1v · 2025-06-20T11:27:53Z

What is the impact of those docker logs that we are trying to mitigate/remove?

It's about ergonomics and usability. Seeing logs regarding an interactive shell is certainly misleading and therefore makes the UI experience really bad - people need to download the log as it's massive.

For instance, see https://buildkite.com/elastic/elastic-agent-extended-testing/builds/9082#01978c8c-60aa-4e4a-a0a9-1c61f59e7329/6-7

Then, if you click on open https://buildkite.com/organizations/elastic/pipelines/elastic-agent-extended-testing/builds/9082/jobs/01978c8c-60aa-4e4a-a0a9-1c61f59e7329/log

Maybe, it's me, but having a UI that cannot show relevant details because logs are massive sounds like something to improve. Hence this PR. However, I don't use this CI massively. Probably I'm biased here.

If you think the changes are not relevant, I'm happy to change the scope and focus only on the docker compose if agreed, or close this one.

tools: use docker compose and quiet for pulling

0b6c617

v1v added the Cleanup label Jun 19, 2025

mergify bot assigned v1v Jun 19, 2025

-q, --quiet Suppress the pull output

f6beb7a

v1v changed the title ~~tools: use docker compose and quiet for pulling~~ tools: use docker compose and quiet for pulling when using docker or docker compose Jun 19, 2025

v1v added skip-changelog backport-active-all Automated backport with mergify to all the active branches labels Jun 19, 2025

v1v commented Jun 19, 2025

View reviewed changes

dev-tools/mage/integtest_docker.go Outdated Show resolved Hide resolved

v1v and others added 3 commits June 19, 2025 22:30

Update dev-tools/mage/integtest_docker.go

4abceb9

fix other docker commands

5b2cb7e

Merge branch 'feature/remove-verbose-pulling-containers' of https://g…

887bf77

…ithub.com/v1v/elastic-agent into feature/remove-verbose-pulling-containers * 'feature/remove-verbose-pulling-containers' of https://github.com/v1v/elastic-agent: Update dev-tools/mage/integtest_docker.go

v1v marked this pull request as ready for review June 20, 2025 06:31

v1v requested a review from a team as a code owner June 20, 2025 06:31

v1v requested review from michalpristas, michel-laterman, pkoutsovasilis and a team June 20, 2025 06:31

pierrehilbert added the Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team label Jun 20, 2025

pkoutsovasilis reviewed Jun 20, 2025

View reviewed changes

dliappis reviewed Jun 20, 2025

View reviewed changes

v1v commented Jun 20, 2025

View reviewed changes

dev-tools/mage/integtest_docker.go Outdated Show resolved Hide resolved

v1v commented Jun 20, 2025

View reviewed changes

dev-tools/mage/integtest_docker.go Outdated Show resolved Hide resolved

Apply suggestions from code review

512ba55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

tools: use docker compose and quiet for pulling when using docker or docker compose #8594

tools: use docker compose and quiet for pulling when using docker or docker compose #8594

Uh oh!

v1v commented Jun 19, 2025 •

edited

Loading

Uh oh!

mergify bot commented Jun 19, 2025

Uh oh!

v1v Jun 19, 2025

Uh oh!

Uh oh!

elasticmachine commented Jun 20, 2025

Uh oh!

pkoutsovasilis left a comment

Uh oh!

pchila commented Jun 20, 2025

Uh oh!

dliappis left a comment

Uh oh!

v1v commented Jun 20, 2025

Uh oh!

v1v Jun 20, 2025

Uh oh!

Uh oh!

v1v Jun 20, 2025

Uh oh!

Uh oh!

pkoutsovasilis commented Jun 20, 2025

Uh oh!

elastic-sonarqube bot commented Jun 20, 2025

Uh oh!

elasticmachine commented Jun 20, 2025

Uh oh!

v1v commented Jun 20, 2025

Uh oh!

Uh oh!

tools: use docker compose and quiet for pulling when using docker or docker compose #8594

Are you sure you want to change the base?

tools: use docker compose and quiet for pulling when using docker or docker compose #8594

Uh oh!

Conversation

v1v commented Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Disruptive User Impact

How to test this PR locally

CI

Related issues

Questions to ask yourself

Uh oh!

mergify bot commented Jun 19, 2025

Uh oh!

v1v Jun 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticmachine commented Jun 20, 2025

Uh oh!

pkoutsovasilis left a comment

Choose a reason for hiding this comment

Uh oh!

pchila commented Jun 20, 2025

Uh oh!

dliappis left a comment

Choose a reason for hiding this comment

Uh oh!

v1v commented Jun 20, 2025

Uh oh!

v1v Jun 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

v1v Jun 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pkoutsovasilis commented Jun 20, 2025

Uh oh!

elastic-sonarqube bot commented Jun 20, 2025

Quality Gate passed

Uh oh!

elasticmachine commented Jun 20, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

Uh oh!

v1v commented Jun 20, 2025

Uh oh!

Uh oh!

v1v commented Jun 19, 2025 •

edited

Loading