Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker Compose V2 config command does not include version tag in the output #8796

Closed
kofarrelly opened this issue Oct 14, 2021 · 12 comments
Closed

Comments

@kofarrelly
Copy link

Docker Compose V2 config command does not include version tag in the output

Steps to reproduce the issue:

  1. Create a docker compose file including a version tag
  2. Run docker-compose --file=docker-compose.yml --env-file=all.env config

Describe the results you received:
Output produced does not include the version: tag so Docker defaults to config v1 and fails

Describe the results you expected:
Output should include the version: tag from the compose file

Additional information you deem important (e.g. issue happens only occasionally):
Works as expected with version 1.29.2

Output of docker compose version:

Docker Compose version v2.0.1

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.6.3-docker)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 51
  Running: 13
  Paused: 0
  Stopped: 38
 Images: 14
 Server Version: 20.10.9
 Storage Driver: devicemapper
  Pool Name: docker-8:3-77971944-pool
  Pool Blocksize: 65.54kB
  Base Device Size: 10.74GB
  Backing Filesystem: xfs
  Udev Sync Supported: true
  Data file: /dev/loop0
  Metadata file: /dev/loop1
  Data loop file: /opt/docker/devicemapper/devicemapper/data
  Metadata loop file: /opt/docker/devicemapper/devicemapper/metadata
  Data Space Used: 6.326GB
  Data Space Total: 107.4GB
  Data Space Available: 59.06GB
  Metadata Space Used: 27.85MB
  Metadata Space Total: 2.147GB
  Metadata Space Available: 2.12GB
  Thin Pool Minimum Free Space: 10.74GB
  Deferred Removal Enabled: true
  Deferred Deletion Enabled: true
  Deferred Deleted Device Count: 0
  Library Version: 1.02.170-RHEL7 (2020-03-24)
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: 
  Is Manager: true
  ClusterID: 
  Managers: 1
  Nodes: 1
  Default Address Pool: 10.0.0.0/8
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 10.100.1.57
  Manager Addresses:
   10.100.1.57:2377
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 3.10.0-1160.15.2.el7.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 23.39GiB
 Name: 
 ID: 
 Docker Root Dir: /opt/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:

 Registry Mirrors:

 Live Restore Enabled: false

Additional environment details:

@raspy
Copy link

raspy commented Nov 5, 2021

This is important as the output fails to be usable for docker stack deploy, e.g.:

$ docker-compose -f .\ha.yml -f .\application.yml -f .\local-db.yml -f .\local-testing.yml config | docker stack deploy -c - rt
unsupported Compose file version: 1.0

Lack of version tag makes parser default it to 1.0, which is unsuitable for Docker Swarm.

@ndeloof
Copy link
Contributor

ndeloof commented Nov 5, 2021

docker stack is not compliant with the compose specification, there's nothing we can do here.
Please vote for docker/cli#2527 :P

@ndeloof ndeloof closed this as completed Nov 5, 2021
@dazinator
Copy link

dazinator commented Jun 9, 2022

@ndeloof
I've just started seeing this issue, interestingly i've updated to docker-compose 2.6.0 - not sure what version I was using prior but all was working.

Looks like others have also encountered this in the past based on: https://stackoverflow.com/questions/57406409/unsupported-compose-file-version-1-0-even-when-i-have-the-right-compatability

I'm not understanding the proposed solution that this issue was closed in favour of.. If its not possible to include this version tag in the output any longer by default, then perhaps a new arg could be added to the docker-compose config cli command so that we can force it to output the version header and allow our workflows to work. e.g docker-compose config --output-version or something. I say this because the wider issue looks more systemic and in the meantime workflows have been broken..

@ndeloof
Copy link
Contributor

ndeloof commented Jun 9, 2022

we can't force a version header, as there's no version we can use here: this is not 3.10 or anything else, this is just "the compose specification". The main question is: why would you try to combine docker-compose with another command?
docker stack can read a compose file from multiple yaml files, so you don't need docker-compose config for this purpose

@raspy
Copy link

raspy commented Jun 9, 2022

My use case was that different yaml files were written in different versions (some are version 2) and docker stack rejects such deployments. Passing them through docker-compose config allowed for version "normalization" to have single version taken from the very first yaml file.

@ndeloof
Copy link
Contributor

ndeloof commented Jun 9, 2022

docker-compose v1 did not supported merging compose file with distinct version (see

"Version mismatch: file {} specifies version {} but "
) , this is only possible with compose v2 thanks to the compose specification.

@dazinator
Copy link

dazinator commented Jun 9, 2022

@ndeloof

The main question is: why would you try to combine docker-compose with another command?
docker stack can read a compose file from multiple yaml files, so you don't need docker-compose config for this purpose

Even after manually appending the version header myself, stack deploy hit another error

services.reverse-proxy.ports.0.published must be a integer

Looking at the output, compose outputs the published ports as strings now:

ports:
    - mode: host
      target: 80
      published: "80"
    - mode: host
      target: 443
      published: "443"
    - target: 8082
      published: "8082"

Despite them being specified as int in my source compose file.
So I am guessing this is just another place where the spec has diverged.

However to come back to your question:

When running docker-compose.yml files locally using docker-compose up for example, .env files can be specified, and these can be used in the docker-compose.yml file as parameters. The .env file is usually tailored for local development environment. However when plugging in a CI/CD process to deploy this to a swarm, the .env paramaters used in the compose file need to be set differently. Our way of solving this is that the CI/CD server creates its own .env file called something like "deploy.env" and we pre-process the compose.yml file using that .env file instead. We then feed that into stack deploy, and it means we have a single compose file, that we are able to run locally, or deploy to a swarm or any onward environment by setting an appropriate env file first. We also use local swarms when running locally, so we can also deploy locally using this mechanism rather than using docker-compose up.

Are you saying docker stack deploy supports .env files in the same way now so docker compose config is no longer necessary? Or should we be doing something else?

@dazinator
Copy link

dazinator commented Jun 9, 2022

Note: others have discussed this here: https://stackoverflow.com/questions/58666953/how-can-i-pass-the-variables-i-have-placed-in-the-env-file-to-the-containers-in#:~:text=Loading%20the.env%20file%20is%20a%20feature%20of%20docker-compose,set%20%2Ba%20docker%20stack%20deploy%20-c%20docker-compose.yml%20stack_name

Note: We opted to use the docker compose config way as that works in powershell core, allowing us to have a single deployment script which works on windows and linux.

Another option proposed there is to load the .env file into the shell prior to doing docker stack deploy with something like this:

set -a; . ./.env; set +a
envsubst <docker-compose.yml >docker-compose.processed.yml

However set -a; . ./.env; set +a seems shell specific, meaning we'd have to have a different script for linux vs windows environments. If there is a way to achieve this in powershell core perhaps that would solve our problem, but not currently sure what that is

@raspy
Copy link

raspy commented Jun 9, 2022

@ndeloof you made me doubt, so I double checked and it works for me; it keeps version from the first file in the chain:

$ grep version artifactory-ha.yml
version: '2'
$ grep version local-db.yml
version: '3.2'
$ docker-compose -f artifactory-ha.yml -f local-db.yml config | grep version
version: '2'
$ docker-compose -f local-db.yml -f artifactory-ha.yml config | grep version
version: '3.2'
$ docker-compose --version
docker-compose version 1.29.2, build 5becea4c

@dazinator
Copy link

Ok I have found a workaround to replace the reliance on docker-compose config as a preprocessor.

In my powershell script I now do the following to set environment variables from the .env file myself

$envFileContent = Get-Content -Path $envFileName
    ForEach ($line in $envFileContent) 
    {
        if($line)
        {
            $varNameValSplit = $line.Split("=")
            if($varNameValSplit.Length -eq 2)
            {
                $varName = $varNameValSplit[0]
                $varVal = $varNameValSplit[1]
                [Environment]::SetEnvironmentVariable($varName, $varVal)
                Write-Output "Set var $varName"
            }           
        }     
    }
    docker stack deploy -c $composeFileName --with-registry-auth $stackName

@Radiergummi
Copy link

@dazinator So if I understand this correctly, you still use the docker-compose file with the interleaved environment variables (e.g. image: foo:${TAG}), and by setting the variables locally, on the machine that deploys the stack to the cluster, those get replaced with their values during the stack deploy?

I'm facing the same, annoying, unnecessary, chain of problems with different docker components you did, and would really like to confirm this simple solution works.
Thanks a bunch!

@dazinator
Copy link

@dazinator So if I understand this correctly, you still use the docker-compose file with the interleaved environment variables (e.g. image: foo:${TAG}), and by setting the variables locally, on the machine that deploys the stack to the cluster, those get replaced with their values during the stack deploy?

I'm facing the same, annoying, unnecessary, chain of problems with different docker components you did, and would really like to confirm this simple solution works. Thanks a bunch!

Yes, setting the environment variables in the same process that runs the docker stack deploy command worked for me (in my case pwsh)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants