error removing var/run to make way for new symlink: unlinkat #1002

hadrien-toma · 2020-01-25T19:28:58Z

Actual behavior
Can not build the image, was working and suddenly was not.

Expected behavior
Be able to build the image.

To Reproduce
File system:

issue
├── context
│   ├── builder.ash
│   └── Dockerfile
└── index.bash

1 directory, 3 files

Create an issue directory
Create the following index.bash file under the issue directory:

#!/bin/bash

hereDir=`dirname $0 | while read a; do cd $a && pwd && break; done `

pids=""
rc=0

trap 'sigintTrap' 2

sigintTrap() {
	if [ "${pids}X" != "X" ]; then
		kill -9 ${pids}
	fi
	exit 2
}

minikube delete
minikube start

minikube mount ${hereDir}:/issue &
pids="${pids} $!"

minikube dashboard &
pids="${pids} $!"

cat <<EOF | kubectl apply --force -f -
apiVersion: batch/v1
kind: Job
metadata:
  name: kaniko-job
spec:
  activeDeadlineSeconds: 1000
  template:
    metadata:
      name: kaniko-job
    spec:
      containers:
      - args:
        - -c
        - |
          /busybox/sh <<'EOF'

          /kaniko/executor \
            --context "/issue/context" \
            --no-push \
            --dockerfile "/issue/context/Dockerfile" \
            --insecure

          EOF
        command: [ "/busybox/sh" ]
        name: kaniko-job
        image: gcr.io/kaniko-project/executor:debug
        volumeMounts:
        - name: issue
          mountPath: /issue
      restartPolicy: OnFailure
      volumes:
      - name: issue
        hostPath:
          path: /issue
EOF

for pid in ${pids}; do
	wait ${pid} || let "rc=1"
done

if [ "${rc}" == "1" ]; then
	exit 1
fi

Create a context directory in the issue directory
Create the following Dockerfile file under the context directory:

FROM alpine:latest

COPY ./builder.ash /builder.ash

RUN chmod +x /builder.ash && /builder.ash ; rm -rf /builder.ash

Create the following builder.ash file under the context directory:

#!/bin/ash

echo "Building..."

Give execution permission to the index.bash script: chmod +x issue/index.bash
Run the index.bash script: ./issue/index.bash
I get the following in the pod:

�[36mINFO�[0m[0000] Resolved base name alpine:latest to alpine:latest 
�[36mINFO�[0m[0000] Resolved base name alpine:latest to alpine:latest 
�[36mINFO�[0m[0000] Retrieving image manifest alpine:latest      
�[36mINFO�[0m[0001] Retrieving image manifest alpine:latest      
�[36mINFO�[0m[0002] Built cross stage deps: map[]                
�[36mINFO�[0m[0002] Retrieving image manifest alpine:latest      
�[36mINFO�[0m[0003] Retrieving image manifest alpine:latest      
�[36mINFO�[0m[0004] Unpacking rootfs as cmd COPY ./builder.ash /builder.ash requires it. 
error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/secrets/kubernetes.io/serviceaccount/..2020_01_25_19_26_04.767922246: read-only file system

Additional Information

> docker version && minikube version && kubectl version && uname -a

Client: Docker Engine - Community
Version:           19.03.5
API version:       1.40
Go version:        go1.12.12
Git commit:        633a0ea838
Built:             Wed Nov 13 07:29:52 2019
OS/Arch:           linux/amd64
Experimental:      false

Server: Docker Engine - Community
Engine:
 Version:          19.03.5
 API version:      1.40 (minimum version 1.12)
 Go version:       go1.12.12
 Git commit:       633a0ea838
 Built:            Wed Nov 13 07:28:22 2019
 OS/Arch:          linux/amd64
 Experimental:     false
containerd:
 Version:          1.2.10
 GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
runc:
 Version:          1.0.0-rc8+dev
 GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
docker-init:
 Version:          0.18.0
 GitCommit:        fec3683
minikube version: v1.6.2
commit: 54f28ac5d3a815d1196cd5d57d707439ee4bb392
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-18T23:30:10Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:12:17Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}
Linux mewtwo-ubuntu 5.3.0-26-generic #28~18.04.1-Ubuntu SMP Wed Dec 18 16:40:14 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

The text was updated successfully, but these errors were encountered:

eduscrakozabrus · 2020-01-26T11:39:51Z

I am have this error in gitlab ci

.gitlab-ci.yaml
build:
stage: build
image:
name: gcr.io/kaniko-project/executor:debug
entrypoint: [""]
script:
- echo "{"auths":{"$CI_REGISTRY":{"username":"$CI_REGISTRY_USER","password":"$CI_REGISTRY_PASSWORD"}}}" > /kaniko/.docker/config.json
- /kaniko/executor --cache=true --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME-$CI_COMMIT_SHA

INFO[0004] Unpacking rootfs as cmd ADD Gemfile* /app/ requires it.
60 error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/secrets/kubernetes.io/serviceaccount/..2020_01_26_11_37_35.908221403: read-only file system

eduscrakozabrus · 2020-01-26T11:42:49Z

solution worked for me
change version of kaniko: gcr.io/kaniko-project/executor:debug-v0.15.0

hadrien-toma · 2020-01-26T12:13:54Z

Thank you @eduscrakozabrus, worked for me too 🙏. So is this a bug in the newest version?

followsound · 2020-01-26T12:35:16Z

I've also been having this issue - this solved it

axyjo · 2020-01-26T14:33:47Z

Looks like a duplicate of #1001

bozzo · 2020-01-27T16:10:45Z

I had the same issue, i've fixed it by adding --additional-whitelist /var/run/secrets with the gcr.io/kaniko-project/executor:debug image.

cvgw · 2020-01-27T17:19:36Z

Strong recommendation to everyone; use an explicitly tagged release of the kaniko image. Don't use latest in your critical systems. ~~This change has not been released and is only affecting people using latest~~ I was incorrect, latest is only built on release. This issue is only affecting people who are using the debug tag which is built on every commit to master. But still, use an explicit version!

rokerkony · 2020-01-27T17:24:18Z

@cvgw thx for the suggestion, it makes totally sense. It would be great to mention it in the readme as it always advises to use :latest

tejal29 · 2020-01-28T00:03:55Z

closing this as dupe of #1001

fischerman · 2020-02-04T07:36:18Z

@tejal29 What is the solution to this problem? #1001 seems to be related to CloudBuild (which I have no knowledge about). This issue here seems to be related to Kubernetes. AFAIK whitelisting /var/run exposes the service account token in the image, which is a security thread. Not whitelisting breaks the build.

bnouvelbmll · 2020-02-04T10:11:26Z

Bug still present in debug-v0.17.0 - reverting to earlier version fix the build but not ideal.

'''
error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/secrets/kubernetes.io/serviceaccount/..2020_02_04_10_04_34.525459306: read-only file system
'''

consideRatio · 2020-02-04T12:59:25Z

I had the same issue, i've fixed it by adding --additional-whitelist /var/run/secrets with the gcr.io/kaniko-project/executor:debug image.

@bozzo, where is this parameter passed? To kaniko executor? It didn't recognize that flag. I'm confused about that.

tomislater · 2020-02-04T13:03:36Z

I have the same issue (0.17.0):

with --whitelist-var-run=false

error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/secrets/kubernetes.io/serviceaccount/..2020_02_04_12_59_06.412495393: read-only file system

0.16.0 works btw.

bozzo · 2020-02-04T13:25:04Z

@consideRatio It was an option added to the debug image. I don't know if it still present. btw, executor:debug-v0.17.0 worked for me with Gitlab Runners on Kubernetes, by running:

/kaniko/executor --context "$DOCKER_CONTEXT_PATH" --dockerfile "$DOCKER_DOCKERFILE_PATH/Dockerfile" --destination "$DOCKER_IMAGE" --cache --cache-dir="$CACHE_DIR" --verbosity info $DOCKER_BUILD_ARGS

consideRatio · 2020-02-04T14:37:45Z

@bozzo ah thanks for helping me out! I tried the following that does work for v0.16.0 on my gitlab-runner on k8s, but it fails for me with kaniko v0.17.0 with the error of this issue's title.

  script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    # Kaniko reference:
    # https://github.com/GoogleContainerTools/kaniko/blob/master/README.md#additional-flags
    - >
      /kaniko/executor
      --context $CI_PROJECT_DIR/images
      --dockerfile $CI_PROJECT_DIR/images/Dockerfile
      --destination $CI_REGISTRY_IMAGE:cpu-$CI_COMMIT_SHORT_SHA

--whitelist-var-run
Ignore /var/run when taking image snapshot. Set it to false to preserve /var/run/* in destination image. (Default true).

I think what happens in my case, is that the COPY statement in my Dockerfile triggers some similar logic that would require "whitelisting" but it does it outside the scope of a image snapshot situation, and hence the --whitelist-var-run flag isn't relevant.

 INFO[0004] Unpacking rootfs as cmd COPY base-notebook-assets/fix-permissions /usr/local/bin/fix-permissions requires it. 
 error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/secrets/kubernetes.io/serviceaccount/..2020_02_04_14_26_54.673390439: read-only file system
 ERROR: Job failed: command terminated with exit code 1

ARG BASE_CONTAINER=ubuntu:bionic-20200112
FROM $BASE_CONTAINER

COPY base-notebook-assets/fix-permissions /usr/local/bin/fix-permissions

cvgw · 2020-02-04T21:03:28Z

Some info which will hopefully help clear up confusion:

tldr;
<= v0.16.0 = no flag
v0.16.0 <= debug < v0.17.0 = --additional-whitelist=/var/run
v0.17.0 = no flag

prior to v0.16.0 kaniko always whitelisted the path /var/run; meaning that /var/run and any sub paths were ignored.

between v0.16.0 and v0.17.0 introduced the --additional-whitelist flag and removed /var/run from the hardcoded whitelist. The default for --additional-whitelist was an empty list meaning that to get the same behavior as prior to v0.16.0 users needed to supply --additional-whitelist=/var/run. If you were using the debug tag during that time you may have seen the --additional-whitelist flag.

v0.17.0 removed the --additional-whitelist flag and added a new flag --whitelist-var-run. The default of the new flag --whitelist-var-run is true. This means that the default behavior in v0.17.0 (with respect to whitelisting /var/run) is the same as prior to v0.16.0

Updated, thanks @consideRatio for pointing out the mistake

consideRatio · 2020-02-04T21:36:38Z

Thank you for an excellent summary @cvgw!

For me, the --additional-whitelist within the debug-v0.16.0 image isn't recognized. It also didn't recognize the --whitelist-var-run=true or false option.

Using debug-v0.17.0 I crash directly without specifying any flags with this issue's titles as an error. It also crash like this if i specify --whitelist-var-run=true or false - a flag that it now recognizes.

The debug-v0.16.0 run that didn't fail due to this, later failed due to issues that i hope were resolved in debug-v0.17.0.

   ...
   File "/opt/conda/lib/python3.6/os.py", line 220, in makedirs
     mkdir(name, mode)
 PermissionError: [Errno 13] Permission denied: '/home/jovyan/.ipyvolume'

tejal29 · 2020-02-04T22:11:26Z

hey folks, Sorry we had a bad release v0.17.0.
I fixed the /var/run issue and released a new v0.17.1 patch release.

tejal29 · 2020-02-04T22:12:37Z

Bug still present in debug-v0.17.0 - reverting to earlier version fix the build but not ideal.

'''
error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/secrets/kubernetes.io/serviceaccount/..2020_02_04_10_04_34.525459306: read-only file system
'''

@bnouvelbmll This should now be fixed in v0.17.1 latest release pushed an hour ago.

tejal29 · 2020-02-04T22:15:08Z

@tejal29 What is the solution to this problem? #1001 seems to be related to CloudBuild (which I have no knowledge about). This issue here seems to be related to Kubernetes. AFAIK whitelisting /var/run exposes the service account token in the image, which is a security thread. Not whitelisting breaks the build.

@fischerman Sorry for the inconvenience. We added the flag --whitelist-var-run which whitelisted /var/run by default however there was bug #1025 due to which it got added after extracting base image. We fixed the bug and now your build should work fine.

tejal29 · 2020-02-04T22:16:30Z

I have the same issue (0.17.0):

with --whitelist-var-run=false

error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/secrets/kubernetes.io/serviceaccount/..2020_02_04_12_59_06.412495393: read-only file system

0.16.0 works btw.

@tomislater This is should be fixed now in v0.17.1. . We added the flag --whitelist-var-run which whitelisted /var/run by default however there was bug #1025 due to which it got added after extracting base image. We fixed the bug and now your build should work fine.

consideRatio · 2020-02-04T22:42:52Z

@tejal29 I wanted to confirm that v0.17.1 doesn't have the issue, and if i actively specify --whitelist-var-run=false I can get the same error again - confirming the flag's actively returning to the other behavior.

Thank you for your work on this! ❤️ 🎉!

olivierboudet · 2020-02-14T15:42:13Z

it is fixed for me also with v0.17.1

hadrien-toma · 2020-02-14T16:10:54Z

Fixed for me too, I close the issue, thanks all 👍 🚀 !

cvgw mentioned this issue Jan 27, 2020

Fix #506 add option additional-whitelist #975

Merged

4 tasks

tejal29 closed this as completed Jan 28, 2020

tejal29 mentioned this issue Jan 28, 2020

Revert additional whitelist flag #1009

Merged

tejal29 mentioned this issue Feb 4, 2020

update initialWhitelist instead of whitelist #1025

Merged

tejal29 mentioned this issue Feb 4, 2020

Version can be in the process of being released GoogleContainerTools/skaffold#3634

Merged

tejal29 reopened this Feb 4, 2020

hadrien-toma closed this as completed Feb 14, 2020

bitsofinfo mentioned this issue Jun 8, 2020

0.23 - error removing var/run to make way for new symlink #1306

Closed

imjasonh mentioned this issue Jan 5, 2022

Snapshotting is taking > 10 minutes #970

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

error removing var/run to make way for new symlink: unlinkat #1002

error removing var/run to make way for new symlink: unlinkat #1002

hadrien-toma commented Jan 25, 2020 •

edited

Loading

eduscrakozabrus commented Jan 26, 2020

eduscrakozabrus commented Jan 26, 2020

hadrien-toma commented Jan 26, 2020

followsound commented Jan 26, 2020

axyjo commented Jan 26, 2020

bozzo commented Jan 27, 2020

cvgw commented Jan 27, 2020 •

edited

Loading

rokerkony commented Jan 27, 2020

tejal29 commented Jan 28, 2020 •

edited

Loading

fischerman commented Feb 4, 2020

bnouvelbmll commented Feb 4, 2020

consideRatio commented Feb 4, 2020

tomislater commented Feb 4, 2020 •

edited

Loading

bozzo commented Feb 4, 2020

consideRatio commented Feb 4, 2020

cvgw commented Feb 4, 2020 •

edited

Loading

consideRatio commented Feb 4, 2020 •

edited

Loading

tejal29 commented Feb 4, 2020

tejal29 commented Feb 4, 2020

tejal29 commented Feb 4, 2020

tejal29 commented Feb 4, 2020

consideRatio commented Feb 4, 2020 •

edited

Loading

olivierboudet commented Feb 14, 2020

hadrien-toma commented Feb 14, 2020

error removing var/run to make way for new symlink: unlinkat #1002

error removing var/run to make way for new symlink: unlinkat #1002

Comments

hadrien-toma commented Jan 25, 2020 • edited Loading

eduscrakozabrus commented Jan 26, 2020

eduscrakozabrus commented Jan 26, 2020

hadrien-toma commented Jan 26, 2020

followsound commented Jan 26, 2020

axyjo commented Jan 26, 2020

bozzo commented Jan 27, 2020

cvgw commented Jan 27, 2020 • edited Loading

rokerkony commented Jan 27, 2020

tejal29 commented Jan 28, 2020 • edited Loading

fischerman commented Feb 4, 2020

bnouvelbmll commented Feb 4, 2020

consideRatio commented Feb 4, 2020

tomislater commented Feb 4, 2020 • edited Loading

bozzo commented Feb 4, 2020

consideRatio commented Feb 4, 2020

cvgw commented Feb 4, 2020 • edited Loading

consideRatio commented Feb 4, 2020 • edited Loading

tejal29 commented Feb 4, 2020

tejal29 commented Feb 4, 2020

tejal29 commented Feb 4, 2020

tejal29 commented Feb 4, 2020

consideRatio commented Feb 4, 2020 • edited Loading

olivierboudet commented Feb 14, 2020

hadrien-toma commented Feb 14, 2020

hadrien-toma commented Jan 25, 2020 •

edited

Loading

cvgw commented Jan 27, 2020 •

edited

Loading

tejal29 commented Jan 28, 2020 •

edited

Loading

tomislater commented Feb 4, 2020 •

edited

Loading

cvgw commented Feb 4, 2020 •

edited

Loading

consideRatio commented Feb 4, 2020 •

edited

Loading

consideRatio commented Feb 4, 2020 •

edited

Loading