MinIO deployment write permissions issue

[ericgtkb]

Validation Checklist

• I confirm that this is a Kubeflow-related issue.
• I am reporting this in the appropriate repository.
• I have followed the Kubeflow installation guidelines.
• The issue report is detailed and includes version numbers where applicable.
• This issue pertains to Kubeflow development.
• I am available to work on this issue.
• You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-platform.

Version

master

Detailed Description

First time posting here, just wanted to quickly thank y'all for maintaining this project.

Not sure if this issue belongs to this repo or kubeflow/pipelines. Will move the issue there if it's more appropriate there.

I was trying to deploy the master branch but got an issue where the minio pod for the kubeflow pipeline deployment suck in CrashLoopBackOff. This is a bare-metal kubeadm cluster with longhorn for storage.

After looking into the minio pod's log it seems to be some permission issue.

ERROR Unable to initialize backend: Unable to write to the backend
      > Please ensure MinIO binary has write permissions for the backend

After checking out v1.9.1 and redeploying, everything worked, so I suspect the issue is related to the newer changes.

Steps to Reproduce

Run the single-command installation with the latest master branch to reproduce.

while ! kustomize build example | kubectl apply --server-side --force-conflicts -f -; do echo "Retrying to apply resources"; sleep 20; done

Screenshots or Videos (Optional)

MinIO pod logs:

$ kubectl get logs -n kubeflow minio-4d823345b3-xzz73
ERROR Unable to initialize backend: Unable to write to the backend
      > Please ensure MinIO binary has write permissions for the backend
      HINT:
        Verify if MinIO binary is running as the same user who has write permissions for the backend

Since v1.9.1 worked, I also ran git diff for the minio manifests. Looks like maybe the new changes to the minio-deployment.yaml is the root cause?

((HEAD detached at v1.9.1))$ git diff master -- apps/pipeline/upstream/third-party/minio
diff --git a/apps/pipeline/upstream/third-party/minio/base/minio-deployment.yaml b/apps/pipeline/upstream/third-party/minio/base/minio-deployment.yaml
index 6025517a..a1bd9630 100644
--- a/apps/pipeline/upstream/third-party/minio/base/minio-deployment.yaml
+++ b/apps/pipeline/upstream/third-party/minio/base/minio-deployment.yaml
@@ -34,16 +34,6 @@ spec:
         name: minio
         ports:
         - containerPort: 9000
-        securityContext:
-          allowPrivilegeEscalation: false
-          seccompProfile:
-            type: RuntimeDefault
-          runAsNonRoot: true
-          runAsUser: 1000
-          runAsGroup: 0
-          capabilities:
-            drop:
-            - ALL
         volumeMounts:
         - mountPath: /data
           name: data

[ericgtkb]

Didn't see #3040 when I was creating this issue. Closing this issue.