Minio for pipeline no longer starts on some clusters

[fraenkel]

Validation Checklist

• I confirm that this is a Kubeflow-related issue.
• I am reporting this in the appropriate repository.
• I have followed the Kubeflow installation guidelines.
• The issue report is detailed and includes version numbers where applicable.
• This issue pertains to Kubeflow development.
• I am available to work on this issue.
• You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-platform.

Version

master

Detailed Description

When the mino pod starts, the logs have the following error:

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ You are running an older version of MinIO released 5 years ago ┃
┃ Update: https://docs.min.io/docs/deploy-minio-on-kubernetes    ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

ERROR Unable to initialize backend: Unable to write to the backend
      > Please ensure MinIO binary has write permissions for the backend
      HINT:
        Verify if MinIO binary is running as the same user who has write permissions for the backend

The /data directory is

drwxr-xr-x    2 root     root          4096 Mar  6 23:10 data

which means given the new securityContext, there is no way to write to /data.

Easiest solution is to add

securityContext:
  fsGroup: 1000

Steps to Reproduce

Launch the pipeline app.

Screenshots or Videos (Optional)

No response

[milinddethe15]

no error produced on my local kind cluster on running pipeline manifest from master:

not sure why you and #3041 are getting this error? or why not me?

I followed this guide to apply manifest:
https://github.com/kubeflow/manifests/tree/master/apps/pipeline/upstream#envplatform-agnostic-install-on-any-kubernetes-cluster

[fraenkel]

@milinddethe15 The issue is the provisioner used with KIND.
The local-path provisioner does the following:

      mkdir -m 0777 -p "$VOL_DIR"

So the GID 0 which doesn't have write perrmision now does.

[juliusvonkohout]

@fraenkel @milinddethe15 @ericgtkb Please create a PR and tag me. We also need to open a second PR against kubeflow/pipelines with the same changes. Please also examine whether we still need runasuser or whether we can remove

manifests/apps/pipeline/upstream/third-party/minio/base/minio-deployment.yaml

Line 42 in 8d0f404

runAsUser: 1000

to use the default minio user (hopefully not root) and fix this

[juliusvonkohout]

@fraenkel are you also on our slack or linkedin ? Feel free to reach out https://www.linkedin.com/in/juliusvonkohout/

[juliusvonkohout]

#3045 is the upstream PR.