Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rebase on KFP commit 78380095385be25e69b891ccb312b2857a200fdd #145

Merged
merged 32 commits into from
Feb 28, 2025

Conversation

mprahl
Copy link

@mprahl mprahl commented Feb 26, 2025

Description of your changes:

Rebase on the latest commit in the upstream master branch.

Checklist:

Sorry, something went wrong.

Ze Mao and others added 21 commits February 11, 2025 15:32
… job components/utils

Signed-off-by: Ze Mao <[email protected]>
PiperOrigin-RevId: 725795071

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
* rebase resolution

Signed-off-by: Daniel Dowler <[email protected]>

* removing go-licenses install instruction

Signed-off-by: Daniel Dowler <[email protected]>

---------

Signed-off-by: Daniel Dowler <[email protected]>

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Signed-off-by: Daniel Dowler <[email protected]>

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
…ubeflow#11539)

Check if there is a difference in generated files



fix lint



fix make file

Signed-off-by: chahatsagarmain <[email protected]>
Signed-off-by: Ze Mao <[email protected]>
PiperOrigin-RevId: 726255520

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Signed-off-by: Anish Asthana <[email protected]>

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Signed-off-by: Humair Khan <[email protected]>

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
…beflow#11631)

Signed-off-by: Daniel Dowler <[email protected]>

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Signed-off-by: arpechenin <[email protected]>

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
)

fix:



add log upload functionality



changes



test



update permission



update permission



fail test



remove exit



update permission



update



update



add condition



test



test



Remove exit



test



fix pod name



Remove exit



refactor



test



remove exit

Signed-off-by: chahatsagarmain <[email protected]>

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
… and .tar.gz files (kubeflow#11623)

* fix(frontend): restrict file explorer to show only .yaml, .yml, .zip, and .tar.gz files

Signed-off-by: muzzlol <[email protected]>

* test(frontend): update NewPipelineVersion snapshot for file filters

Signed-off-by: muzzlol <[email protected]>

---------

Signed-off-by: muzzlol <[email protected]>

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
…r format (sidecar) (kubeflow#11606)

This allows dsl.import to leverage Modelcar container images in an OCI
repository. This works by having an init container prepull the image and
then adding a sidecar container when the launcher container is running.
The Modelcar container adds a symlink to its /models directory in an
emptyDir volume that is accessible by the launcher container. Once the
launcher is done running the user code, it stops the Modelcar
containers.

This approach has the benefit of leveraging image pull secrets
configured on the Kubernetes cluster rather than require separate
credentials for importing the artifact. Additionally, no data is copied
to the emptyDir volume, so the storage cost is just pulling the Modelcar
container image on the Kubernetes worker node.

Note that once Kubernetes supports OCI images as volume mounts for
several releases, consider replacing the init container with that
approach.

This also adds a new environment variable of PIPELINE_RUN_AS_USER to
set the runAsUser on all Pods created by Argo Workflows.

Resolves:
kubeflow#11584

Signed-off-by: mprahl <[email protected]>

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
…sks (kubeflow#11651)

Signed-off-by: VaniHaripriya <[email protected]>

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Signed-off-by: Helber Belmiro <[email protected]>

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
…ubeflow#11278)

* Do not invoke get image methods twice.

Signed-off-by: carter.fendley <[email protected]>

* Add configurable driver / launcher log level

Signed-off-by: carter.fendley <[email protected]>

* Add configurable driver / launcher log level

Signed-off-by: carter.fendley <[email protected]>

* Update argocompiler golden files

Signed-off-by: carter.fendley <[email protected]>

* Handle errors from flag setting and tests

Signed-off-by: carter.fendley <[email protected]>

* Update gold files & image masking to use ghcr

Signed-off-by: carter.fendley <[email protected]>

* Update tests with new images

Signed-off-by: carter.fendley <[email protected]>

* Use unified var for driver / launcher log level + patch user code
launcher

Signed-off-by: carter.fendley <[email protected]>

* Add PIPELINE_LOG_LEVEL to deployment for discoverability

Signed-off-by: carter.fendley <[email protected]>

---------

Signed-off-by: carter.fendley <[email protected]>

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
… (kubeflow#11627)

Signed-off-by: zazulam <[email protected]>

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
…ubeflow#11657)

Signed-off-by: izuku-sds <[email protected]>

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Signed-off-by: Tarek Abouzeid <[email protected]>

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
…w#11662)

Signed-off-by: Humair Khan <[email protected]>

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
This skips uploading the model to S3 when the model URI starts with
oci://.

Signed-off-by: mprahl <[email protected]>

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
…1551)

Signed-off-by: mprahl <[email protected]>
HumairAK and others added 2 commits February 26, 2025 23:08

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Signed-off-by: Humair Khan <[email protected]>
@dsp-developers
Copy link

Change to PR detected. A new PR build was completed.
A set of new images have been built to help with testing out this PR:
API Server: quay.io/opendatahub/ds-pipelines-api-server:pr-145
DSP DRIVER: quay.io/opendatahub/ds-pipelines-driver:pr-145
DSP LAUNCHER: quay.io/opendatahub/ds-pipelines-launcher:pr-145
Persistence Agent: quay.io/opendatahub/ds-pipelines-persistenceagent:pr-145
Scheduled Workflow Manager: quay.io/opendatahub/ds-pipelines-scheduledworkflow:pr-145
MLMD Server: quay.io/opendatahub/mlmd-grpc-server:latest
MLMD Envoy Proxy: registry.redhat.io/openshift-service-mesh/proxyv2-rhel8:2.3.9-2
UI: quay.io/opendatahub/ds-pipelines-frontend:pr-145

@mprahl mprahl force-pushed the rebase-again branch 3 times, most recently from f21e3d6 to acf49cd Compare February 27, 2025 18:01

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
…1680)

Often times, pods could not be scheduled because of insufficient CPU
and the worker would run out of disk space.

Signed-off-by: mprahl <[email protected]>
@mprahl mprahl marked this pull request as ready for review February 27, 2025 21:54
@openshift-ci openshift-ci bot requested review from hbelmiro and HumairAK February 27, 2025 21:55
@mprahl mprahl marked this pull request as draft February 27, 2025 21:55
@dsp-developers
Copy link

Change to PR detected. A new PR build was completed.
A set of new images have been built to help with testing out this PR:
API Server: quay.io/opendatahub/ds-pipelines-api-server:pr-145
DSP DRIVER: quay.io/opendatahub/ds-pipelines-driver:pr-145
DSP LAUNCHER: quay.io/opendatahub/ds-pipelines-launcher:pr-145
Persistence Agent: quay.io/opendatahub/ds-pipelines-persistenceagent:pr-145
Scheduled Workflow Manager: quay.io/opendatahub/ds-pipelines-scheduledworkflow:pr-145
MLMD Server: quay.io/opendatahub/mlmd-grpc-server:latest
MLMD Envoy Proxy: registry.redhat.io/openshift-service-mesh/proxyv2-rhel8:2.3.9-2
UI: quay.io/opendatahub/ds-pipelines-frontend:pr-145

1 similar comment
@dsp-developers
Copy link

Change to PR detected. A new PR build was completed.
A set of new images have been built to help with testing out this PR:
API Server: quay.io/opendatahub/ds-pipelines-api-server:pr-145
DSP DRIVER: quay.io/opendatahub/ds-pipelines-driver:pr-145
DSP LAUNCHER: quay.io/opendatahub/ds-pipelines-launcher:pr-145
Persistence Agent: quay.io/opendatahub/ds-pipelines-persistenceagent:pr-145
Scheduled Workflow Manager: quay.io/opendatahub/ds-pipelines-scheduledworkflow:pr-145
MLMD Server: quay.io/opendatahub/mlmd-grpc-server:latest
MLMD Envoy Proxy: registry.redhat.io/openshift-service-mesh/proxyv2-rhel8:2.3.9-2
UI: quay.io/opendatahub/ds-pipelines-frontend:pr-145

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
…1683)

The KFP SDK execution tests weren't using the SDK from the pull request.
Additionally, this commit adds support for choosing the right SDK in
forks.

Signed-off-by: mprahl <[email protected]>

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
* Fix format string in get_kfp_package_path

Signed-off-by: mprahl <[email protected]>

* Use the in repo SDK for "k8s execution tests"

Signed-off-by: mprahl <[email protected]>

---------

Signed-off-by: mprahl <[email protected]>
HumairAK and others added 3 commits February 28, 2025 15:47

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
In 9afe23e we introduced blackend dropping of unknown fields for
unmarshalling, but going forward we want to handle this more on a case
by case basis. In the case for driver we should drop them because by
this point the api server has declare the pipeline spec is acceptable,
so the driver should not fail here. As such we keep driver changes, but
revert those utilized by the api server.

Signed-off-by: Humair Khan <[email protected]>

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
…electors, tolerations, pull secrets (kubeflow#11621)

This change adds the necessary proto changes to accomodate parameterized
input for various k8s platform fields: secrets, configmaps, node
selectors, tolerations, and image pull secrets.

Signed-off-by: Humair Khan <[email protected]>
@mprahl mprahl marked this pull request as ready for review February 28, 2025 17:15
@openshift-ci openshift-ci bot requested review from DharmitD and gmfrasca February 28, 2025 17:15
@mprahl mprahl changed the title Rebase on KFP commit eb7286d859c9f4700ca24adb47516d6207c99e25 Rebase on KFP commit 78380095385be25e69b891ccb312b2857a200fdd Feb 28, 2025
@dsp-developers
Copy link

Change to PR detected. A new PR build was completed.
A set of new images have been built to help with testing out this PR:
API Server: quay.io/opendatahub/ds-pipelines-api-server:pr-145
DSP DRIVER: quay.io/opendatahub/ds-pipelines-driver:pr-145
DSP LAUNCHER: quay.io/opendatahub/ds-pipelines-launcher:pr-145
Persistence Agent: quay.io/opendatahub/ds-pipelines-persistenceagent:pr-145
Scheduled Workflow Manager: quay.io/opendatahub/ds-pipelines-scheduledworkflow:pr-145
MLMD Server: quay.io/opendatahub/mlmd-grpc-server:latest
MLMD Envoy Proxy: registry.redhat.io/openshift-service-mesh/proxyv2-rhel8:2.3.9-2
UI: quay.io/opendatahub/ds-pipelines-frontend:pr-145

Copy link

@hbelmiro hbelmiro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

Verified in the cluster.

@mprahl
Copy link
Author

mprahl commented Feb 28, 2025

/approve

Copy link

openshift-ci bot commented Feb 28, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mprahl

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-bot openshift-merge-bot bot merged commit 4f466b3 into opendatahub-io:master Feb 28, 2025
63 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet