Feature: Support Using Zarf With an External Registry and Repository #754

YrrepNoj · 2022-09-20T19:00:13Z

Description

This PR introduces the ability to connect to an already existing (and reachable) Container Registry and/or Git Repository during the zarf init command.

Closes #570 (Support using an external git server)
Closes #560 (Support using an external registry)
This implementation will serve as a good midway point on having a fully HA in-cluster registry #375.

PR Feature List

Added several flags to the init command to support using an external git repository
Added several flags to the init command to support using an external container registry
Update zarf connect registry to direct to {HOST}/v2/_catalog (this was confusing some other people since it would originally seem like the registry was returning an empty page)
Add utility function to create a tunnel to a service URL
Created slightly better regexp for replacing the host from a containerImage url
semi-refactored the zarf package deploy logic

Breaking Changes List

We are changing the structure of the names of repos & containers we are pushing (we are simplifying the name and adding a sha1 hash of the original name to the end of the name)

If the 'external' git server still exists within the k8s cluster, create a tunnel when pushing repos to it

general cleanup and oranization of functions in hooks package for zarf agent

Makes sure the svc url comes at the end of the provided URL. This captures edge cases where someone has a subdomain that looks just like a service url would. eg) https://svc.cluster.local.example.com

update GitHub workflows to not explicitly create init-package if it's not needed (if it already exists)

The get-admin-password was a bad name because it wasn't clear what 'admin' meant. We are now making the name more explicit

update the flag git-username to git-account and add a note in the usage text to describe what to do if the repository belongs under an org

verify that the git-user and git-password has been provided if using an external git server

give users the ability to set some of the flags for the GitServerInfo without needing to host an extneral repository

ready check now verifies that the zarf-state has been loaded by verifying the distro field has been populated

jeff-mccoy · 2022-09-25T15:38:32Z

I think we should make testing with code commit and ECR a top priority, without that functionality this PR is far less valuable.

YrrepNoj · 2022-09-26T15:04:43Z

@RothAndrew @jeff-mccoy

Using ECR ✅ -> Not tested yet but I don't foresee any issues with testing this eventually)

Using CodeCommit 🚫
Zarf needs the ability to 'push to create repo'. This is something supported by Gitea and GitLab but I don't see anything in AWS docs that supports this (and the time I spent playing around didn't bear any good results)

I agree that CodeCommit is probably important enough for us to support as soon as possible, but that will require a bit of a lift in logic and won't be in this PR.

RothAndrew · 2022-09-26T15:29:05Z

It'll be kinda lame if we have to build in custom support for repo creation per provider, but look at the bright side, maybe we'll actually have a use case that is well suited to using Go's new generics support 😁

Racer159 · 2022-09-26T16:56:31Z

AWS code commit would require other changes than just push to create. It doesn't have a user/org hierarchy by default either:

aws             https://git-codecommit.us-east-1.amazonaws.com/v1/repos/hello-kitteh (fetch)
aws             https://git-codecommit.us-east-1.amazonaws.com/v1/repos/hello-kitteh (push)
bitbucket       https://bitbucket.org/racer159/hello-kitteh (fetch)
bitbucket       https://bitbucket.org/racer159/hello-kitteh (push)
gitlab          https://gitlab.com/Racer159/hello-kitteh (fetch)
gitlab          https://gitlab.com/Racer159/hello-kitteh (push)

Racer159 · 2022-09-26T16:59:27Z

I would be for approving this for now and then having another issue to look at genericizing the provider for git to work with other things like BitBucket and AWS Code Commit.

YrrepNoj · 2022-09-26T17:53:03Z

@Racer159 I'll work on adding a note to describe the gotchas for using an external repo and registry.

I believe this is ready for another round of reviews in the meantime.

cc @JasonvanBrackel

Racer159

This worked for my test cases, and generally looks good. I think the only potential gotcha is if GitRepositoryUrls ever change we wouldn't catch the update

src/internal/agent/hooks/flux.go

…rl hostname does not match the hostname in the zarfstate

Racer159

lgtm

…754) ## Description This PR introduces the ability to connect to an already existing (and reachable) Container Registry and/or Git Repository during the `zarf init` command. Closes #570 (Support using an external git server) Closes #560 (Support using an external registry) This implementation will serve as a good midway point on having a fully HA in-cluster registry #375. ## PR Feature List - Added several flags to the `init` command to support using an external git repository - Added several flags to the `init` command to support using an external container registry - Update `zarf connect registry` to direct to `{HOST}/v2/_catalog` (this was confusing some other people since it would originally seem like the registry was returning an empty page) - Add utility function to create a tunnel to a service URL - Created slightly better regexp for replacing the host from a `containerImage` url - semi-refactored the `zarf package deploy` logic ## Breaking Changes List - We are changing the structure of the names of repos & containers we are pushing (we are simplifying the name and adding a sha1 hash of the original name to the end of the name) Co-authored-by: Wayne Starr <[email protected]> Co-authored-by: Megamind <[email protected]>

YrrepNoj force-pushed the 560-external-registry-and-repository branch 2 times, most recently from e97eaca to c1d8b01 Compare September 22, 2022 03:24

YrrepNoj mentioned this pull request Sep 22, 2022

Handle External Git Repository and Container Registry on zarf init #666

Closed

YrrepNoj added 27 commits September 22, 2022 22:32

create InitOptions for init specific flags

2d27030

fix Makefile dev-agent-image target

9463e01

save zarfState during init for future flux git url mutations

ac9b174

set GitServerInfo as we seed the cluster

4ecdc56

read git information from GitServerInfo within config

19ae80a

check if we need to tunnel to external git server

205ff7e

If the 'external' git server still exists within the k8s cluster, create a tunnel when pushing repos to it

add simple manual test for external-git on init

dbccfe3

remove unnecessarly git-read-user flags on init

7b56681

general cleanup and oranization of functions in hooks package for zarf agent

add more explicit check to determin if url is an internal svc url

4a93afa

Makes sure the svc url comes at the end of the provided URL. This captures edge cases where someone has a subdomain that looks just like a service url would. eg) https://svc.cluster.local.example.com

fix typo in Makefile

131b7a3

have e2e test only make init-package if not already existing

ed2af3d

update GitHub workflows to not explicitly create init-package if it's not needed (if it already exists)

update zarfState json text for GitServer

0d54d88

add back explicit creation of init package in github workflow pipeline

7b2754b

verify init-package exists during the 'test-e2e' make target

87027bb

rename get-admin-password command to get-git-password

1cf2d1f

The get-admin-password was a bad name because it wasn't clear what 'admin' meant. We are now making the name more explicit

update flag and usage text for git-account for the prepare command

45ddcb6

update the flag git-username to git-account and add a note in the usage text to describe what to do if the repository belongs under an org

add validation checking to init command flags

96ac579

verify that the git-user and git-password has been provided if using an external git server

fill in GitServer values without overriding

ba201e6

give users the ability to set some of the flags for the GitServerInfo without needing to host an extneral repository

wip - registryInfo struct

87608f0

wip

e1cb572

add json annotation to ContainerRegistryInfo

805160a

move nodeport from InitOptions to ContainerRegistryInfo

b456b66

simplify containerRegistryInfo var names

8266771

check for external registry before seeding

f0fbb7d

clean up workflow around seed registry a little

148b686

change ready check for valuate template

d0f4705

ready check now verifies that the zarf-state has been loaded by verifying the distro field has been populated

set default value for nodeport flag during init

660cf70

Merge branch 'master' into 560-external-registry-and-repository

dcc661f

cleanup typos, regexp explanations, git fetch logic, and go.mod

09fc8c9

Merge branch 'master' into 560-external-registry-and-repository

8c3b006

YrrepNoj added 2 commits September 26, 2022 13:23

Zarf agent webhook no longer tries to update flux repo URL on updates

bc6e0b1

update 'zarf tool' command docs

849189d

YrrepNoj and others added 2 commits September 26, 2022 15:21

update go.mod and go.sum

5dedef9

Merge branch 'master' into 560-external-registry-and-repository

3738e13

Racer159 mentioned this pull request Sep 28, 2022

Support git repo caching on package create #785

Merged

5 tasks

Merge branch 'master' into 560-external-registry-and-repository

a174b02

Racer159 self-requested a review September 28, 2022 19:43

Racer159 requested changes Sep 28, 2022

View reviewed changes

src/internal/agent/hooks/flux.go Show resolved Hide resolved

YrrepNoj and others added 3 commits September 28, 2022 22:54

perform hostname mutation on flux webhook update trigger if the git u…

6b8df96

…rl hostname does not match the hostname in the zarfstate

Merge branch 'master' into 560-external-registry-and-repository

367040d

cleanup flux webhook to check operation type -> more DRY code

9530176

Racer159 self-requested a review September 29, 2022 19:53

Racer159 previously approved these changes Sep 29, 2022

View reviewed changes

Tunnel Update

09f13ef

jeff-mccoy dismissed Racer159’s stale review via 09f13ef September 29, 2022 21:13

jeff-mccoy enabled auto-merge (squash) September 29, 2022 21:16

Racer159 self-requested a review September 29, 2022 21:19

Racer159 approved these changes Sep 29, 2022

View reviewed changes

jeff-mccoy merged commit eb841ee into master Sep 29, 2022

bdfinst deleted the 560-external-registry-and-repository branch January 12, 2023 22:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature: Support Using Zarf With an External Registry and Repository #754

Feature: Support Using Zarf With an External Registry and Repository #754

YrrepNoj commented Sep 20, 2022 •

edited

Loading

jeff-mccoy commented Sep 25, 2022

YrrepNoj commented Sep 26, 2022 •

edited

Loading

RothAndrew commented Sep 26, 2022

Racer159 commented Sep 26, 2022

Racer159 commented Sep 26, 2022

YrrepNoj commented Sep 26, 2022 •

edited

Loading

Racer159 left a comment

Racer159 left a comment

Feature: Support Using Zarf With an External Registry and Repository #754

Feature: Support Using Zarf With an External Registry and Repository #754

Conversation

YrrepNoj commented Sep 20, 2022 • edited Loading

Description

PR Feature List

Breaking Changes List

jeff-mccoy commented Sep 25, 2022

YrrepNoj commented Sep 26, 2022 • edited Loading

RothAndrew commented Sep 26, 2022

Racer159 commented Sep 26, 2022

Racer159 commented Sep 26, 2022

YrrepNoj commented Sep 26, 2022 • edited Loading

Racer159 left a comment

Choose a reason for hiding this comment

Racer159 left a comment

Choose a reason for hiding this comment

YrrepNoj commented Sep 20, 2022 •

edited

Loading

YrrepNoj commented Sep 26, 2022 •

edited

Loading

YrrepNoj commented Sep 26, 2022 •

edited

Loading