Proposal: Goki monorepo

[rcoreilly]

Proposal (from Kai and Randy)

Create a github.com/goki/goki repository that contains all of the dependencies for gi up through the current gix extras repository, with the following exceptions:

• freetype -- used by fyne and potentially others -- maintained fork of go official package that got abandoned.
• vulkan -- fork of package theoretically maintained by others and we'll probably transition to something else.

We also need to decide about gix -- could draw the line at giv, which would reduce size and dependencies, but probably the marginal benefits there are not worth the cost of maintaining a separate repo. What about etable?

Also, what about: ordmap, glop, grr, gengo, and possibly laser? See discussion below about major con of monorepo: makes it unlikely for others to be able to use these otherwise useful tools. Possible ways of "getting the best of both worlds" include maintaining separate stable versions of select repos, or just allowing others to effectively do the same thing by copying our code into their own projects.

Cons of separate repos

• Currently, almost every non-trivial change involves code changes across multiple repos.
• Any change requires a release cycle across all the repos.
• The release cycle is a nightmare due to the various interdependencies. Minor repos end up with many patch-level bumps.
• Versions are random and inconsistent -- nobody can really know what "the" version of the system is.
• Difficult for contributors to make PRs across multiple different repos (Easier development workflow #775).
• Repeated repo infrastructure: CI, LICENSE, CONTRIBUTING, github configuration (issue labels, Discussion structure..), etc.
• Issues often involve different repos and it is difficult to know where they are -- they get lost.
• No unified milestones.
• Currently using custom gsm tool to mitigate inherent slowness of dealing with so many repos (no single git pull, diff, etc) -- but it is still slow.
• Most of the current go.mod is stuffed with goki.dev/* entries.

Cons of monorepo

• most important: strongly discourages anyone from using any of the infrastructure bits (enums, gti, goosi, grr, glop etc) -- this is the most important motivation for having separate repos. OTOH, most of these things are most likely to be used in a GUI context (e.g., plain stringer is sufficient outside of enums). The main exceptions are: ordmap, glop, grr, gengo, and possibly laser.
• Large download to get everything. OTOH, any actual developer would need all of these things anyway. In which case, it is faster and easier to do in one swell foop.
• Likewise, go mod tidy in the new monorepo will get all of the dependencies for everything. OTOH, most of those dependencies are shared, and the theoretical savings would only apply to "optional" packages in gix, basically.

Pros of monorepo

• One go.mod, one version, one download -- just so much simpler. Currently we are entirely dependent on gsm to mimic what we would otherwise automatically get from a monorepo. Our editors are rooted at the parent goki.dev level, replicating the directory structure of the proposed monorepo, except we now have a bunch of extra overhead files in every dir that would otherwise be gone.
• Shorter mod urls: goki.dev/gi vs. goki.dev/gi/v2/gi

[rcoreilly]

Strategy:

• fork gi to giv1
• on gi: remove all tags, prs
• rename goki -> gokitool
• rename gi -> goki
• rename giv1 -> gi
• clean new goki history to only include current files: https://stackoverflow.com/questions/17901588/new-repo-with-copied-history-of-only-currently-tracked-files -- but this probably won't work because github keeps PRs as entirely separate entities, and only they can delete them by request -- and they only do so for sensitive data.. grr.. https://www.secjuice.com/github-complete-cleaning-sensitive-secrets/ https://stackoverflow.com/questions/18318097/delete-a-closed-pull-request-from-github -- probably need to just do a full hard reset of entire repo and then copy history for subtrees as we do for everything else. https://curia.me/github-how-to-reset-the-content-of-a-repository/
• merge all the existing repos into subtrees: https://stackoverflow.com/questions/1425892/how-do-you-merge-two-git-repositories -- keeps the commit history

[kkoreilly]

Another key point in favor of a monorepo: it means that you have easily reproducible and functional points in history. If you want to run the code at commit x, you just have to run git checkout x and you are done. With our current polyrepo system, it is basically impossible to do that, because there are tons of incompatible changes across many repositories, so you would have to manually pin down the correct version for tons of repositories. This also applies for end users: with a monorepo, if someone wants to use version x in their app, they can just run go get goki.dev@x. There is one unified version for the whole system that is guaranteed to work. With a polyrepo system, they would have to manage tons of dependency versions to do that. This is also related to the go.work problem: because we are actively developing across multiple repositories, we have to have everything in a work file created with gsm work. However, if someone else wanted to recreate that, they would have to clone every repository first and always keep them up to date. If they don't do that, their code doesn't build (and thus the CI also fails). Also, when developing apps outside of the goki organization, you need to manually make your own go.work and keep it up-to-date with all of the new repositories we keep making, which is an unnecessary extra burden.

[kkoreilly]

We are moving forward with this for the reasons listed above.

[kkoreilly]

The codebase will be unstable during this change.

[rcoreilly]

first pass completed! just need to update icons before merging into goki monorepo.

here's the script to prepare existing repo as a subdir:

#!/bin/bash -x

repo=$1
git clone https://github.com/goki/$repo
cd $repo
/bin/rm -rf .gitignore .goki LICENSE .github .travis.yml
git commit -am "cleanup"
mkdir $repo
mv * $repo
git filter-branch --index-filter \
     'git ls-files -s | sed "s#\t#&$repo/#" |
      GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
      git update-index --index-info &&
      mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD
git add $repo
git commit -am "add $repo files"
ls -la .

# in target repo:
# git pull ../$repo --allow-unrelated-histories
# git push --force

[rcoreilly]

clearly the right way to go! Now in https://github.com/cogentcore/core