[Discovery] Docker Desktop alternatives

[jmbowman]

A/C:

• Create a document comparing the options
• Kick off a discussion within 2U and with the broader Open edX community on recommendations for Tutor & devstack
• Ticket selection of a preferred option for 2U (may vary by OS/circumstances) and document

Docker changed the licensing of Docker Desktop last year to no longer be free as in beer (and was never free as in freedom). In addition to the direct monetary cost to 2U, this also makes it less than ideal as the recommended dev environment for an AGPL software platform. And finally, we're moving more towards adopting Kubernetes as our default orchestration system, and Kubernetes has deprecated dockerd support in favor of containerd. Given all this, it's time to start looking at alternatives.

Do some discovery into enumerating options for consideration, and do an all-way comparison against our current Docker Desktop. Alternatives which have been already been proposed include:

• Colima
• Minikube
• Podman Desktop
• Rancher Desktop
• OrbStack

A partial list of points worth comparing:

• License
• Monetary cost
• Required hardware resources (RAM, CPU, etc.)
• Ease of use
• Kubernetes support
• containerd & nerdctl support (current Kubernetes recommendations)
• Supported host operating systems
• Level of support for Apple ARM systems
• Support for rootless operation
• Support for mounting container directories into the host filesystem (or vice versa)

Potentially useful reading:

• https://blog.kloia.com/a-quick-introduction-to-the-docker-desktop-alternative-rancher-desktop-f327fe325000
• https://codeengineered.com/blog/2022/docker-desktop-vs-rancher-desktop/
• https://www.thoughtworks.com/radar/platforms?blipid=202203015

[kdmccormick]

Kubernetes deprecated dockershim, not docker: https://kubernetes.io/blog/2020/12/02/dont-panic-kubernetes-and-docker/

That said, I agree that looking into AGPL alternatives to docker is not a bad idea.

[jmbowman]

Right, I noted that "dockerd" support was deprecated, not "docker". The article you linked to mentions:

But it’s not the end of the world, and you don’t need to panic—you just need to change your container runtime from Docker to another supported container runtime.

Just the fact that it takes multiple blog posts over 2 years to explain "no, it's not Docker use that's deprecated, it's just the use of the primary dockerd daemon (actually it's secondary now, Docker Desktop also comes with containerd these days), you can still run Kubernetes in Docker Desktop if you want to, ..." makes me want to look at alternatives.

[jmbowman]

Some benchmarks of macOS file system performance for several of the options, and mention of a tool that improves it further for each of them: docker/roadmap#7 (comment)

[kdmccormick]

@dianakhuang brought up OrbStack at the last DevX meetup. It's a closed-source solution that significantly outperforms Docker Desktop on macOS. It is free for now, but will eventually require a paid license. Unclear currently how the cost will compare to Docker Devstack.

@ashultz0 has been using it successfully with Devstack; so far it seems drop-in.

@regisb tentatively suggested that we could begin considering it a bug if Tutor doesn't work with OrbStack (as well as Docker). We aren't doing this yet, but it is something worth considering if OrbStack shows a lot of value.

@zacharis278 tried it with Tutor but it hung while booting up LMS. We suspect this may be related to the Quickdev plugin, which is on its way out, hopefully by Quince. However, Diana mentioned:

i haven't looked into it, but there is a question with the Palm Tutor changes whether or not OrbStack supports the new compose syntax yet, which we haven't tested

[kdmccormick]

Max from RaccoonGang:

I use the Orb on daily bases for Olive and Palm (testing it in BTR WG). So Palm is compatible with Orb for sure.
Everything works pretty similar to Docker desktop except a bit faster 🙂
Really enjoying having tutor palm as long as additional NixOS as a separate VM in Orb without significant impact on my Host machine (Mac M1).The issue I’m facing at now is running full tutor dev start - 7 MFE containers running simultaneously npm run start. It’s hard. But the same issue I have had w/ Docker Desktop as well.BTW, currently, I’m running Tutor Palm in Orb (not Dev but Local deploy), K8S in the Docker desktop (ElasticSearch cluster for Harmony testing) and NixOS VM. Challenging for M1, but doable.P.S. the only thing I’m scared of:

OrbStack is completely free to use during beta, but it will become a paid product afterwards. We're still working out the details (personal vs. business use, subscription vs. perpetual license, pricing, OSS and student discounts, etc.) and will share more information as we get closer to launch.
If you have ideas for pricing or licensing, please let us know.

[kdrag0n]

👋 OrbStack developer here. Great to see that you're considering OrbStack for edX development!

i haven't looked into it, but there is a question with the Palm Tutor changes whether or not OrbStack supports the new compose syntax yet, which we haven't tested

We include the latest version of Docker Compose, so it should be supported.

The issue I’m facing at now is running full tutor dev start - 7 MFE containers running simultaneously npm run start. It’s hard. But the same issue I have had w/ Docker Desktop as well.

If it happens with Docker Desktop as well, then it may not be an OrbStack bug, but we'd be happy to investigate given more details and steps to reproduce.

the only thing I’m scared of:

We've shared initial pricing plans here: https://twitter.com/OrbStack/status/1656326409995055104

Let me know if I can help with anything else!

[davidjoy]

In the description of this task we have some justification for why it's necessary:

• Docker Desktop costs money now
• Docker Desktop is closed-source ("never free as in freedom")
• K8s deprecated dockerd

And further down in the thread, we imply a fourth:

• Docker Desktop is slower than alternatives

If these are reasons to drop Docker Desktop, it seems like they're also constraints on anything else we'd choose... otherwise why drop Docker? So the question is, which of the alternatives listed above best satisfy these criteria?

• Free
• Open-source
• k8s friendly
• Performance is similar or faster than Docker Desktop

When phrased this way, do those ring true as our criteria?

Taking it one step further, is there a single environment that we can support and recommend? As we're all acutely aware of already, the further we fragment our dev environments across the community, the harder it is to help each other diagnose and fix dev environment issues. Adding another variable and supporting multiple environments won't make that better - not that we're necessarily doing that here, but it's something to be mindful of. Can we meaningfully coalesce around one choice and avoid further fragmentation?

[kdmccormick]

When providing a community choices, I think sometimes they cause confusion and duplicated work. Think Tutor vs Devstack: new devs don't know which one to use; we have two different teams driving and maintaining each stack; each has a different community of users asking for questions and helping one another; etc. That's what I'd call "fragmentation".

Other times, choices make everyone happier with minimal work from maintainers. Think editors: I don't care if someone is working in Vim or VSCode, I can still help them debug their stack all the same. Telling people which editor they should use would actually cause more strife--being agnostic is the best choice. Let's call this "having options".

Of course, it's a spectrum. Something like Linux vs macOS feels like it could sit in the middle.

Having tried both Docker (on Ubuntu and macOS) and OrbStack (on macOS), I haven't noticed any differences between the two except that OrbStack feels significantly faster and more reliable on macOS. I've been hearing the same from other Open edX devs. Given that, I think this sort of choice is much more in the "having options" camp than the "fragmentation" camp. As a Tutor maintainer I think I'd be comfortable saying: "Tutor is tested & most actively supported on Docker for Linux... but it should work on any Docker-like runtime in any Unix-like OS, and if it doesn't, that's probably a bug." @regisb, let me know if you disagree.

That said, I understand that this is a 2U ticket and you folks may need to choose one or the other for licensing purposes.

[kdrag0n]

We'd be happy to consider offering free OrbStack licenses for open-source development if that applies in this case (i.e. not too much of a commercial focus).

OrbStack will also be getting built-in Kubernetes support, most likely with Docker as the runtime for easier development.

[kdmccormick]

We'd be happy to consider offering free OrbStack licenses for open-source development

Very exciting to hear!

if that applies in this case (i.e. not too much of a commercial focus)

I think this will depend on the specifics of your free license terms :) The project (Open edX) is completely free and open-source, and it is steered by a non-profit (Axim). Still, many/most of our contributions come from for-profit companies like 2U, so part of choosing development tools for Open edX means finding ones that both our non-profit and for-profit community members can use.

[kdrag0n]

Thanks for explaining. It sounds like developers at 2U would be required to pay for a license.

Strictly speaking, contributors contributing on behalf of the non-profit company would also need a paid license as having a non-profit label doesn't carry that much weight, but that can be handled with free open-source licenses it's an issue.

[dianakhuang]

initial wiki page for this comparison: https://openedx.atlassian.net/wiki/spaces/AC/pages/3845914644/Container+Runtime+Comparison

[jmbowman]

I filled in some data for Docker Desktop, Rancher Desktop, Minikube, and OrbStack yesterday. Remaining tasks:

• Try Colima and document the results
• Try Podman Desktop and document the results
• Create a minimal test case for the Rancher Desktop issue with npm ci in the LMS, submit upstream to see if there's a fix or workaround
• Add any other notes to the doc that seem relevant
• Make a tentative decision

[kdrag0n]

Thanks for the comparison @jmbowman!

[Colima] is practically the only option from this list that can be cleanly installed on macOS CI runners

OrbStack actually supports headless/CI usage as well, with no GUI app involved. Just wrote a docs page for this use case to make it more clear: https://docs.orbstack.dev/headless

[jmbowman]

The context on the CI issue is actions/runner-images#2150 . There were several twists and turns, but I think it ended up at "GitHub can't pre-install Docker Desktop on macOS Actions runners for legal reasons, end users can install it but have to pay some unclear licensing fee (max number of concurrent runners x $24/month?) and eat up runner time with the installation, Colima is now pre-installed by default". It's nice that OrbStack can be used in CI, but it still has the "need to spend time installing it" problem and possibly some lack of clarity around license fees once the beta period ends. You might want to address that specific use case once you decide on the final post-beta licensing terms; 2x-3x Colima performance in CI would be pretty compelling if it was clearly allowed and didn't require accounting/monitoring work to figure out what we owe and if we're willing to pay it.

We don't actually do much macOS testing in CI right now, but that's mainly because of the Docker installation/licensing difficulty; the majority of our dev machines use macOS (with a nontrivial minority on Ubuntu), and we do occasionally hit problems that only occur on macOS due to Linux/Mac command syntax differences, etc. The Colima development on runners is relatively recent, so we haven't tried it out yet (and weren't sure it would really be representative of our local Docker Desktop installations).

We're still assembling final notes, but it looks like things are trending towards recommending:

• OrbStack for better performance and developer experience for organizations willing to pay for it in the long term
• Colima for organizations and use cases where "free and open source" is a more important consideration
• Docker Desktop still supported at best effort for those reluctant to change from the status quo yet

In general, I think we want to go with "try not to needlessly break compatibility with any of the Docker engine options, but have 1-2 clear default recommendations". And whether 2U can go with OrbStack or not will depend on the outcome of a vendor review process, which I'm trying to get approval to start in the next day or so.

[rgraber]

Blocking on vendor review of OrbStack

[robrap]

@jristau1984: Do you want to follow up on the Vendor Review ticket for OrbStack that Jeremy B. had created? Note that this would not only make developer's lives easier, but would also save the company money.

[jristau1984]

@robrap this decision has been recently made by IT, and the chosen tool is Podman. The directive was given to directors earlier this week, but not surprisingly has not made it to all team members yet.

[robrap]

@jristau1984: We're hoping this decision is changed and Orbstack can still be an option. That said, I am closing this ticket, because this ticket was basically completed as written.