Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor Docker setup, follow best-practices in containerization and make caldera easier to deploy #3114

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

daw1012345
Copy link
Contributor

@daw1012345 daw1012345 commented Dec 26, 2024

Description

This project is currently quite difficult to get working. This is one of (at least) three pull requests intended to make caldera simpler to deploy or get running locally. Docker is a great tool for this, and should be the easiest and quickest deployment option.

Unfortunately, the current Dockerfile doesn't even build (due to an outdated version of Ubuntu).
This PR:

  • Updates the Dockerfile to use a staged build to compile the UI and copies it over to the final container. This avoids the dance of installing node and npm, building the UI, then removing node and npm.
  • Ensures that previous builds of caldera do not pollute the build within the container. For example, if caldera was ever compiled outside of Docker (especially the UI), it will not be recompiled in the container. This can cause a large headache, for example when the Docker build does not re-build the UI when the VITE_BASE_URL was changed.
  • Containers should contain all basic requirements for all of the functionality to work. For example - if a container is compiled without emu support and then published to DockerHub, a user will not be able to easily get all the dependencies into the container (without modifying the Dockerfile and building it themselves). This PR ensures this is the case.
  • The emu and atomic plugins depend on large submodules to work, but are capable of fetching them when they are enabled. This PR introduces two flavours of images - slim and full. The full (default) flavor contains said submodules to allow the container to work in environments without an internet connection, while the slim flavour disables the plugins and only fetches the submodules when they are enabled.
  • Modify the workflow to build and push both slim and full images to GHCR (tagged appropriately).
  • Adds a dependency on setuptools to work with recent versions of Python3
  • Makes Docker the recommended deployment option in the README
  • Every deployment of caldera using any of the currently provided Docker containers has the same password as it is generated during build-time.

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update

How Has This Been Tested?

Repeated build tests of both flavors of caldera (full + slim). The resulting containers appear to work when interacted with.

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation
  • [NOT RELEVANT] I have added tests that prove my fix is effective or that my feature works

@rfulwell
Copy link
Contributor

rfulwell commented Dec 27, 2024

These changes are super interesting and I look forward to going through them in detail! 🚀

Unfortunately, the current Dockerfile doesn't even build (due to an outdated version of Ubuntu).

In the meantime, here's the minimal fix for the Docker build which was approved and just waiting on a merge:
#3099 @daw1012345

@rfulwell
Copy link
Contributor

Docker is a great tool for [deployment], and should be the easiest and quickest deployment option.

💯

Thank you for tackling these changes! I fully agree that this project will be much more useful when Docker deployment is trivial. 👍

@daw1012345
Copy link
Contributor Author

Thanks @rfulwell. I made a few additional change to the Docker setup, I was wondering if you could test it yourself as well to make sure everything works? I am by no means a caldera power user and would like to make sure these changes don't break anything.

@daw1012345
Copy link
Contributor Author

And thank you for the original PR fixing the outdated Ubuntu version in the Dockerfile, hope that gets merged quickly as well.

Follow best-practices and make caldera easier to use
@elegantmoose
Copy link
Contributor

On our queue.

echo $TZ > /etc/timezone
# This file uses a staged build, using a different stage to build the UI (magma)
# Build the UI
FROM node:23 as ui-build

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

=> WARN: FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 3)


# This is the runtime stage
# It containes all dependencies required by caldera
FROM debian:bookworm-slim as runtime

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

=> WARN: FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 13)

@endiz
Copy link

endiz commented Mar 12, 2025

While I don't use docker for this project, I think quite a lot of people do to quickly get started. This PR would be a great service to the community.

@elegantmoose elegantmoose requested review from uruwhy and removed request for clenk March 12, 2025 15:25
@elegantmoose
Copy link
Contributor

@endiz Yep, @uruwhy is gonna look at this one once he has a minute.


RUN ./update-agents.sh
# Install pip requirements
RUN pip3 install --break-system-packages --no-cache-dir -r requirements.txt
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thoughts on installing packages within a virtual env?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was a conscious choice. We already are in a Docker container, we don't have to worry about polluting anything.

WORKDIR /usr/src/app
# Make sure emu can always be used in container (even if not enabled right now)
RUN cd /usr/src/app/plugins/emu; \
pip3 install --break-system-packages -r requirements.txt
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thoughts on installing packages within a virtual env?

daw1012345 and others added 2 commits March 12, 2025 21:01
Co-authored-by: Daniel Matthews <[email protected]>
Co-authored-by: Daniel Matthews <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants