Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce images size #1636

Merged
merged 37 commits into from
Jun 26, 2020
Merged
Changes from 1 commit
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
5c4bab5
Use nodemon instead of PM2
May 24, 2020
a3daaeb
Reduce production image size
May 24, 2020
dfff4d9
Nit
May 24, 2020
c184da6
Update scripts
May 25, 2020
104092b
Fix plugin-dev
May 25, 2020
6ca77b1
Save 4mo
May 25, 2020
93bc34e
Move images
May 26, 2020
c76aafa
asynchronous extract
May 27, 2020
cd5a6b1
Fix images
May 27, 2020
ee0bb17
Nit
May 27, 2020
e423165
nit
May 27, 2020
437bdca
Reduce size
May 28, 2020
c429fb4
wip
May 28, 2020
b04ce12
Plugin-dev ok
May 28, 2020
9e0d229
merge ok
May 28, 2020
f87ed79
Add wait-elasticsearch
May 28, 2020
9b98ace
kuzzleio/kuzzle ok
Jun 19, 2020
6862c52
kuzzleio/alpine ok
Jun 19, 2020
ff1ae5e
all images ok
Jun 19, 2020
bb3818f
Add README
Jun 19, 2020
19f3314
Fix tests
Jun 19, 2020
383f8b5
trigger ci
Jun 19, 2020
271413d
fix test
Jun 19, 2020
0df3a7b
fix sh
Jun 19, 2020
119bb7d
use node for startup scripts
Jun 19, 2020
70e5fa8
Bump node 12.18.1
Jun 22, 2020
41d1cf5
Merge branch 'master' into reduce-production-image-size
Aschen Jun 22, 2020
17660d7
Fix dependencies install in tests
Jun 23, 2020
e331000
Update docker/scripts/run-dev.sh
Aschen Jun 23, 2020
f14f4c4
Update scripts
Jun 23, 2020
b33d5b9
Merge branch 'master' into reduce-production-image-size
Aschen Jun 23, 2020
bd5a9c2
Update scripts
Jun 23, 2020
8b219d4
Merge branch 'reduce-production-image-size' of github.com:kuzzleio/ku…
Jun 23, 2020
3764998
replace everest by scratch
Jun 24, 2020
5904564
Merge branch 'reduce-production-image-size' of github.com:kuzzleio/ku…
Jun 24, 2020
2525a86
fix scratch image
Jun 24, 2020
1ea882f
factorize steps
Jun 25, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Nit
Aschen committed May 27, 2020
commit ee0bb176dc22f051a8ebdd21af09ee71565d7378
6 changes: 3 additions & 3 deletions .ci/privatebuild/Dockerfile
Original file line number Diff line number Diff line change
@@ -68,11 +68,11 @@ ENV NODE_ENV=production

COPY --from=dependencies /var/app/ /var/app/

RUN ln -s /var/app/docker-compose/scripts/run.sh /usr/local/bin/kuzzle \
RUN ln -s /var/app/docker/scripts/run.sh /usr/local/bin/kuzzle \
&& chmod a+x /usr/local/bin/kuzzle \
&& chmod a+x /var/app/docker-compose/scripts/docker-entrypoint.sh
&& chmod a+x /var/app/docker/scripts/docker-entrypoint.sh

ENTRYPOINT ["/var/app/docker-compose/scripts/docker-entrypoint.sh"]
ENTRYPOINT ["/var/app/docker/scripts/docker-entrypoint.sh"]

CMD ["kuzzle", "start"]

41 changes: 2 additions & 39 deletions .ci/scripts/run-test.sh
Original file line number Diff line number Diff line change
@@ -10,48 +10,11 @@ echo "Testing Kuzzle against node v$NODE_VERSION"
n $NODE_VERSION

npm install --silent --unsafe-perm
npm install --silent --unsafe-perm --only=dev
chmod -R 777 node_modules/
docker-compose/scripts/install-plugins.sh

echo "[$(date --rfc-3339 seconds)] - Waiting for elasticsearch to be available"
spinner="/"
while ! curl -f -s -o /dev/null "$elastic_host"
do
printf '\r'
echo -n "[$(date --rfc-3339 seconds)] - Still trying to connect to $elastic_host [$spinner]"
if [ "$spinner" = "/" ]; then spinner="\\"; else spinner="/" ; fi
sleep 1
done
# create a tmp index just to force the shards to init
echo "[$(date --rfc-3339 seconds)] - Elasticsearch is up. Waiting for shards to be active (can take a while)"
E=$(curl -s "$elastic_host/_cluster/health?wait_for_status=yellow&timeout=60s")

if ! (echo ${E} | grep -E '"status":"(yellow|green)"' > /dev/null); then
echo "============ Cluster health response"
echo $E
echo "============ Cluster allocation explanation"
curl -s http://$elastic_host/_cluster/allocation/explain?pretty
echo "[$(date --rfc-3339 seconds)] - Could not connect to elasticsearch in time. Aborting..."
exit 1
fi
./docker/scripts/install-plugins.sh

node bin/start-kuzzle-server --enable-plugins functional-test-plugin &

echo "[$(date --rfc-3339 seconds)] - Starting Kuzzle..."
timeout=$((20 * 60))
while ! curl -f -s -o /dev/null http://localhost:7512
do
printf '\r'
echo -n "[$(date --rfc-3339 seconds)] - Still trying to connect to Kuzzle [$spinner]"
if [ "$spinner" = "/" ]; then spinner="\\"; else spinner="/" ; fi

timeout=$((timeout - 1))
if [ $timeout -eq 0 ]; then
echo "[$(date --rfc-3339 seconds)] - Timeout"
fi

sleep 1
done
echo "[$(date)] - Starting Kuzzle..."

npm run test:functional
6 changes: 3 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -24,7 +24,7 @@ We use most of the [NPM Coding Style](https://docs.npmjs.com/misc/coding-style)

## Tools

For development only, we built a specific docker-compose file: `docker-compose/dev.yml`. You can use it to profile, debug, test a variable on the fly, add breakpoints and so on, thanks to [chrome-devtools](https://developer.chrome.com/devtools).
For development only, we built a specific docker-compose file: `docker-compose.yml`. You can use it to profile, debug, test a variable on the fly, add breakpoints and so on, thanks to [chrome-devtools](https://developer.chrome.com/devtools).
Check the logs at the start of Kuzzle using the development docker image to get the appropriate debug URL.

How to run the development stack (needs Docker 1.10+ and Docker Compose 1.8+):
@@ -52,14 +52,14 @@ Everytime a modification is detected in the source files, the server is automati

The development stack include a endpoint to access Kuzzle API through SSL on port `7443`.

The certificates are privately signed, using provided [CA certificate](docker-compose/nginx/kuzzleCA.crt).
The certificates are privately signed, using provided [CA certificate](docker/nginx/kuzzleCA.crt).
Domains accepted:
- localhost
- *.kuzzle.loc

You'll need to import the CA certificate to your browser and possibly your system local authorities to make it verified.
Once done, your browser should not complain when reaching https://localhost:7443.
The CA certificate is here: [docker-compose/nginx/kuzzleCA.crt](docker-compose/nginx/kuzzleCA.crt)
The CA certificate is here: [docker/nginx/kuzzleCA.crt](docker/nginx/kuzzleCA.crt)

Using node.js, for instance when using the sdk, you'll need to pass the CA cert using the `NODE_EXTRA_CA_CERTS` environment variable:

14 changes: 8 additions & 6 deletions docker/images/kuzzle.alpine/Dockerfile
Original file line number Diff line number Diff line change
@@ -5,7 +5,6 @@ FROM node:12.16.3-alpine3.11 as builder

ADD ./bin /var/app/bin
ADD ./config /var/app/config
ADD ./docker /var/app/docker

ADD ./lib /var/app/lib

@@ -23,7 +22,8 @@ WORKDIR /var/app

ENV NODE_ENV=production

RUN apk add --no-cache \
RUN set -x \
&& apk add --no-cache \
curl \
python \
make \
@@ -66,7 +66,8 @@ FROM node:12.16.3-alpine3.11 as minifier

COPY --from=builder /var/app /var/app

RUN apk add --no-cache \
RUN set -x \
&& apk add --no-cache \
xz \
binutils \
&& strip /usr/local/bin/node \
@@ -86,11 +87,12 @@ LABEL description="Run your Kuzzle backend in production mode"
ENV NODE_ENV=production

ADD ./docker/scripts/promises.sh /promises.sh
ADD ./docker/scripts/entrypoint.sh /bin/entrypoint
ADD ./docker/scripts/extract.sh /bin/extract
ADD ./docker/scripts/run.sh /bin/kuzzle

COPY --from=minifier /lib.tar.xz /everest/lib.tar.xz
COPY --from=minifier /usr.tar.xz /everest/usr.tar.xz
COPY --from=builder /var/app/docker/scripts/entrypoint.sh /bin/entrypoint
COPY --from=builder /var/app/docker/scripts/extract.sh /bin/extract
COPY --from=builder /var/app/docker/scripts/run.sh /bin/kuzzle
COPY --from=minifier /app.tar.xz /everest/app.tar.xz

RUN chmod a+x /bin/extract \
11 changes: 6 additions & 5 deletions docker/images/kuzzle/Dockerfile
Original file line number Diff line number Diff line change
@@ -5,7 +5,6 @@ FROM node:12.16.3-stretch-slim as builder

ADD ./bin /var/app/bin
ADD ./config /var/app/config
ADD ./docker /var/app/docker

ADD ./lib /var/app/lib

@@ -68,7 +67,8 @@ FROM node:12.16.3-stretch-slim as minifier
COPY --from=builder /usr/bin/strip /bin/strip
COPY --from=builder /var/app /var/app

RUN apt-get update && apt-get install -y binutils \
RUN set -x \
&& apt-get update && apt-get install -y binutils \
&& rm -rf /var/lib/apt/lists/* \
&& /bin/strip /usr/local/bin/node \
&& apt-get remove -y binutils \
@@ -87,11 +87,12 @@ LABEL description="Run your Kuzzle backend in production mode"
ENV NODE_ENV=production

ADD ./docker/scripts/promises.sh /promises.sh
ADD ./docker/scripts/entrypoint.sh /bin/entrypoint
ADD ./docker/scripts/extract.sh /bin/extract
ADD ./docker/scripts/run.sh /bin/kuzzle

COPY --from=minifier /lib.tar.xz /everest/lib.tar.xz
COPY --from=minifier /usr.tar.xz /everest/usr.tar.xz
COPY --from=builder /var/app/docker/scripts/entrypoint.sh /bin/entrypoint
COPY --from=builder /var/app/docker/scripts/extract.sh /bin/extract
COPY --from=builder /var/app/docker/scripts/run.sh /bin/kuzzle
COPY --from=minifier /app.tar.xz /everest/app.tar.xz

RUN chmod a+x /bin/extract \
13 changes: 7 additions & 6 deletions docker/images/plugin-dev/Dockerfile
Original file line number Diff line number Diff line change
@@ -73,18 +73,19 @@ FROM debian:stretch-slim as plugin-dev
LABEL io.kuzzle.vendor="Kuzzle <[email protected]>"
LABEL description="Develop new plugin or protocol for Kuzzle with ease"

COPY --from=builder /usr/bin/xz /bin/xz
ADD ./docker/scripts/promises.sh /promises.sh
ADD ./docker/scripts/entrypoint.sh /bin/entrypoint
ADD ./docker/scripts/extract.sh /bin/extract
ADD ./docker/scripts/run-plugin.sh /bin/run-plugin

COPY --from=builder /usr/bin/xz /bin/xz
COPY --from=builder /lib.tar.xz /everest/lib.tar.xz
COPY --from=builder /usr.tar.xz /everest/usr.tar.xz
COPY --from=builder /var/app/docker/scripts/entrypoint.sh /bin/entrypoint
COPY --from=builder /var/app/docker/scripts/extract.sh /bin/extract
COPY --from=builder /var/app/docker/scripts/run-plugin.sh /bin/run-plugin
COPY --from=builder /app.tar.xz /everest/app.tar.xz

RUN apt-get update \
RUN set -x \
&& apt-get update \
&& apt-get install xz-utils \
&& rm -rf /var/lib/apt/lists/* \
&& rm /bin/sh \
&& ln -s /bin/bash /bin/sh \
&& chmod a+x /bin/extract \
51 changes: 24 additions & 27 deletions docker/scripts/extract.sh
Original file line number Diff line number Diff line change
@@ -2,41 +2,38 @@

source /promises.sh

if [ -f /everest/lib.tar.xz ] && [ -f /everest/usr.tar.xz ] && [ -f /everest/app.tar.xz ]; then
init_promises "strict"
init_promises "strict"

echo "[ℹ] Decompressing system.."
cd /everest
cd /everest

promise_run tar xf lib.tar.xz
promise_then cp -r lib/* ../lib/.
files=$(ls -I app.tar.xz *.tar.xz)

promise_run tar xf usr.tar.xz
promise_then cp -r usr/* ../usr/.
echo "[ℹ] Decompressing system.."
for file in $files;
do
basename="${file%.tar.xz}"

echo "[ℹ] Decompressing Kuzzle.."
cd /everest
promise_run tar xf app.tar.xz
promise_run tar xf $file
promise_then cp -r $basename/* /$basename/.
done

await_promises
echo "[ℹ] Decompressing Kuzzle.."

echo "[ℹ] Copying Kuzzle.."
mkdir -p /var/app
for file in bin config default.config.js docker lib node_modules package-lock.json package.json; do
promise_run cp -r var/app/$file ../var/app/$file
done
promise_run tar xf app.tar.xz

mkdir -p /var/app/plugins/enabled
mkdir -p /var/app/plugins/available
promise_run cp -r var/app/plugins/available/* ../var/app/plugins/available/.
promise_run cp -r var/app/plugins/enabled/* ../var/app/plugins/enabled/.
await_promises

await_promises
echo "[ℹ] Copying Kuzzle.."
mkdir -p /var/app
for file in bin config default.config.js docker lib node_modules package-lock.json package.json; do
promise_run cp -r var/app/$file ../var/app/$file
done

echo "[✔] System ready"
else
mkdir -p /var/app/plugins/enabled
mkdir -p /var/app/plugins/available
promise_run cp -r var/app/plugins/available/* ../var/app/plugins/available/.
promise_run cp -r var/app/plugins/enabled/* ../var/app/plugins/enabled/.

echo "[x] Missing /system/system.tar.xz and /everest/app.tar.xz"
exit 1
await_promises

fi
echo "[✔] System ready"
4 changes: 2 additions & 2 deletions docker/scripts/run-dev.sh
Original file line number Diff line number Diff line change
@@ -10,11 +10,11 @@ if [ ! -z "$TRAVIS" ] || [ ! -z "$REBUILD" ]; then
npm ci --unsafe-perm
chmod -R 777 node_modules/
npm rebuild all --unsafe-perm
docker-compose/scripts/install-plugins.sh
./docker/scripts/install-plugins.sh
elif [ ! -d "./node_modules/" ]; then
git submodule update --init --recursive
npm ci --unsafe-perm
./docker-compose/scripts/install-plugins.sh
./docker/scripts/install-plugins.sh
fi

echo "[$(date)] - Starting Kuzzle..."