Skip to content

Commit 82aa2d5

Browse files
sichapmansichapman
authored
Merge pull request #124 from LandRegistry/develop
1.20.0
2 parents 53e74ec + d0d3288 commit 82aa2d5

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

43 files changed

+389
-42
lines changed

‎.gitignore

+3
Original file line numberDiff line numberDiff line change
@@ -15,3 +15,6 @@ logs/
1515
.vscode/
1616
.idea/
1717
.after-up-once
18+
node_modules/
19+
package.json
20+
package-lock.json

‎README.md

+19-10
Original file line numberDiff line numberDiff line change
@@ -75,31 +75,31 @@ For an application repository to leverage the full power of the dev-env...
7575

7676
Docker containers are used to run all apps. So some files are needed to support that.
7777

78-
##### `/fragments/docker-compose-fragment.yml`
78+
##### `/fragments/compose-fragment.yml`
7979

80-
This is used by the environment to construct an application container and then launch it. Standard [Compose file](https://docs.docker.com/compose/compose-file/) structure applies - and all apps must use the same Compose file version (which must be 2) - but some recommendations are:
80+
This is used by the environment to construct an application container and then launch it. Standard [Compose Spec](https://github.com/compose-spec/compose-spec/blob/master/spec.md) structure applies - but some recommendations are:
8181

8282
* Container name and service name should match
8383
* Any ports that need to be accessed from the host machine (as opposed to from other containers) should be mapped
8484
* A `volumes` entry should map the path of the app folder to wherever the image expects source files to be (if they are to be accessed dynamically, and not copied in at image build time)
8585
* If the provided log collator is to be used, then a syslog logging driver needs to be present, forwarding to logstash:25826.
86-
* If you wish to run the container as the host user so you have full access to any files created by the container (this is only a problem on Linux and Windows), environment variables `OUTSIDE_UID` and `OUTSIDE_GID` are provided for use in the fragment as build args (which can then be used in the `Dockerfile` to create a matching user and set them as the container-executor).
86+
* If you wish to run the container as the host user so you have full access to any files created by the container (this is only a problem on Linux and WSL), environment variables `OUTSIDE_UID` and `OUTSIDE_GID` are provided for use in the fragment as build args (which can then be used in the `Dockerfile` to create a matching user and set them as the container-executor).
8787

88-
Although generally an application should only have itself in it's compose fragment, there is no reason why other containers based on other Docker images cannot also be listed in this file, if they are not provided already by the dev-env.
88+
Although generally an application should only have itself in its compose fragment, there is no reason why other containers based on other Docker images cannot also be listed in this file, if they are not provided already by the dev-env.
8989

9090
Note that when including directives such as a Dockerfile build location or host volume mapping for the source code, the Compose context root `.` is considered to be the dev-env's /apps/ folder, not the location of the fragment. Ensure relative paths are set accordingly.
9191

92-
[Example](snippets/docker-compose-fragment.yml)
92+
[Example](snippets/compose-fragment.yml)
9393

94-
##### `/fragments/docker-compose-fragment.3.7.yml`
94+
##### `/fragments/docker-compose-fragment.yml` and `/fragments/docker-compose-fragment.3.7.yml`
9595

96-
An optional variant of `docker-compose-fragment.yml` with a version of `3.7`. The development environment will select the highest compose file version supplied by all applications in the environment. If all applications supply a `docker-compose-fragment.3.7.yml`, then the environment will use the `3.7` files, otherwise it falls back to the version `2` compose files.
97-
98-
Compose 3.7 support requires Docker engine version 18.06.0 or later.
96+
Optional variants of `compose-fragment.yml` with a version of `2` and `3.7` respectively. Support for these is still present for backwards compatibility with older apps.The development environment will select the highest compose file version supplied by _all_ applications in the environment (2 --> 3.7 --> unversioned).
9997

10098
If the environment cannot identify a universal compose file version, then provisioning will fail.
10199

102-
[Example](snippets/docker-compose-fragment.3.7.yml)
100+
[2 Example](snippets/docker-compose-fragment.yml)
101+
102+
[3.7 Example](snippets/docker-compose-fragment.3.7.yml)
103103

104104
##### `/Dockerfile`
105105

@@ -133,6 +133,7 @@ The list of allowable commodity values is:
133133
16. cadence-web
134134
17. activemq
135135
18. ibmmq
136+
19. localstack
136137

137138
* The file may optionally also indicate that one or more services are resource intensive ("expensive") when starting up. The dev env will start those containers seperately - 3 at a time - and wait until each are declared healthy (or crash and get restarted 10 times) before starting any more. This requires a healthcheck command specified here or in the Dockerfile/docker-compose-fragment (in which case just use 'docker' in this file).
138139
* If one of these expensive services prefers another one to be considered "healthy" before a startup attempt is made (such as a database, to ensure immediate connectivity and no expensive restarts) then the dependent service can be specified here, with a healthcheck command following the same rules as above.
@@ -304,6 +305,13 @@ cadence core services.
304305
*Running Cadence web locally*
305306
- In a web browser enter http://localhost:5004
306307

308+
###### Localstack
309+
[Localstack](https://localstack.cloud) is a cloud stack testing and mocking framework for developing against various AWS services.
310+
311+
A default Localstack configuration is provided with a minimal number of enabled services available (S3 only at present). Localstack does not *require* the use of any other external configuration file (as applications can manage buckets programatically through methods such as the [AWS SDK](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/examples-s3-buckets.html)). However, if additional configuration (such as new buckets) are necessary before application startup, you can use a `localstack-init-fragment.sh` to perform this provisioning; an example of which is provided [here](snippets/localstack-init-fragment.sh).
312+
313+
Localstack is available at <http://localstack:4566> within the Docker network, and <http://localhost:4566> on the host.
314+
307315
#### Other files
308316

309317
**`/fragments/custom-provision.sh`**
@@ -336,6 +344,7 @@ If you want to make use of this functionality, ensure that `logstash` is also pr
336344

337345
* Ensure that you give Docker enough CPU and memory to run all your apps.
338346
* The `run.sh destroy` command should be a last resort, as you will have to rebuild all images from scratch. Try the `fullreset` alias as that will just remove your app containers and recreate them. They are most likely to be the source of any corruption. Remember to alter `.commodities.yml` and `.custom_provision.yml` if you need to, and `run.sh reload`.
347+
* A memory limit of 384mb is set for intermediate containers during the image build process - but only if using Docker Compose V1 and you have Buildkit disabled in advanced Docker settings.
339348

340349
### Useful commands
341350

‎apps/root-compose-fragment.yml

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
services: {}

‎logic.rb

+14-9
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,7 @@
7676

7777
# Does a version check and self-update if required
7878
if options['self_update']
79-
this_version = '1.15.1'
79+
this_version = '1.20.0'
8080
puts colorize_lightblue("This is a universal dev env (version #{this_version})")
8181
# Skip version check if not on master (prevents infinite loops if you're in a branch that isn't up to date with the
8282
# latest release code yet)
@@ -197,13 +197,16 @@
197197
puts colorize_lightblue('Building images...')
198198
# v2 already builds in parallel
199199
if ENV['DC_VERSION'] == '2'
200-
if run_command("#{ENV['DC_CMD']} build " + (options['nopull'] ? '' : '--pull')) != 0
201-
puts colorize_red('Something went wrong when building your app images. Check the output above.')
202-
exit
200+
if run_command("#{ENV['DC_CMD']} build --memory 384m " + (options['nopull'] ? '' : '--pull')) != 0
201+
puts colorize_yellow('Build command failed. Trying without a memory limit')
202+
if run_command("#{ENV['DC_CMD']} build " + (options['nopull'] ? '' : '--pull')) != 0
203+
puts colorize_red('Something went wrong when building your app images. Check the output above.')
204+
exit
205+
end
203206
end
204207
else
205-
if run_command("#{ENV['DC_CMD']} build --parallel " + (options['nopull'] ? '' : '--pull')) != 0
206-
puts colorize_yellow('Build command failed. Trying without --parallel')
208+
if run_command("#{ENV['DC_CMD']} build --memory 384m --parallel " + (options['nopull'] ? '' : '--pull')) != 0
209+
puts colorize_yellow('Build command failed. Trying without --parallel and no memory limit')
207210
# Might not be running a version of compose that supports --parallel, try one more time
208211
if run_command("#{ENV['DC_CMD']} build " + (options['nopull'] ? '' : '--pull')) != 0
209212
puts colorize_red('Something went wrong when building your app images. Check the output above.')
@@ -212,7 +215,6 @@
212215
end
213216
end
214217

215-
216218
end
217219

218220
if options['provision_commodities']
@@ -351,10 +353,13 @@
351353
if service_healthy
352354
puts colorize_green('It is!')
353355
else
354-
puts colorize_yellow('Not yet')
355-
# Check if the container has crashed and restarted
356356
output_lines = []
357+
run_command("docker logs --tail 1 #{service['compose_service']}",
358+
output_lines)
359+
puts colorize_yellow("Not yet (Last log line: #{output_lines[0]})")
360+
# Check if the container has crashed and restarted
357361
restart_count = 0
362+
output_lines = []
358363
run_command("docker inspect --format=\"{{json .RestartCount}}\" #{service['compose_service']}",
359364
output_lines)
360365
# Find the count in all the lines that have come out

‎scripts/add-aliases.sh

+5
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,10 @@ function manage(){
8888
ex ${1} python3 manage.py ${@:2}
8989
}
9090

91+
function localstack(){
92+
ex localstack awslocal ${@:1}
93+
}
94+
9195
function fullreset(){
9296
stop ${1}
9397
remove ${1}
@@ -146,5 +150,6 @@ function devenv-help(){
146150
add-to-docker-compose
147151
<name of new compose fragment> - looks in fragments folder of loaded apps to search for a new docker-compose-fragment including the provided parameter eg docker-compose-syt2-fragment then runs docker-compose up
148152
cadence-cli - runs the command line tool to interact with cadence orchestrator
153+
localstack - run localstack (aws) commands in the localstack container
149154
EOF
150155
}

‎scripts/commodities.rb

+3
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@
99
require_relative 'provision_elasticsearch5'
1010
require_relative 'provision_elasticsearch'
1111
require_relative 'provision_wiremock'
12+
require_relative 'provision_localstack'
1213

1314
require 'fileutils'
1415
require 'open3'
@@ -155,6 +156,8 @@ def provision_commodities(root_loc, new_containers)
155156
provision_wiremock(root_loc, new_containers)
156157
# Hosts File
157158
provision_hosts(root_loc)
159+
# Localstack
160+
provision_localstack(root_loc, new_containers)
158161
end
159162

160163
def container_to_commodity(container_name)

‎scripts/docker/activemq/compose-fragment.yml

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
services:
2+
activemq:
3+
container_name: activemq
4+
build: ../scripts/docker/activemq/
5+
ports:
6+
- "61616:61616"
7+
- "8161:8161"
8+
- "6672:5672"
9+
- "61613:61613"
10+
- "1883:1883"
11+
- "61614:61614"
12+
platform: "linux/amd64"

‎scripts/docker/auth/compose-fragment.yml

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
services:
2+
openldap:
3+
container_name: openldap
4+
build: ../scripts/docker/auth/openldap
5+
ports:
6+
- "1389:389"
7+
8+
keycloak:
9+
container_name: keycloak
10+
build: ../scripts/docker/auth/keycloak
11+
ports:
12+
- "8180:8080"
13+
depends_on:
14+
- openldap
15+
platform: "linux/amd64"

‎scripts/docker/auth/openldap/Dockerfile

+4-1
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,10 @@ RUN rm -rf /etc/ldap/slapd.d/* && \
2121
rm /etc/ldap/config.ldif && \
2222
chown -R openldap:openldap /etc/ldap/slapd.d /var/run/openldap
2323

24+
COPY tini /
25+
RUN chmod +x tini
26+
2427
EXPOSE 389
2528
VOLUME /var/lib/ldap
2629

27-
ENTRYPOINT ["slapd", "-u", "openldap", "-g", "openldap", "-h", "ldap:///", "-d", "stats,stats2"]
30+
ENTRYPOINT ["/tini", "--", "slapd", "-u", "openldap", "-g", "openldap", "-h", "ldap:///", "-d", "stats,stats2"]

‎scripts/docker/auth/openldap/tini

23.5 KB
Binary file not shown.

‎scripts/docker/cadence-web/compose-fragment.yml

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
services:
2+
cadence-web:
3+
container_name: cadence-web
4+
build: ../scripts/docker/cadence-web/
5+
environment:
6+
- "CADENCE_TCHANNEL_PEERS=cadence:7933"
7+
ports:
8+
- "5004:8088"
9+
depends_on:
10+
- cadence
11+
platform: "linux/amd64"

‎scripts/docker/cadence/compose-fragment.yml

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
services:
2+
cadence:
3+
container_name: cadence
4+
build: ../scripts/docker/cadence/
5+
ports:
6+
- "7933:7933"
7+
- "7934:7934"
8+
- "7935:7935"
9+
- "7939:7939"
10+
env_file:
11+
- ../scripts/docker/cadence/.env_list
12+
depends_on:
13+
postgres-13:
14+
condition: service_healthy
15+
platform: "linux/amd64"

‎scripts/docker/db2_community/Dockerfile

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
FROM hmlandregistry/db2-cgroupaware:11.5.7.0
1+
FROM hmlandregistry/db2-cgroupaware:11.5.7.0a
22

33
EXPOSE 50000 55000
44

‎scripts/docker/db2_community/compose-fragment.yml

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
services:
2+
db2_community:
3+
container_name: db2_community
4+
build: ../scripts/docker/db2_community/
5+
ports:
6+
- "50002:50000"
7+
- "55002:55000"
8+
env_file:
9+
- ../scripts/docker/db2_community/.env_list
10+
privileged: true
11+
platform: "linux/amd64"
12+
deploy:
13+
resources:
14+
limits:
15+
memory: 2GB

‎scripts/docker/db2_devc/compose-fragment.yml

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
services:
2+
db2_devc:
3+
container_name: db2_devc
4+
build: ../scripts/docker/db2_devc/
5+
ports:
6+
- "50001:50000"
7+
- "55001:55000"
8+
env_file:
9+
- ../scripts/docker/db2_devc/.env_list
10+
privileged: true
11+
platform: "linux/amd64"

‎scripts/docker/elasticsearch/compose-fragment.yml

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
services:
2+
elasticsearch:
3+
container_name: elasticsearch
4+
build: ../scripts/docker/elasticsearch/
5+
ports:
6+
- "9200:9200"
7+
- "9300:9300"
8+
platform: "linux/amd64"

‎scripts/docker/elasticsearch5/compose-fragment.yml

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
services:
2+
elasticsearch5:
3+
container_name: elasticsearch5
4+
build: ../scripts/docker/elasticsearch5
5+
ports:
6+
- "9202:9200"
7+
- "9302:9300"
8+
restart: on-failure
9+
platform: "linux/amd64"

‎scripts/docker/ibmmq/Dockerfile

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
FROM ibmcom/mq:9.2.0.0-r1
1+
FROM ibmcom/mq:9.2.4.0-r1
22

33
# Auto-accept the license
44
# Create default users and channels

‎scripts/docker/ibmmq/compose-fragment.yml

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
services:
2+
ibmmq:
3+
container_name: ibmmq
4+
build: ../scripts/docker/ibmmq/
5+
ports:
6+
- "1414:1414"
7+
- "9443:9443"
8+
- "9157:9157"
9+
platform: "linux/amd64"

‎scripts/docker/localstack/.env_list

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
SERVICES=s3
2+
DEBUG=1
3+
DATA_DIR=/tmp/localstack/data

‎scripts/docker/localstack/Dockerfile

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
FROM localstack/localstack:0.14.0
2+

‎scripts/docker/localstack/compose-fragment.yml

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
services:
2+
localstack:
3+
container_name: localstack
4+
build: ../scripts/docker/localstack/
5+
ports:
6+
- "4566:4566"
7+
env_file: ../scripts/docker/localstack/.env_list

‎scripts/docker/localstack/docker-compose-fragment.3.7.yml

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
version: '3.7'
2+
services:
3+
localstack:
4+
container_name: localstack
5+
build: ../scripts/docker/localstack/
6+
ports:
7+
- "4566:4566"
8+
env_file: ../scripts/docker/localstack/.env_list

‎scripts/docker/localstack/docker-compose-fragment.yml

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
version: '2'
2+
services:
3+
localstack:
4+
container_name: localstack
5+
build: ../scripts/docker/localstack/
6+
ports:
7+
- "4566:4566"
8+
env_file: ../scripts/docker/localstack/.env_list

‎scripts/docker/logging/Dockerfile

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
FROM python:3.9.5-slim
1+
FROM python:3.10.4-slim
22
ENV PYTHONUNBUFFERED yes
33

44
# Add Tini

‎scripts/docker/logging/compose-fragment.yml

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
services:
2+
# Back-compat: So apps compose files dont need to care if its logstash or not, call this container logstash as well
3+
logstash:
4+
container_name: logstash
5+
build: ../scripts/docker/logging
6+
volumes:
7+
- ../logs:/log-dir
8+
ports:
9+
- "25826:25826"
10+
restart: on-failure

‎scripts/docker/nginx/Dockerfile

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
FROM nginx:1.16
1+
FROM nginx:1.20
22

33
RUN apt-get update && apt-get install openssl && \
44
rm /etc/nginx/conf.d/default.conf && \

‎scripts/docker/nginx/compose-fragment.yml

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
services:
2+
nginx:
3+
container_name: nginx
4+
build: ../scripts/docker/nginx/
5+
ports:
6+
- "80:80"
7+
- "443:443"

‎scripts/docker/nginx/server.conf

+3-10
Original file line numberDiff line numberDiff line change
@@ -13,21 +13,14 @@ server {
1313
listen *:443 ssl;
1414
server_name _;
1515

16-
ssl on;
17-
1816
# We generated these during docker image creation (see dockerfile)
1917
ssl_certificate /etc/nginx/ssl/ssl.crt;
2018
ssl_certificate_key /etc/nginx/ssl/ssl.key;
2119

22-
# The following from https://cipherli.st/
23-
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
20+
ssl_protocols TLSv1.2 TLSv1.3;
2421
ssl_prefer_server_ciphers on;
25-
ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
26-
ssl_ecdh_curve secp384r1; # Requires nginx >= 1.1.0
27-
ssl_session_cache shared:SSL:10m;
28-
ssl_session_tickets off; # Requires nginx >= 1.5.9
29-
ssl_stapling on; # Requires nginx >= 1.3.7
30-
ssl_stapling_verify on; # Requires nginx => 1.3.7
22+
ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA HIGH !RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS";
23+
3124
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
3225
add_header X-Frame-Options DENY;
3326
add_header X-Content-Type-Options nosniff;

0 commit comments

Comments
 (0)