Flow
Since the CI + CD workflow is very complex, to simplify and understand it, we
can assume it to take place across several stages.
flowchart TD
preparation[Preparation] --- a[ ]
%% a is a fork
a --> frontend_tests[Frontend tests] & docker_preparation[Docker preparation] & documentation_tests[Documentation tests]
docker_preparation --> dockerised_tests[Dockerised tests]
frontend_tests --- e[ ]
%% e is a fork
dockerised_tests --- f[ ]
%% f is a fork
e & f --- b[ ]
%% b is a join
b --> docker_publishing[Docker publishing] --> deployment[Deployment]
documentation_tests ---- c[ ]
e & f --- c
%% c is a join
c --> documentation_emit[Documentation emit]
documentation_emit & deployment --- d[ ]
d --> notification[Notification]
style a height:0,width:0
style b height:0,width:0
style c height:0,width:0
style d height:0,width:0
style e height:0,width:0
style f height:0,width:0
Caution
The stages are a learning tool, not a true representation of how CI works. In
reality, there is no clear lines between stages. Jobs in a later stage will
start as soon as their dependencies are satisfied, without waiting for the full
previous stage to complete first.
For example, the Docker publishing stage starts as soon as
the tests it depends on have passed, even if there are other jobs still running
in the previous stages (like playwright from the
frontend tests stage).
Read on to know more about these stages and the jobs that constitute them. Each
section is also accompanied by a flowchart to help visualise the jobs that
comprise a stage and their relationship with job in prior stages on which this
stage depends.
Preparation
This is the stage of jobs get executed at the very start of the workflow. Since
they are depended upon by other jobs, once this stage is complete, we can use
their resulting state and outputs wherever needed.
flowchart TD
subgraph preparation[Preparation]
get-changes --> add-stack-label
get-image-tag
lint
end
Jobs:
Documentation tests
The documentation tests run outside the Docker containers, so they don’t need to
wait for the Docker containers to be build. This stage happens in parallel with
the frontend tests and the
Docker preparation stages.
These tests are only executed if the documentation has changed. Else they will
be skipped.
flowchart TD
subgraph preparation[Preparation]
lint
get-changes
end
get-changes -- documentation == true || frontend == true --- x
lint --- x
x[ ] --> build-docs
subgraph documentation_tests[Documentation tests]
build-docs
end
style x height:0
style preparation opacity:0.3
Jobs:
Frontend tests
The frontend tests run outside the Docker containers, so they don’t need to wait
for Docker containers to be built. This stage happens in parallel with the
Docker preparation stage. The Playwright tests are very
time-consuming so this stage extends long enough to be contemporary with the
Dockerised tests stage.
These tests are only executed if the frontend has changed. Else they will be
skipped and bypass jobs for nuxt-build and playwright will run instead.
flowchart TD
subgraph preparation[Preparation]
lint
get-changes
end
get-changes -- frontend == true --- x
lint --- x
x[ ] --> nuxt-checks & nuxt-build & playwright
subgraph frontend_tests[Frontend tests]
nuxt-checks
nuxt-build
playwright
end
subgraph bypass_jobs[Bypass jobs]
bypass-nuxt-checks
bypass-playwright
end
nuxt-checks -- skipped --> bypass-nuxt-checks
playwright -- skipped --> bypass-playwright
style x height:0
style preparation opacity:0.3
Jobs:
Docker preparation
A number of our jobs test the services running inside Docker containers, so
before we can run those jobs in Dockerised tests, we must
prepare the Docker images for these services.
The determine-images job determines the images to build (and also publish, see
section on Docker publishing below) based on the changes
observed by get-changes. This information is passed to the build-images job,
which builds the images.
If determine-images finds no images to build, which can happen if there are no
changes to the catalog, ingestion server, API or frontend, the build-images
job will be skipped.
flowchart TD
subgraph preparation[Preparation]
get-changes
get-image-tag
lint
end
subgraph docker_preparation[Docker preparation]
determine-images
build-images
end
get-changes -- changes --> determine-images -- do_build,build_matrix --> build-images
get-image-tag -- image_tag --> build-images
lint --> build-images
style preparation opacity:0.3
Jobs:
Dockerised tests
The tests for the catalog, ingestion server and API all use Docker containers of
their services in the tests. Thus, this stage of testing occurs after the
Docker preparation stage. Unlike the frontend tests, they
do not depend on the lint job, as that is already present as a dependency of
the build-images job.
flowchart TD
subgraph preparation[Preparation]
get-changes
end
subgraph docker_preparation[Docker preparation]
build-images
end
get-changes -- catalog == true --- x
x[ ] --> test-cat & catalog-checks
get-changes -- ingestion_server == true --- y
y[ ] --> test-ing
get-changes -- api == true --- z
z[ ] --> test-api & django-checks
build-images -- success --- x & y & z
subgraph dockerised_tests[Dockerised tests]
test-cat
catalog-checks
test-ing
test-api
django-checks
end
subgraph bypass_jobs[Bypass jobs]
bypass-django-checks
end
django-checks -- skipped --> bypass-django-checks
style x height:0
style y height:0
style z height:0
style preparation opacity:0.3
style docker_preparation opacity:0.3
Jobs:
Documentation emit
After all the proof-of-functionality tests have
concluded, we can initiate the stage of publishing the new documentation. The
documentation is published to the docs site
side-by-side with the publishing of the Docker images.
flowchart TD
subgraph preparation[Preparation]
get-changes
end
subgraph documentation_tests[Documentation tests]
build-docs
end
subgraph frontend_tests[Frontend tests]
nuxt-build
end
subgraph dockerised_tests[Dockerised tests]
test-cat
test-ing
test-api
end
subgraph documentation[Documentation]
emit-docs
end
get-changes -- documentation == true || frontend == true --> emit-docs
build-docs -- success --> emit-docs
test-cat & test-ing & test-api & nuxt-build -. success/skipped .-> emit-docs
style preparation opacity:0.3
style frontend_tests opacity:0.3
style dockerised_tests opacity:0.3
style documentation_tests opacity:0.3
Jobs:
Docker publishing
In this stage we publish the Docker images that have passed the
proof-of-functionality tests to
GHCR. We
publish the new developer docs alongside these images.
The determine-images job determines the images to publish (and also build, see
section on Docker preparation above) based on the changes
observed by get-changes. This information is passed to the publish-images
job, which publishes the images.
If determine-images finds no images to publish, which can happen if there are
no changes to the catalog, ingestion server, API or frontend, the
publish-images job will be skipped.
flowchart TD
subgraph preparation[Preparation]
get-image-tag
end
subgraph frontend_tests[Frontend tests]
nuxt-build
end
subgraph docker_preparation[Docker preparation]
determine-images
build-images
end
subgraph dockerised_tests[Dockerised tests]
test-cat
test-ing
test-api
end
subgraph docker_publishing[Docker publishing]
publish-images
end
get-image-tag -- image_tag --> publish-images
determine-images -- do_publish,publish_matrix --> publish-images
build-images -- success --> publish-images
test-cat & test-ing & test-api & nuxt-build -. success/skipped .-> publish-images
style preparation opacity:0.3
style docker_preparation opacity:0.3
style frontend_tests opacity:0.3
style dockerised_tests opacity:0.3
Jobs:
Deployment
This stage is specifically for the services running on ECS, namely the API and
the frontend. If either of these services have been updated, the workflow also
updates their staging environments to use the Docker images published in the
Docker publishing stage.
flowchart TD
subgraph preparation[Preparation]
get-changes
get-image-tag
end
subgraph deployment[Deployment]
deploy-api
deploy-frontend
end
subgraph docker_publishing[Docker publishing]
publish-images
end
subgraph frontend_tests[Frontend tests]
playwright
end
get-image-tag -- image_tag --> deploy-api & deploy-frontend
get-changes -- api == true --> deploy-api
get-changes -- frontend == true --> deploy-frontend
publish-images -- success --> deploy-api
publish-images -- success --> deploy-frontend
playwright -- success --> deploy-frontend
style preparation opacity:0.3
style docker_publishing opacity:0.3
style frontend_tests opacity:0.3
Jobs:
Notification
At this stage, the workflow should have completed all its objectives and
delivered on all four of its key outcomes. In this stage we compare our
expectations set by the get-changes job with the actual results of the
workflow and report any discrepancies via Slack.
flowchart TD
subgraph preparation[Preparation]
get-changes
end
subgraph docker_preparation[Docker preparation]
determine-images
end
subgraph documentation_publishing[Documentation publishing]
emit-docs
end
subgraph docker_publishing[Docker publishing]
publish-images
end
subgraph deployment[Deployment]
deploy-api
deploy-frontend
end
subgraph notification[Notification]
send-report
end
get-changes -- frontend,documentation,api --> send-report
determine-images -- do_publish --> send-report
emit-docs -- !success --> send-report
publish-images -- !success --> send-report
deploy-api & deploy-frontend -- !success --> send-report
style preparation opacity:0.3
style docker_preparation opacity:0.3
style documentation_publishing opacity:0.3
style docker_publishing opacity:0.3
style deployment opacity:0.3
Jobs:
Bypass jobs
If a job is marked as a required check in GitHub it must
either pass or be skipped
to allow the PR to be merged. This is different for matrix jobs because if a
matrix is skipped due to an if condition, it is not expanded into individual
jobs but skipped as a whole, leaving the checks associated with that matrix in a
pending state, preventing PRs from being merged.
For such jobs, we use a bypass job, conventionally named bypass-<job name>,
that is run on the opposite of the condition of the original job <job name>.
This bypass job is an identical matrix, except it always succeeds, and satisfies
the required checks by having the same job names as the original.