Deployment runbook#

Setup#

  1. Check Airflow to make sure a data refresh isn’t occurring.

  2. Visit the Ingestion Server Docker image page and copy the SHA of the image tagged latest.

  3. Release the app via GitHub workflow. Click the “Run workflow” button, choose “ingestion_server” from the dropdown, and supply the SHA identified in the previous step.

Deployment#

  1. After the app is built and tagged, deploy staging:

    1. Checkout the infrastructure repository and bump the ingestion server version with the just bump dev ingestion-server command.

    2. just apply dev ingestion-server and verify the plan before deploying.

  2. Deploy production:

    1. Update the value of data_refresh_cleared to true in the production module declaration.

    2. just bump prod ingestion-server command.

    3. just apply prod ingestion-server and verify the plan before deploying.

    4. Restore the value of data_refresh_cleared back to false.

Post-deployment steps#

  1. Check for any Sentry errors in the maintainer’s #openverse-alerts channel, or in the Sentry UI.

  2. Review and Approve the automatically-generated changelog pull request in the repository.

  3. Push up a PR to the infrastructure repository with the Terraform changes you pushed (the version bump for the relevant module). Be sure to restore the value of data_refresh_cleared back to false.

  4. In the event of errors or problems, repeat the deployment process using the latest stable version of the application. You can find the release version number in the changelogs, and then the tag to pass to the action is the version number prefixed with “rel-”, for example “rel-2023.07.03.17.52.00”.

  5. If anything else goes wrong or service is disrupted, consider this a Production Incident and notify the team.