Container Roles

build-container-image

Build one or more container images.

This is one of a collection of roles which are designed to work together to build, upload, and promote container images in a gating context:

All roles accept the same input data, principally a list of dictionaries representing the images to build. YAML anchors can be used to supply the same data to all three roles.

Building

The build-container-image role is designed to be used in check and gate pipelines and simply builds the images. It can be used to verify that the build functions, or it can be followed by the use of subsequent roles to upload the images to a registry.

Uploading

The upload-container-image role uploads the images to a registry. It can be used in one of two modes:

  1. Using tags as part of a two-step promote pipeline. This mode is designed to minimize the time the published registry tag is out of sync with the changes Zuul has merged to the underlying code repository.

    In this mode, the role is intended to run in the gate pipeline. Zuul will build and upload the resulting image with a single tag prefixed with the change ID (e.g. change_12345_<tag>). Thus at the completion of the gate job, all the layers of the new container are uploaded, but the <tag> in the remote repository will not have been updated.

    Once the gate queue is successfully finished Zuul will merge the change to the code-repository. At this point, a small window opens where the <tag> is pointing to a container that does not reflect the state of the code-repository. The merge of the change will trigger the promote pipeline which uses a very quick, executor-only job to retag <tag> to change_12345_<tag>. Since this step does not require any nodes or upload any data, it generally takes only a few seconds. The remote container pointed to by <tag> will now reflect the underlying code closing the out-of-sync window.

    When running in this mode uploads are only made if promote_container_image_method is unset or set to tag. Otherwise we skip upload to the registry.

  2. The second mode allows for use of this job in release and tag pipelines to directly upload a release build with the final set of tags.

    In this mode, upload_container_image_promote: false should be set. The role will build and upload the resulting image to the remote repository with the final tags.

    This should be used with tag and release pipelines, where committed code has been tagged for publishing. The tagged commit is “known good” thanks to gating, so the build and upload process is expected to work unconditionally.

    This can be used in a post-commit pipeline, with the caveat that it has a much longer window where published code is out of sync with the published image, as the image must be completely rebuilt and uploaded after code merge in the gate job.

    The alternative promote method can be thought of as a “speculative” upload. There is a possibility the gate job uploads layers and creates a temporary tag, but either the container upload or another co-gating job fails, causing the gate jobs to fail overall. This causes extra uploads, unsued layers and unused tags that require cleaning up. Since changes have merged before the release pipeline starts, the upload will simply not run if the gate jobs fail. This avoids uploading or tagging anything that will not be used. The trade-off is a higher latency between merging code and publishing final tags.

    Transient network failures can cause upload errors in both cases. Although the promote job may fail, leaving the tag incorrectly unmodified, the promote job’s relatively simplicity minimises potential error. The release pipeline does more work, exposing it to a higher chance of failures such as transient network errors etc., also resulting in the repository tag being out-of-date. In both cases developers must pay close attention as failures in these pipelines are often less noticable than code not merging with a gate-job failure.

Promoting

As discussed above, the promote-container-image role is designed to be used in a promote pipeline.

In tag mode, it re-tags a previously uploaded image by copying the temporary change-id based tags made during upload to the final production tags supplied by build-container-image.container_images.tags. It is intended to run very quickly and with no dependencies, so it can run directly on the Zuul executor.

Once this role completes, the temporary upload tags are no longer required. The role removes the change-id tags from the repository in the registry, and removes any similar change-ids tags. This keeps the repository tidy in the case that gated changes fail to merge after uploading their staged images. Remvoing these tags is a registry specific operation. You should double check the api_token requirements for your registry described below. For more details see remove-registry-tag.

In intermediate-registry mode, this role queries Zuul to find the build performed by the build role in the gate. It then copies this image from the intermediate-registry to the final location in the remote registry.

Dependencies

The build and upload roles require a container runtime that should be installed before use; for example by using either the ensure-docker or ensure-podman roles. The promote job assumes skopeo is available on the executor.

Role Variables

zuul_work_dir
Default: {{ zuul.project.src_dir }}

The project directory. Serves as the base for build-container-image.container_images.context.

container_filename

The default container filename name to use. Serves as the base for build-container-image.container_images.container_filename. This allows a global overriding of the container filename name, for example when building all images from different folders with similarily named containerfiles.

If omitted, the default depends on the container command used. Typically, this is Dockerfile for docker and Containerfile (with a fallback on Dockerfile) for podman.

container_command
Default: podman

The command to use when building the image (E.g., docker).

container_registry_credentials
Type: dict

This is only required for the upload and promote roles. This is expected to be a Zuul Secret in dictionary form. Each key is the name of a registry, and its value a dictionary with information about that registry.

Example:

container_registry_credentials:
  quay.io:
    username: foo
    password: bar
container_registry_credentials{}.[registry name]
Type: dict

Information about a registry. The key is the registry name, and its value a dict as follows:

container_registry_credentials{}.[registry name]{}.username

The registry username.

container_registry_credentials{}.[registry name]{}.password

The registry password.

container_registry_credentials{}.[registry name]{}.repository

Optional; if supplied this is a regular expression which restricts to what repositories the image may be uploaded. The following example allows projects to upload images to repositories within an organization based on their own names:

repository: "^myorgname/{{ zuul.project.short_name }}.*"
container_registry_credentials{}.[registry name]{}.api_token

Optional; When using the promote roles, the registry API is used to remove temporary tags. if your registry requires a token to talk to the registry API, add it here. This is registry dependent; some allow API access via the username/password, but others require issuing a separate token. For more details see remove-registry-tag. Some examples:

  • docker : API is access via username/password, does not require token.

  • quay.io : A token must be generated from an “application” that a user has allowed to operate on its behalf. See https://docs.quay.io/api/.

container_images
Type: list

A list of images to build. Each item in the list should have:

container_images[].context

The build context; this should be a directory underneath build-container-image.zuul_work_dir.

container_images[].container_filename

The filename of the container file, present in the context folder, used for building the image. Provide this if you are using a non-standard filename for a specific image.

container_images[].registry

The name of the target registry (E.g., quay.io). Used by the upload and promote roles.

container_images[].repository

The name of the target repository in the registry for the image. Supply this even if the image is not going to be uploaded (it will be tagged with this in the local registry). This should include the registry name. E.g., quay.io/example/image.

container_images[].path

Optional: the directory that should be passed to the build command. Useful for building images with a container file in the context directory but a source repository elsewhere.

container_images[].build_args
Type: list

Optional: a list of values to pass to the --build-arg parameter.

container_images[].target

Optional: the target for a multi-stage build.

container_images[].tags
Default: ['latest']
Type: list

A list of tags to be added to the image when promoted.

container_images[].siblings
Default: []
Type: list

A list of sibling projects to be copied into {{zuul_work_dir}}/.zuul-siblings. This can be useful to collect multiple projects to be installed within the same Docker context. A -build-arg called ZUUL_SIBLINGS will be added with each sibling project. Note that projects here must be listed in required-projects.

container_build_extra_env
Type: dict

A dictionary of key value pairs to add to the container build environment. This may be useful to enable buildkit with docker builds for example.

promote_container_image_method
Default: tag

A string value indicating whether or not we upload images to the upstream registry pre merge then promote that upload via a retag (tag) or we upload to a downstream registry and later fetch and promote that to the upstream registry post merge (intermediate-registry).

build-docker-image

Build one or more docker images.

This is one of a collection of roles which are designed to work together to build, upload, and promote docker images in a gating context:

The build-docker-image role is designed to be used in check and gate pipelines and simply builds the images. It can be used to verify that the build functions, or it can be followed by the use of subsequent roles to upload the images to Docker Hub.

The upload-docker-image role uploads the images to Docker Hub. It can be used in one of two modes: by default it will upload with a single tag corresponding to the change ID. In this mode, the role role is designed to be used in a job in a gate pipeline so that the build produced by the gate is staged and can later be promoted to production if the change is successful. The other mode allows for use of this job in a release pipeline to directly upload a release build with the final set of tags.

The promote-docker-image role is designed to be used in a promote pipeline. It requires no nodes and runs very quickly on the Zuul executor. It simply re-tags a previously uploaded image for a change with whatever tags are supplied by build-docker-image.docker_images.tags. It also removes the change ID tag from the repository in Docker Hub, and removes any similar change ID tags more than 24 hours old. This keeps the repository tidy in the case that gated changes fail to merge after uploading their staged images.

They all accept the same input data, principally a list of dictionaries representing the images to build. YAML anchors can be used to supply the same data to all three jobs.

Use the ensure-docker role to install Docker before using this role.

Role Variables

zuul_work_dir
Default: {{ zuul.project.src_dir }}

The project directory. Serves as the base for build-docker-image.docker_images.context.

docker_dockerfile
Default: Dockerfile

The default Dockerfile name to use. Serves as the base for build-docker-image.docker_images.dockerfile. This allows a global overriding of Dockerfile name, for example when building all images from different folders with similarily named dockerfiles.

docker_registry
Default: ''

The container registry the images should be tagged for, by default zuul will push the image to dockerhub.

docker_credentials
Type: dict

This is only required for the upload and promote roles. This is expected to be a Zuul Secret with two keys:

docker_credentials{}.username

The Docker Hub username.

docker_credentials{}.password

The Docker Hub password.

docker_credentials{}.repository

Optional; if supplied this is a regular expression which restricts to what repositories the image may be uploaded. The following example allows projects to upload images to repositories within an organization based on their own names:

repository: "^myorgname/{{ zuul.project.short_name }}.*"
docker_use_buildkit
Default: false
Type: bool

Use BuildKit when creating images.

docker_images
Type: list

A list of images to build. Each item in the list should have:

docker_images[].context

The docker build context; this should be a directory underneath build-docker-image.zuul_work_dir.

docker_images[].dockerfile

The filename of the dockerfile, present in the context folder, used for building the image. Provide this if you are using a non-standard filename for a specific image.

docker_images[].repository

The name of the target repository in dockerhub for the image. Supply this even if the image is not going to be uploaded (it will be tagged with this in the local registry).

docker_images[].path

Optional: the directory that should be passed to docker build. Useful for building images with a Dockerfile in the context directory but a source repository elsewhere.

docker_images[].build_args
Type: list

Optional: a list of values to pass to the docker --build-arg parameter.

docker_images[].target

Optional: the target for a multi-stage build.

docker_images[].tags
Default: ['latest']
Type: list

A list of tags to be added to the image when promoted.

docker_images[].siblings
Default: []
Type: list

A list of sibling projects to be copied into {{zuul_work_dir}}/.zuul-siblings. This can be useful to collect multiple projects to be installed within the same Docker context. A -build-arg called ZUUL_SIBLINGS will be added with each sibling project. Note that projects here must be listed in required-projects.

docker_images[].labels
Default: []
Type: list

A list of labels to attach to the built image, in the form of “key=value”.

docker_images[].arch
Default: []
Type: list

A list of architectures to build on. When enabling this on any image, all of them will be built with docker buildx.

Valid values are linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6.

collect-container-logs

An ansible role to collect all container logs.

Role Variables

container_command
Default: docker

Container run-time CLI command

deploy-openshift

Deploy openshift using oc cluster up.

ensure-docker

An ansible role to install docker and configure it to use mirrors if available.

Role Variables

mirror_fqdn
Default: {{ zuul_site_mirror_fqdn }}

The base host for mirror servers.

docker_mirror

URL to override the generated docker hub mirror url based on ensure-docker.mirror_fqdn.

use_upstream_docker
Default: True

By default this role adds repositories to install docker from upstream docker. Set this to False to use the docker that comes with the distro.

docker_use_buildset_registry
Default: False

This role does not enable the usage of the buildset registry by default, this variable allows enabling the usage of the buildset registry after installing Docker.

docker_compose_install
Default: False

This role does not install docker-compose by default but you can use this setting to install docker-compose as well.

docker_update_channel
Default: stable

Which update channel to use for upstream docker. The two choices are stable, which is the default and updates quarterly, and edge which updates monthly.

docker_insecure_registries
Default: undefined

Declare this with a list of insecure registries to define the registries which are allowed to communicate with HTTP only or HTTPS with no valid certificate.

docker_gpg_key
Default: string

The raw content of the upstream docker gpg key, as found here https://download.docker.com/linux/fedora/gpg

docker_distro_packages
Default: list

List of packages to be installed when use_upstream_docker is set to false. The package set is defined by default using distro specific variables. If the package set needs to be changed this option can be overridden as needed.

docker_upstream_distro_required_packages
Default: list

List of packages to be installed when use_upstream_docker is set to true. The package set is defined by default using distro specific variables and contains a list of supporting packages required to be installed prior to installing docker-ce. If the package set needs to be changed this option can be overridden as needed.

docker_upstream_distro_remove_packages
Default: list

List of packages to be removed before installing new ones. It is used for avoiding potential conflicts. For example it can remove docker package before trying to install docker-ce. The default value is distro specific.

docker_upstream_distro_packages
Default: list

List of packages to be installed when use_upstream_docker is set to true. The package set is defined by default using distro specific variables. If the package set needs to be changed this option can be overridden as needed.

docker_download_fqdn
Default: download.docker.com

Add default option to set the docker download fqdn.

docker_mirror_base_url
Default: https://{{ docker_download_fqdn }}/linux/{ubuntu,centos,fedora}

By default this option sets the repository base url. This variable is based on ensure-docker.docker_download_fqdn. When this option is unset, the role will use distro specific variables which are loaded at the time of execution.

docker_userland_proxy
Type: bool

Set to false to disable the docker userland proxy. This variable is useful when docker is causing routing problem, such as when a kubernetes deployment is unable to reach its own service.

ensure-kubernetes

An ansible role to install kubernetes.

Role Variables

ensure_kubernetes_type
Default: minikube

The kubernetes distribution to use. Currently `minikube or `microk8s`. Note that `microk8s` is only implemented for Ubuntu Jammy distributions currently.

ensure_kubernetes_microk8s_channel
Default: latest/stable

The snap channel to use for `microk8s`. See https://microk8s.io/docs/setting-snap-channel.

ensure_kubernetes_microk8s_addons
Default: ['dns', 'storage']

The addons for microk8s`. See https://microk8s.io/docs/addons

install_kubernetes_with_cluster
Default: True

If true, installs a Minikube cluster.

minikube_version
Default: latest

The version of Minikube to install.

minikube_dns_resolvers
Default: []

List of dns resolvers to configure in k8s. Use this to override the resolvers that are found by default.

kubernetes_runtime
Default: docker

Which kubernetes runtime to use for minikube; values are docker or cri-o.

ensure_kubernetes_minikube_addons
Default: []

List of addons to configure in k8s. Use this to enable the addons.

ensure-openshift

Setup openshift requirements and pull the container images. The deploy-openshift role can be used to start the services.

This role only works on CentOS.

Role Variables

origin_version
Default: v3.11.0

The origin version.

ensure-podman

Install podman container manager

Role Variables

ensure_podman_validate
Default: false

Used to enable validation of podman engine.

ensure-skopeo

Install skopeo

Role Variables

ensure_skopeo_install_from_upstream
Default: false

Build and install skopeo from upstream. This installs a go development environment and build dependencies. The binary is installed into /usr/local/bin.

Currently only implemented for Ubuntu Jammy.

ensure_skopeo_install_from_upstream_version
Default: v1.9.3

When building skopeo locally, the version tag to use.

ensure-quay-repo

This role primarily exists to create a new public repository in quay. This role can be used to create private repos as well, but repos are created by default in quay if you simply push to them.

Users of this role will need to generate an application token with create repository permissions. Additional permissions are not necessary.

When invoking this role you should set no_log: true on the include_role task to prevent disclosure of your token.

** Role Variables **

container_registry_credentials
Type: dict

Required. This is expected to be a Zuul secret in dictionary form. For convenience this is in the same format as the container_registry_credentials variable used by the other container roles. Specify an api_token which is issued from an application assigned to an organisation. See https://docs.quay.io/api/ This application API token needs create permissions. If the registry name is quay.io we know that the registry type is quay. If you are running a private Quay installation you can manually set type to quay to force this behavior.

This role will only take action on image repos that map to container registries for which both an api_token is set and type can be determined to be quay.

You may also set api_url on the registry credentials if the API is not hosted at the root of the registry name. Most installations should be able to ignore this and use the default of https://{{ $name }}.

Example:

container_registry_credentials:
  quay.io:
    type: 'quay'
    api_token: 'abcd1234'
    api_url: 'https://quay.io'
container_images
Type: list

Required. A list of dictionaries. This provides info about the image repositories to be created in a quay registry. For convenience this is in the same format as the container_images variable used by other container roles. Specify a registry (this should match up with your credentials to locate the api token), namespace, repo_shortname, repo_description, and visibility attributes. By default visibility will be public.

Example:

container_images:
  - registry: quay.io
    namespace: myquayorg
    repo_shortname: myimage
    repo_description: The best container image
pause-buildset-registry

Pause a buildset registry

Utility role to pause a job providing a buildset registry.

Role Variables

buildset_registry

Location of external buildset registry. If this is defined, the job will not pause.

promote-container-image

Promote one or more previously uploaded container images.

This is one of a collection of roles which are designed to work together to build, upload, and promote container images in a gating context:

All roles accept the same input data, principally a list of dictionaries representing the images to build. YAML anchors can be used to supply the same data to all three roles.

Building

The build-container-image role is designed to be used in check and gate pipelines and simply builds the images. It can be used to verify that the build functions, or it can be followed by the use of subsequent roles to upload the images to a registry.

Uploading

The upload-container-image role uploads the images to a registry. It can be used in one of two modes:

  1. Using tags as part of a two-step promote pipeline. This mode is designed to minimize the time the published registry tag is out of sync with the changes Zuul has merged to the underlying code repository.

    In this mode, the role is intended to run in the gate pipeline. Zuul will build and upload the resulting image with a single tag prefixed with the change ID (e.g. change_12345_<tag>). Thus at the completion of the gate job, all the layers of the new container are uploaded, but the <tag> in the remote repository will not have been updated.

    Once the gate queue is successfully finished Zuul will merge the change to the code-repository. At this point, a small window opens where the <tag> is pointing to a container that does not reflect the state of the code-repository. The merge of the change will trigger the promote pipeline which uses a very quick, executor-only job to retag <tag> to change_12345_<tag>. Since this step does not require any nodes or upload any data, it generally takes only a few seconds. The remote container pointed to by <tag> will now reflect the underlying code closing the out-of-sync window.

    When running in this mode uploads are only made if promote_container_image_method is unset or set to tag. Otherwise we skip upload to the registry.

  2. The second mode allows for use of this job in release and tag pipelines to directly upload a release build with the final set of tags.

    In this mode, upload_container_image_promote: false should be set. The role will build and upload the resulting image to the remote repository with the final tags.

    This should be used with tag and release pipelines, where committed code has been tagged for publishing. The tagged commit is “known good” thanks to gating, so the build and upload process is expected to work unconditionally.

    This can be used in a post-commit pipeline, with the caveat that it has a much longer window where published code is out of sync with the published image, as the image must be completely rebuilt and uploaded after code merge in the gate job.

    The alternative promote method can be thought of as a “speculative” upload. There is a possibility the gate job uploads layers and creates a temporary tag, but either the container upload or another co-gating job fails, causing the gate jobs to fail overall. This causes extra uploads, unsued layers and unused tags that require cleaning up. Since changes have merged before the release pipeline starts, the upload will simply not run if the gate jobs fail. This avoids uploading or tagging anything that will not be used. The trade-off is a higher latency between merging code and publishing final tags.

    Transient network failures can cause upload errors in both cases. Although the promote job may fail, leaving the tag incorrectly unmodified, the promote job’s relatively simplicity minimises potential error. The release pipeline does more work, exposing it to a higher chance of failures such as transient network errors etc., also resulting in the repository tag being out-of-date. In both cases developers must pay close attention as failures in these pipelines are often less noticable than code not merging with a gate-job failure.

Promoting

As discussed above, the promote-container-image role is designed to be used in a promote pipeline.

In tag mode, it re-tags a previously uploaded image by copying the temporary change-id based tags made during upload to the final production tags supplied by build-container-image.container_images.tags. It is intended to run very quickly and with no dependencies, so it can run directly on the Zuul executor.

Once this role completes, the temporary upload tags are no longer required. The role removes the change-id tags from the repository in the registry, and removes any similar change-ids tags. This keeps the repository tidy in the case that gated changes fail to merge after uploading their staged images. Remvoing these tags is a registry specific operation. You should double check the api_token requirements for your registry described below. For more details see remove-registry-tag.

In intermediate-registry mode, this role queries Zuul to find the build performed by the build role in the gate. It then copies this image from the intermediate-registry to the final location in the remote registry.

Dependencies

The build and upload roles require a container runtime that should be installed before use; for example by using either the ensure-docker or ensure-podman roles. The promote job assumes skopeo is available on the executor.

Role Variables

zuul_work_dir
Default: {{ zuul.project.src_dir }}

The project directory. Serves as the base for build-container-image.container_images.context.

container_filename

The default container filename name to use. Serves as the base for build-container-image.container_images.container_filename. This allows a global overriding of the container filename name, for example when building all images from different folders with similarily named containerfiles.

If omitted, the default depends on the container command used. Typically, this is Dockerfile for docker and Containerfile (with a fallback on Dockerfile) for podman.

container_command
Default: podman

The command to use when building the image (E.g., docker).

container_registry_credentials
Type: dict

This is only required for the upload and promote roles. This is expected to be a Zuul Secret in dictionary form. Each key is the name of a registry, and its value a dictionary with information about that registry.

Example:

container_registry_credentials:
  quay.io:
    username: foo
    password: bar
container_registry_credentials{}.[registry name]
Type: dict

Information about a registry. The key is the registry name, and its value a dict as follows:

container_registry_credentials{}.[registry name]{}.username

The registry username.

container_registry_credentials{}.[registry name]{}.password

The registry password.

container_registry_credentials{}.[registry name]{}.repository

Optional; if supplied this is a regular expression which restricts to what repositories the image may be uploaded. The following example allows projects to upload images to repositories within an organization based on their own names:

repository: "^myorgname/{{ zuul.project.short_name }}.*"
container_registry_credentials{}.[registry name]{}.api_token

Optional; When using the promote roles, the registry API is used to remove temporary tags. if your registry requires a token to talk to the registry API, add it here. This is registry dependent; some allow API access via the username/password, but others require issuing a separate token. For more details see remove-registry-tag. Some examples:

  • docker : API is access via username/password, does not require token.

  • quay.io : A token must be generated from an “application” that a user has allowed to operate on its behalf. See https://docs.quay.io/api/.

container_images
Type: list

A list of images to build. Each item in the list should have:

container_images[].context

The build context; this should be a directory underneath build-container-image.zuul_work_dir.

container_images[].container_filename

The filename of the container file, present in the context folder, used for building the image. Provide this if you are using a non-standard filename for a specific image.

container_images[].registry

The name of the target registry (E.g., quay.io). Used by the upload and promote roles.

container_images[].repository

The name of the target repository in the registry for the image. Supply this even if the image is not going to be uploaded (it will be tagged with this in the local registry). This should include the registry name. E.g., quay.io/example/image.

container_images[].path

Optional: the directory that should be passed to the build command. Useful for building images with a container file in the context directory but a source repository elsewhere.

container_images[].build_args
Type: list

Optional: a list of values to pass to the --build-arg parameter.

container_images[].target

Optional: the target for a multi-stage build.

container_images[].tags
Default: ['latest']
Type: list

A list of tags to be added to the image when promoted.

container_images[].siblings
Default: []
Type: list

A list of sibling projects to be copied into {{zuul_work_dir}}/.zuul-siblings. This can be useful to collect multiple projects to be installed within the same Docker context. A -build-arg called ZUUL_SIBLINGS will be added with each sibling project. Note that projects here must be listed in required-projects.

container_build_extra_env
Type: dict

A dictionary of key value pairs to add to the container build environment. This may be useful to enable buildkit with docker builds for example.

promote_container_image_method
Default: tag

A string value indicating whether or not we upload images to the upstream registry pre merge then promote that upload via a retag (tag) or we upload to a downstream registry and later fetch and promote that to the upstream registry post merge (intermediate-registry).

promote_container_image_method
Default: tag
Type: string

If tag (the default), then this role will update tags created by the upload-container-image role. Set to intermediate-registry to have this role copy an image created and pushed to an intermediate registry by the build-container-role. In that case, the variables below provide the extra information needed to perform the query.

promote_container_image_api

Only required for the intermediate-registry method. The Zuul API endpoint to use. Example: https://zuul.example.org/api/tenant/{{ zuul.tenant }}

promote_container_image_pipeline

Only required for the intermediate-registry method. The pipeline in which the previous build ran.

promote_container_image_job

Only required for the intermediate-registry method. The job of the previous build.

promote_container_image_query
Default: change={{ zuul.change }}&patchset={{ zuul.patchset }}&pipeline={{ promote_container_image_pipeline }}&job_name={{ promote_container_image_job }}

Only required for the intermediate-registry method. The query to use to find the build. Normally the default is used.

promote-docker-image

Promote one or more previously uploaded docker images.

This is one of a collection of roles which are designed to work together to build, upload, and promote docker images in a gating context:

The build-docker-image role is designed to be used in check and gate pipelines and simply builds the images. It can be used to verify that the build functions, or it can be followed by the use of subsequent roles to upload the images to Docker Hub.

The upload-docker-image role uploads the images to Docker Hub. It can be used in one of two modes: by default it will upload with a single tag corresponding to the change ID. In this mode, the role role is designed to be used in a job in a gate pipeline so that the build produced by the gate is staged and can later be promoted to production if the change is successful. The other mode allows for use of this job in a release pipeline to directly upload a release build with the final set of tags.

The promote-docker-image role is designed to be used in a promote pipeline. It requires no nodes and runs very quickly on the Zuul executor. It simply re-tags a previously uploaded image for a change with whatever tags are supplied by build-docker-image.docker_images.tags. It also removes the change ID tag from the repository in Docker Hub, and removes any similar change ID tags more than 24 hours old. This keeps the repository tidy in the case that gated changes fail to merge after uploading their staged images.

They all accept the same input data, principally a list of dictionaries representing the images to build. YAML anchors can be used to supply the same data to all three jobs.

Use the ensure-docker role to install Docker before using this role.

Role Variables

zuul_work_dir
Default: {{ zuul.project.src_dir }}

The project directory. Serves as the base for build-docker-image.docker_images.context.

docker_dockerfile
Default: Dockerfile

The default Dockerfile name to use. Serves as the base for build-docker-image.docker_images.dockerfile. This allows a global overriding of Dockerfile name, for example when building all images from different folders with similarily named dockerfiles.

docker_registry
Default: ''

The container registry the images should be tagged for, by default zuul will push the image to dockerhub.

docker_credentials
Type: dict

This is only required for the upload and promote roles. This is expected to be a Zuul Secret with two keys:

docker_credentials{}.username

The Docker Hub username.

docker_credentials{}.password

The Docker Hub password.

docker_credentials{}.repository

Optional; if supplied this is a regular expression which restricts to what repositories the image may be uploaded. The following example allows projects to upload images to repositories within an organization based on their own names:

repository: "^myorgname/{{ zuul.project.short_name }}.*"
docker_use_buildkit
Default: false
Type: bool

Use BuildKit when creating images.

docker_images
Type: list

A list of images to build. Each item in the list should have:

docker_images[].context

The docker build context; this should be a directory underneath build-docker-image.zuul_work_dir.

docker_images[].dockerfile

The filename of the dockerfile, present in the context folder, used for building the image. Provide this if you are using a non-standard filename for a specific image.

docker_images[].repository

The name of the target repository in dockerhub for the image. Supply this even if the image is not going to be uploaded (it will be tagged with this in the local registry).

docker_images[].path

Optional: the directory that should be passed to docker build. Useful for building images with a Dockerfile in the context directory but a source repository elsewhere.

docker_images[].build_args
Type: list

Optional: a list of values to pass to the docker --build-arg parameter.

docker_images[].target

Optional: the target for a multi-stage build.

docker_images[].tags
Default: ['latest']
Type: list

A list of tags to be added to the image when promoted.

docker_images[].siblings
Default: []
Type: list

A list of sibling projects to be copied into {{zuul_work_dir}}/.zuul-siblings. This can be useful to collect multiple projects to be installed within the same Docker context. A -build-arg called ZUUL_SIBLINGS will be added with each sibling project. Note that projects here must be listed in required-projects.

docker_images[].labels
Default: []
Type: list

A list of labels to attach to the built image, in the form of “key=value”.

docker_images[].arch
Default: []
Type: list

A list of architectures to build on. When enabling this on any image, all of them will be built with docker buildx.

Valid values are linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6.

pull-from-intermediate-registry

Pull artifacts from the intermediate registry

This role will pull any artifacts built for changes ahead of this change which have been placed in an intermediate registry into the buildset registry for this buildset.

Run this in a trusted pre-playbook at the start of a job (which, in the case of multiple dependent jobs in a buildset, should be at the root of the job dependency graph).

This requires the run-buildset-registry role already applied. It also requires an externally managed “intermediate” registry operating for the use of Zuul, and it requires “skopeo” and “socat” to be installed on the Zuul executors.

Role Variables

buildset_registry

Information about the registry, as returned by run-buildset-registry.

buildset_registry.host

The host (IP address) of the registry.

buildset_registry.port

The port on which the registry is listening.

buildset_registry.username

The username used to access the registry via HTTP basic auth.

buildset_registry.password

The password used to access the registry via HTTP basic auth.

buildset_registry.cert

The (self-signed) certificate used by the registry.

intermediate_registry

Information about the registry. This is expected to be provided as a secret.

intermediate_registry.host

The host (IP address) of the registry.

intermediate_registry.port

The port on which the registry is listening.

intermediate_registry.username

The username used to access the registry via HTTP basic auth.

intermediate_registry.password

The password used to access the registry via HTTP basic auth.

push-to-intermediate-registry

Push artifacts to the intermediate registry

This role will push any images built by build-docker-image or build-container-image into an intermediate registry.

Run this in a trusted post-playbook at the end of a job after the image build.

This requires the run-buildset-registry role already applied. It also requires an externally managed “intermediate” registry operating for the use of Zuul, and it requires “skopeo” and “socat” to be installed on the Zuul executors.

Role Variables

buildset_registry

Information about the registry, as returned by run-buildset-registry.

buildset_registry.host

The host (IP address) of the registry.

buildset_registry.port

The port on which the registry is listening.

buildset_registry.username

The username used to access the registry via HTTP basic auth.

buildset_registry.password

The password used to access the registry via HTTP basic auth.

buildset_registry.cert

The (self-signed) certificate used by the registry.

intermediate_registry

Information about the registry. This is expected to be provided as a secret.

intermediate_registry.host

The host (IP address) of the registry.

intermediate_registry.port

The port on which the registry is listening.

intermediate_registry.username

The username used to access the registry via HTTP basic auth.

intermediate_registry.password

The password used to access the registry via HTTP basic auth.

docker_images
Type: list

A list of images built. Each item in the list should have:

docker_images[].repository

The name of the target repository for the image.

docker_images[].tags
Default: ['latest']
Type: list

A list of tags to be added to the image.

container_images
Type: list

A list of images built. Each item in the list should have:

container_images[].repository

The name of the target repository for the image.

container_images[].tags
Default: ['latest']
Type: list

A list of tags to be added to the image.

remove-registry-tag

Remove tags from registry

This role creates a generic interface for removing tags from a container registry. The OCI distribution API (implemented essentially all registries) does specify a tag deletion endpoint, but as at 2023-03 essentially no registries implement it. This means practically we must talk to the per-registry API directly to remove tags. The methods to delete tags are generally similar across registries, but differ slightly in endpoint names, etc.

This role can run in two modes; either removing a single specific tag, or it can run a cleanup process removing all tags that match a given prefix and have not been modified in a given amount of time.

For public registries this role should guess the API from the repository name. If you are running against a private registry, you will need to explicitly specify the API type and URL prefix to communicate to using arguments below.

Role Variables

remove_registry_tag_repository
Type: string

Required. This must be the full repository; e.g. quay.io/organisation/image

container_registry_credentials
Type: dict

Required. This is expected to be a Zuul secret in dictionary form. For convenience this is in the same format as the container_registry_credentials variable used by the other container roles. You must specify the correct variables for the registry you are communicating with:

  • quay.io : Specify an api_key which is issued from an application assigned to an organisation. See https://docs.quay.io/api/

  • docker.io : Username and password

Example:

container_registry_credentials:
  quay.io:
    api_token: 'abcd1234'
  docker.io:
    username: 'username'
    password: 'password'
remove_registry_tag_tag
Type: string

Optional. If set, the specific tag to remove.

remove_registry_tag_regex
Default: '^change_.*$|^{{ zuul.pipeline }}_.*$'
Type: string

Optional. If remove-registry-tag.remove_registry_tag_tag is unset, any tags matching this regex and exceeding the age in remove-registry-tag.remove_registry_tag_age will be removed. The default is tags matching those created by the promote upload roles.

remove_registry_tag_age
Default: 86400
Type: int

Optional. The age, in seconds, a tag that matches remove-registry-tag.remove_registry_tag_regex last-modified timestamp must exceed to be removed.

remove_registry_tag_api_type
Type: string

Optional. By default the role will guess the API type from the repository name. However, if you need to override this choice specify one of:

  • quay

  • docker

remove_registry_tag_api_url
Type: string

Optional. This role will use the default URL for the given registry API. If you need to override this choice, specify this variable.

run-buildset-registry

Runs a container registry for the use of this buildset.

This may be used for a single job running on a single node, or it may be used at the root of a job graph so that multiple jobs running for a single change can share the registry.

Role Variables

buildset_registry_root
Default: {{ ansible_user_dir }}/buildset_registry

Path for the registry volumes.

buildset_registry_port
Default: 5000

The port on which the registry should listen.

container_command
Default: docker

The command to use to run the registry container (E.g., podman).

Return Values

buildset_registry

Information about the registry.

buildset_registry.host

The host (IP address) of the registry.

buildset_registry.port

The port on which the registry is listening.

buildset_registry.username

The username used to access the registry via HTTP basic auth.

buildset_registry.password

The password used to access the registry via HTTP basic auth.

buildset_registry.cert

The (self-signed) certificate used by the registry.

upload-container-image

Upload one or more container images.

This is one of a collection of roles which are designed to work together to build, upload, and promote container images in a gating context:

All roles accept the same input data, principally a list of dictionaries representing the images to build. YAML anchors can be used to supply the same data to all three roles.

Building

The build-container-image role is designed to be used in check and gate pipelines and simply builds the images. It can be used to verify that the build functions, or it can be followed by the use of subsequent roles to upload the images to a registry.

Uploading

The upload-container-image role uploads the images to a registry. It can be used in one of two modes:

  1. Using tags as part of a two-step promote pipeline. This mode is designed to minimize the time the published registry tag is out of sync with the changes Zuul has merged to the underlying code repository.

    In this mode, the role is intended to run in the gate pipeline. Zuul will build and upload the resulting image with a single tag prefixed with the change ID (e.g. change_12345_<tag>). Thus at the completion of the gate job, all the layers of the new container are uploaded, but the <tag> in the remote repository will not have been updated.

    Once the gate queue is successfully finished Zuul will merge the change to the code-repository. At this point, a small window opens where the <tag> is pointing to a container that does not reflect the state of the code-repository. The merge of the change will trigger the promote pipeline which uses a very quick, executor-only job to retag <tag> to change_12345_<tag>. Since this step does not require any nodes or upload any data, it generally takes only a few seconds. The remote container pointed to by <tag> will now reflect the underlying code closing the out-of-sync window.

    When running in this mode uploads are only made if promote_container_image_method is unset or set to tag. Otherwise we skip upload to the registry.

  2. The second mode allows for use of this job in release and tag pipelines to directly upload a release build with the final set of tags.

    In this mode, upload_container_image_promote: false should be set. The role will build and upload the resulting image to the remote repository with the final tags.

    This should be used with tag and release pipelines, where committed code has been tagged for publishing. The tagged commit is “known good” thanks to gating, so the build and upload process is expected to work unconditionally.

    This can be used in a post-commit pipeline, with the caveat that it has a much longer window where published code is out of sync with the published image, as the image must be completely rebuilt and uploaded after code merge in the gate job.

    The alternative promote method can be thought of as a “speculative” upload. There is a possibility the gate job uploads layers and creates a temporary tag, but either the container upload or another co-gating job fails, causing the gate jobs to fail overall. This causes extra uploads, unsued layers and unused tags that require cleaning up. Since changes have merged before the release pipeline starts, the upload will simply not run if the gate jobs fail. This avoids uploading or tagging anything that will not be used. The trade-off is a higher latency between merging code and publishing final tags.

    Transient network failures can cause upload errors in both cases. Although the promote job may fail, leaving the tag incorrectly unmodified, the promote job’s relatively simplicity minimises potential error. The release pipeline does more work, exposing it to a higher chance of failures such as transient network errors etc., also resulting in the repository tag being out-of-date. In both cases developers must pay close attention as failures in these pipelines are often less noticable than code not merging with a gate-job failure.

Promoting

As discussed above, the promote-container-image role is designed to be used in a promote pipeline.

In tag mode, it re-tags a previously uploaded image by copying the temporary change-id based tags made during upload to the final production tags supplied by build-container-image.container_images.tags. It is intended to run very quickly and with no dependencies, so it can run directly on the Zuul executor.

Once this role completes, the temporary upload tags are no longer required. The role removes the change-id tags from the repository in the registry, and removes any similar change-ids tags. This keeps the repository tidy in the case that gated changes fail to merge after uploading their staged images. Remvoing these tags is a registry specific operation. You should double check the api_token requirements for your registry described below. For more details see remove-registry-tag.

In intermediate-registry mode, this role queries Zuul to find the build performed by the build role in the gate. It then copies this image from the intermediate-registry to the final location in the remote registry.

Dependencies

The build and upload roles require a container runtime that should be installed before use; for example by using either the ensure-docker or ensure-podman roles. The promote job assumes skopeo is available on the executor.

Role Variables

zuul_work_dir
Default: {{ zuul.project.src_dir }}

The project directory. Serves as the base for build-container-image.container_images.context.

container_filename

The default container filename name to use. Serves as the base for build-container-image.container_images.container_filename. This allows a global overriding of the container filename name, for example when building all images from different folders with similarily named containerfiles.

If omitted, the default depends on the container command used. Typically, this is Dockerfile for docker and Containerfile (with a fallback on Dockerfile) for podman.

container_command
Default: podman

The command to use when building the image (E.g., docker).

container_registry_credentials
Type: dict

This is only required for the upload and promote roles. This is expected to be a Zuul Secret in dictionary form. Each key is the name of a registry, and its value a dictionary with information about that registry.

Example:

container_registry_credentials:
  quay.io:
    username: foo
    password: bar
container_registry_credentials{}.[registry name]
Type: dict

Information about a registry. The key is the registry name, and its value a dict as follows:

container_registry_credentials{}.[registry name]{}.username

The registry username.

container_registry_credentials{}.[registry name]{}.password

The registry password.

container_registry_credentials{}.[registry name]{}.repository

Optional; if supplied this is a regular expression which restricts to what repositories the image may be uploaded. The following example allows projects to upload images to repositories within an organization based on their own names:

repository: "^myorgname/{{ zuul.project.short_name }}.*"
container_registry_credentials{}.[registry name]{}.api_token

Optional; When using the promote roles, the registry API is used to remove temporary tags. if your registry requires a token to talk to the registry API, add it here. This is registry dependent; some allow API access via the username/password, but others require issuing a separate token. For more details see remove-registry-tag. Some examples:

  • docker : API is access via username/password, does not require token.

  • quay.io : A token must be generated from an “application” that a user has allowed to operate on its behalf. See https://docs.quay.io/api/.

container_images
Type: list

A list of images to build. Each item in the list should have:

container_images[].context

The build context; this should be a directory underneath build-container-image.zuul_work_dir.

container_images[].container_filename

The filename of the container file, present in the context folder, used for building the image. Provide this if you are using a non-standard filename for a specific image.

container_images[].registry

The name of the target registry (E.g., quay.io). Used by the upload and promote roles.

container_images[].repository

The name of the target repository in the registry for the image. Supply this even if the image is not going to be uploaded (it will be tagged with this in the local registry). This should include the registry name. E.g., quay.io/example/image.

container_images[].path

Optional: the directory that should be passed to the build command. Useful for building images with a container file in the context directory but a source repository elsewhere.

container_images[].build_args
Type: list

Optional: a list of values to pass to the --build-arg parameter.

container_images[].target

Optional: the target for a multi-stage build.

container_images[].tags
Default: ['latest']
Type: list

A list of tags to be added to the image when promoted.

container_images[].siblings
Default: []
Type: list

A list of sibling projects to be copied into {{zuul_work_dir}}/.zuul-siblings. This can be useful to collect multiple projects to be installed within the same Docker context. A -build-arg called ZUUL_SIBLINGS will be added with each sibling project. Note that projects here must be listed in required-projects.

container_build_extra_env
Type: dict

A dictionary of key value pairs to add to the container build environment. This may be useful to enable buildkit with docker builds for example.

promote_container_image_method
Default: tag

A string value indicating whether or not we upload images to the upstream registry pre merge then promote that upload via a retag (tag) or we upload to a downstream registry and later fetch and promote that to the upstream registry post merge (intermediate-registry).

upload_container_image_promote
Default: true
Type: bool

If true (the default), then this role will upload the image(s) to the container registry with special tags designed for use by the promote-container-image role. Set to false to use this role to directly upload images with the final tag (e.g., as part of an un-gated release job).

upload-docker-image

Upload one or more docker images.

This is one of a collection of roles which are designed to work together to build, upload, and promote docker images in a gating context:

The build-docker-image role is designed to be used in check and gate pipelines and simply builds the images. It can be used to verify that the build functions, or it can be followed by the use of subsequent roles to upload the images to Docker Hub.

The upload-docker-image role uploads the images to Docker Hub. It can be used in one of two modes: by default it will upload with a single tag corresponding to the change ID. In this mode, the role role is designed to be used in a job in a gate pipeline so that the build produced by the gate is staged and can later be promoted to production if the change is successful. The other mode allows for use of this job in a release pipeline to directly upload a release build with the final set of tags.

The promote-docker-image role is designed to be used in a promote pipeline. It requires no nodes and runs very quickly on the Zuul executor. It simply re-tags a previously uploaded image for a change with whatever tags are supplied by build-docker-image.docker_images.tags. It also removes the change ID tag from the repository in Docker Hub, and removes any similar change ID tags more than 24 hours old. This keeps the repository tidy in the case that gated changes fail to merge after uploading their staged images.

They all accept the same input data, principally a list of dictionaries representing the images to build. YAML anchors can be used to supply the same data to all three jobs.

Use the ensure-docker role to install Docker before using this role.

Role Variables

zuul_work_dir
Default: {{ zuul.project.src_dir }}

The project directory. Serves as the base for build-docker-image.docker_images.context.

docker_dockerfile
Default: Dockerfile

The default Dockerfile name to use. Serves as the base for build-docker-image.docker_images.dockerfile. This allows a global overriding of Dockerfile name, for example when building all images from different folders with similarily named dockerfiles.

docker_registry
Default: ''

The container registry the images should be tagged for, by default zuul will push the image to dockerhub.

docker_credentials
Type: dict

This is only required for the upload and promote roles. This is expected to be a Zuul Secret with two keys:

docker_credentials{}.username

The Docker Hub username.

docker_credentials{}.password

The Docker Hub password.

docker_credentials{}.repository

Optional; if supplied this is a regular expression which restricts to what repositories the image may be uploaded. The following example allows projects to upload images to repositories within an organization based on their own names:

repository: "^myorgname/{{ zuul.project.short_name }}.*"
docker_use_buildkit
Default: false
Type: bool

Use BuildKit when creating images.

docker_images
Type: list

A list of images to build. Each item in the list should have:

docker_images[].context

The docker build context; this should be a directory underneath build-docker-image.zuul_work_dir.

docker_images[].dockerfile

The filename of the dockerfile, present in the context folder, used for building the image. Provide this if you are using a non-standard filename for a specific image.

docker_images[].repository

The name of the target repository in dockerhub for the image. Supply this even if the image is not going to be uploaded (it will be tagged with this in the local registry).

docker_images[].path

Optional: the directory that should be passed to docker build. Useful for building images with a Dockerfile in the context directory but a source repository elsewhere.

docker_images[].build_args
Type: list

Optional: a list of values to pass to the docker --build-arg parameter.

docker_images[].target

Optional: the target for a multi-stage build.

docker_images[].tags
Default: ['latest']
Type: list

A list of tags to be added to the image when promoted.

docker_images[].siblings
Default: []
Type: list

A list of sibling projects to be copied into {{zuul_work_dir}}/.zuul-siblings. This can be useful to collect multiple projects to be installed within the same Docker context. A -build-arg called ZUUL_SIBLINGS will be added with each sibling project. Note that projects here must be listed in required-projects.

docker_images[].labels
Default: []
Type: list

A list of labels to attach to the built image, in the form of “key=value”.

docker_images[].arch
Default: []
Type: list

A list of architectures to build on. When enabling this on any image, all of them will be built with docker buildx.

Valid values are linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6.

upload_docker_image_promote
Default: true
Type: bool

If true (the default), then this role will upload the image(s) to Docker Hub with special tags designed for use by the promote-docker-image role. Set to false to use this role to directly upload images with the final tag (e.g., as part of an un-gated release job).

use-buildset-registry

Adds a buildset registry to the docker configuration.

Use this role on any host which should use the buildset registry.

Role Variables

buildset_registry

Information about the registry, as returned by run-buildset-registry.

buildset_registry.host

The host (IP address) of the registry.

buildset_registry.port

The port on which the registry is listening.

buildset_registry.username

The username used to access the registry via HTTP basic auth.

buildset_registry.password

The password used to access the registry via HTTP basic auth.

buildset_registry.cert

The (self-signed) certificate used by the registry.

buildset_registry_docker_user
Default: {{ ansible_user }}

The system user to configure to use the docker registry. The docker configuration file for this user will be updated. By default, the user Ansible is running as.

buildset_registry_namespaces
Default: [ ('docker.io', 'https://...'), ('quay.io', ...), ('gcr.io', ...)]

The namespaces that the buildset registry supports. Each entry should be a tuple with the first elemnet being the registry host (usually the internet domain name) and the second being the URL the registry is hosted at.

The buildset registry will be consulted first for images in these namespaces. Any others will be fetched only from their upstream sources.

Add any local or third-party registries necessary here.

The default may change in the future as more general-purpose public registries become known.

use-docker-mirror

Configure docker to use mirrors if available.

Role Variables

mirror_fqdn
Default: {{ zuul_site_mirror_fqdn }}

The base host for mirror servers.

docker_mirror

URL to override the generated docker hub mirror url based on ensure-docker.mirror_fqdn.

docker_insecure_registries
Default: undefined

Declare this with a list of insecure registries to define the registries which are allowed to communicate with HTTP only or HTTPS with no valid certificate.