This is an overview of our continuous integration and continuous delivery process
Following the special project during winter 2020 we have decided to add up to what had been done to the development processes that were put in place. While reviewing the development process we have noticed that it would be required to normalize our development practices which has been done in winter 2020. While keeping our updated process in mind we are introducing CI/CD along with our new containerized node to ease up the development effort and add up some automation in testing our modules.
Since we want this process to be clear and to keep track of any changes made to it we have created this page.
The following topics will help you understand the main idea behind those processes and reveal what has been decided and put in place to ease up development effort.
For network access concerns during a test or at the competition we will have to be able to have an alternative workflow.
This process has a strong bond with the Gitflow and Development Process
Continuous integration has been added to our development process since it will have a great impact on developing a test for our module to ensure that we are not having regression upon new code being committed which will help in increasing the quality of our code along with all the peer review procedure that has been put in place by using the Pull Request from GitHub.
Since we are now deploying our module on docker images using continuous integration was a must. Using CI will help us maintain good quality in our code by developing a unit test. It will also ease up our module packaging process. On each build from a feature, develop or master branch a build will be automatically triggered to run our unit test, build our images, publish them and notify us of the process state using slack.
As shown in the diagram above, one of the key elements of the new pipeline is delivery. The team needed a way to reduce coupling in the code and to facilitate the deployment of every single node.
By using Docker tags on every module repository, it allows for the release and deployment of versions of each module. It also enables the team to quickly unify every module in a single cluster of containers.
CI/CD pipelines have been review since we passed from our previous CI/CD tool (Travis-CI) who was used to handle the automation of all our CI/CD pipelines. We are now using Github Action as our CI/CD pipeline tools.
CI/CD pipelines are referred to as Workflows in GitHub actions. Here is a link to github-actions where you will be able to find out additional information on this topic.
If you are creating a new ROS module using ros-package-repo-template, workflow templates are already present in there and will be generated automatically using the
get_started.shscript. If you want to add a workflow to an existing module please take note that you can use those default workflow but additional rework will be needed to fit it to your use case if you are not developing a ROS module.
We will use the proc_mapping node as an example to explain how the repository layout has been generated and how those it get along with the
workflows
Github actions workflows are usually located at the following location in the template or existing ROS modules:
.
├── .github
│ ├── pull_request_template.md
│ └── workflows
│ ├── docker-image-perception-develop.yml
│ ├── docker-image-perception-feature.yml
│ └── docker-image-perception-master.yml
The workflows layout has been build in a way that we have split our workflows according to our gitflow procedure following the master, develop, feature/* conventions that have been established in our gitflow-and-development-workflow page. Please refer to this page for additional gitflow information.
Notice:
master,develop,feature/*workflows have different actions in them and they will be described in detail in the next sections
In this section an overview of a base GitHub action workflow will be given, we will discuss specific branch workflow in the next sections.
name: Docker Image CI - Feature Branch
jobs:
build-ros-perception-x86-64:
name: "Build ROS perception X86/64"
runs-on: ubuntu-latest
env:
BASE_IMAGE: "docker.pkg.github.com/sonia-auv/sonia_common/sonia_common:x86-perception-latest"
ARCH: x86
TARGET_TYPE: perception
TARGET_VERSION: feature
IMAGE_NAME: proc_mapping
GITHUB_REMOTE_URL: docker.pkg.github.com/${{ github.repository }}
steps:
- uses: actions/checkout@v2
- name: Login to Github Package Registry
run: |
echo "${{ secrets.GITHUB_TOKEN }}" | docker login docker.pkg.github.com -u ${{ github.actor }} --password-stdin
notify-success:
name: "Notify Slack - Success"
runs-on: ubuntu-latest
needs:
[
build-ros-perception-x86-64]
]
if: success()
steps:
- name: Notify Slack Success
env:
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
uses: voxmedia/github-action-slack-notify-build@v1
with:
channel: github-ci-notifications
status: SUCCESS
color: good
notify-fail:
name: "Notify Slack - Failure"
runs-on: ubuntu-latest
needs:
[
build-ros-perception-x86-64,
]
if: failure()
steps:
- name: Notify Slack Fail
env:
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
uses: voxmedia/github-action-slack-notify-build@v1
with:
channel: github-ci-notifications
status: FAILED
color: danger
Here is a list of tag that you could encounter while working in a specific workflow please take not that additional tags exist :
name: The name of the current workflow (This name will be displayed into the GitHub action section of a repository when a workflow is launchedjobs: Under this tag, we should have the definition of every step of the workflowbuild-ros-perception-x86-64, notify-success, notify-fail: Are grouping of tasks inside a job, this means that when this workflow is launched the all tree jobs should be executed (based on their dependencies on each other)Here is a list of specific tags for a given job (e.g. build-ros-perception-x86-64):
name: The name of the jobruns-on: The worker OS we run the job onto (for additional information see GitHub actions documentation)env: Environment variables that will be available through a given jobsteps: Every steps a job must accomplish (basic actions should have uses:actions/checkout@v2 by default, name: the name of the action, run: the command that will be runneeds: This tag will make sure a task gets executed only if the specified task has been completedìf: Let you defined conditional execution based on the task defined in the needs sectionDeclared environment variable can be used to substitute value in the
runcommands or add arguments to adocker buildcommand for example
Please refer to this one for base information has the only differences will be overviewed for develop and master workflow will be viewed. Since they are pretty much the same, with only minor differences we won't review them in depth.
name: Docker Image CI - Feature Branch
on:
push:
branches: [feature/**]
jobs:
build-ros-perception-x86-64:
name: "Build ROS perception X86/64"
runs-on: ubuntu-latest
env:
BASE_IMAGE: "docker.pkg.github.com/sonia-auv/sonia_common/sonia_common:x86-perception-latest"
ARCH: x86
TARGET_TYPE: perception
TARGET_VERSION: feature
IMAGE_NAME: proc_mapping
GITHUB_REMOTE_URL: docker.pkg.github.com/${{ github.repository }}
steps:
- uses: actions/checkout@v2
- name: Login to Github Package Registry
run: |
echo "${{ secrets.GITHUB_TOKEN }}" | docker login docker.pkg.github.com -u ${{ github.actor }} --password-stdin
- name: Build the docker image (perception based)
run: |
docker build . --tag build-feature-${GITHUB_REF##*/}-${GITHUB_RUN_NUMBER} --build-arg BUILD_DATE=$(date '+%Y-%m-%d_%H:%M:%S') --build-arg VERSION=${GITHUB_REF##*/}-$(date ' +%Y-%m-%d_%H:%M:%S') --build-arg BASE_IMAGE=${BASE_IMAGE}
- name: Create Docker Image Tag
run: |
docker tag build-feature-${GITHUB_REF##*/}-${GITHUB_RUN_NUMBER} ${GITHUB_REMOTE_URL}/${IMAGE_NAME}:${ARCH}-${TARGET_TYPE}-${TARGET_VERSION}-${GITHUB_REF##*/}
- name: Push Image to Github Packages Registry
run: |
docker push ${GITHUB_REMOTE_URL}/${IMAGE_NAME}:${ARCH}-${TARGET_TYPE}-${TARGET_VERSION}-${GITHUB_REF##*/}
build-ros-perception-arm64:
name: "Build ROS perception AMR64"
runs-on: ubuntu-latest
env:
BASE_IMAGE: "docker.pkg.github.com/sonia-auv/sonia_common/sonia_common:arm64-perception-latest"
ARCH: arm64
TARGET_TYPE: perception
TARGET_VERSION: feature
IMAGE_NAME: proc_mapping
GITHUB_REMOTE_URL: docker.pkg.github.com/${{ github.repository }}
steps:
- uses: actions/checkout@v2
- name: Login to Github Package Registry
run: |
echo "${{ secrets.GITHUB_TOKEN }}" | docker login docker.pkg.github.com -u ${{ github.actor }} --password-stdin
- name: Enable Docker Daemon Experimental Features
run: |
sudo rm /etc/docker/daemon.json
echo '{"experimental": true , "cgroup-parent": "/actions_job" }' | sudo tee -a /etc/docker/daemon.json
sudo service docker restart
docker version
- name: Install QEMU to be able to compile on X86 into ARM64
run: |
sudo apt-get update
sudo apt-get install qemu binfmt-support qemu-user-static
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
- name: Build the docker image (perception based)
run: |
docker build . --tag build-feature-${GITHUB_REF##*/}-${GITHUB_RUN_NUMBER} --build-arg BUILD_DATE=$(date '+%Y-%m-%d_%H:%M:%S') --build-arg VERSION=${GITHUB_REF##*/}-$(date ' +%Y-%m-%d_%H:%M:%S') --build-arg BASE_IMAGE=${BASE_IMAGE}
- name: Create Docker Image Tag
run: |
docker tag build-feature-${GITHUB_REF##*/}-${GITHUB_RUN_NUMBER} ${GITHUB_REMOTE_URL}/${IMAGE_NAME}:${ARCH}-${TARGET_TYPE}-${TARGET_VERSION}-${GITHUB_REF##*/}
- name: Push Image to Github Packages Registry
run: |
docker push ${GITHUB_REMOTE_URL}/${IMAGE_NAME}:${ARCH}-${TARGET_TYPE}-${TARGET_VERSION}-${GITHUB_REF##*/}
notify-success:
name: "Notify Slack - Success"
runs-on: ubuntu-latest
needs:
[
build-ros-perception-x86-64,
build-ros-perception-arm64,
]
if: success()
steps:
- name: Notify Slack Success
env:
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
uses: voxmedia/github-action-slack-notify-build@v1
with:
channel: github-ci-notifications
status: SUCCESS
color: good
notify-fail:
name: "Notify Slack - Failure"
runs-on: ubuntu-latest
needs:
[
build-ros-perception-x86-64,
build-ros-perception-arm64,
]
if: failure()
steps:
- name: Notify Slack Fail
env:
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
uses: voxmedia/github-action-slack-notify-build@v1
with:
channel: github-ci-notifications
status: FAILED
color: danger
First of all, we will overview the env section of the workflow. Please notice that specific environment variables are defined into these steps since they are required to build a proper docker image. This way we can handle multiple source image, architecture, ROS specific image dependencies.
Let's go over these variables:
BASE_IMAGE: In this variable, you will find the source image to base our build on (Docker ARG BASE_IMAGE in our dockerfile)TARGET_TYPE: Refers to the architecture the docker image must run on. Currently we support (x86_64 listed as x86 and arm64)TARGET_VERSION : Target version refers to ROS base image type (robot, perception, ...). We currently use perception for all our imagesIMAGE_NAME: Refers to the current custom SONIA's ROS module name e.g.proc_mappingGITHUB_REMOTE_URL: Refers to GitHub docker package base URLIn the
runarguments additional arguments are provided duringdocker buildcommands please refer to this page ros-node-structure for additional information on their meaning
As you can see in this workflow we have defined 4 jobs, build-ros-perception-x86-64, build-ros-perception-arm64, notify-success, and notify-fail. Both build-ros-perception-x86-64, build-ros-perception-arm64 are the jobs that are building a docker image of the current ROS modules.
Please notice that build-ros-perception-arm64 has additional steps to enable experimental docker daemon, docker client feature, and install qemu emulator since at the time of writing GitHub does not provide any ARM64 workers.
Since we need to add enable specific docker daemon functionalities to be able to pull arm64 images and install qemu to be able to emulate compilation on x86_64 into arm64 this way we can build a docker image that could be used on the Nvidia Jetson platform and also on a computer that have qemu install i.e run arm64 containers on an x86_64 computer using VSCode remote container extension.
Building
arm64container onx86_64is longer since instruction need to be transformed.
Following a recent update by docker, the docker deamon experimental features job has started to fail in the docker image build jobs.
It has come to our attention that previous workflows that worked well are now failing when using the Docker daemon experimental job. Since the pipeline can still run successfully without it, we have temporarily resolved the problem by disabling lines 50 through 55. For your information, the following error message was produced by the failed job:
Run sudo rm /etc/docker/daemon.json
{"experimental": false , "cgroup-parent": "/actions_job", "exec-root": "/path/to/docker/run", "storage-driver": "overlay", "graph": "/path/to/docker/lib" }
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xeu docker.service" for details.
Commit access have been granted to the repository administrators in this branch this way the will be able to increment version into
package.xmlfile before doing a PR to master
Other than that tagging name of the images has been switch to develop. The rest of the workflow definition should be that same as develop
Repository
administratormust increment version intopackage.xmlfile before doing a PR to master, otherwise, workflow won't be completed successfully
Additional jobs have been added to the workflow to create an official release with a git tag and a GitHub release. This way it's easier to go back to a previous version if needed.
Here are the additional task that was added to the workflow
create-release:
name: "Create Github Release"
runs-on: ubuntu-latest
needs:
[
build-ros-perception-x86-64,
build-ros-perception-arm64,
]
steps:
- uses: actions/checkout@v2
- name: Set Target version
run: echo '::set-env name=TARGET_VERSION::'$(sed -n -e 's/.*<version>\(.*\)<\/version>.*/\1/p' package.xml)
- name: Create Git Tag
run: |
git tag ${TARGET_VERSION}
git push origin --tags
- name: Create Github Release
uses: actions/create-release@v1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
tag_name: ${{ env.TARGET_VERSION }}
release_name: Release ${{ env.TARGET_VERSION }}
body: |
A new release for this package has been created
draft: false
prerelease: false
This additional task will fetch the current <version> tag content from package.xml and will use it to create an official GitHub release and a git tag. If the value in the <version> tag has already been used to create a previous release note that the workflow will fail. This is why a repository administrator must update this field before doing their PR from develop to master