Continuous Integration Architecture
In Ceph, we rely on multiple CI pipelines in our development. Most of these pipelines are centered around Jenkins. And their configurations are generated using Jenkins Job Builder.
Let's take the
make check performed by Jenkins as an example.
ceph-pull-requests is a jenkins job which gets triggered by a GitHub pull
request or a trigger phrase like:
jenkins test make check
There are multiple parties involved in this jenkins job:
- Sepia Lab
Sepia Lab is a test lab used by the Ceph project. This lab offers the storage and computing resources required by our CI infra.
- Jenkins agents
are a set of machines which perform the CI jobs. In this case, they
pull the git repo from GitHub and
rebase the pull request against the latest master
set necessary environment variables
is a server offering RESTful API allowing the clients to store and retrieve binary packages. It also creates the repo for uploaded packages automatically. Once a certain repo is created on chacra, the configured shaman server is updated as well, then we can query shaman for the corresponding repo address. Chacra not only hosts Ceph packages, it also hosts quite a few other packages like various build dependencies.
is a server offering RESTful API allowing the clients to query the information of repos hosted by chacra nodes. Shaman is also known for its Web UI. But please note, shaman does not build the packages, it just offers information on the builds.
As the following shows, chacra manages multiple projects whose metadata are stored in a database. These metadata are exposed via Shaman as a web service. chacractl is a utility to interact with the chacra service.
Just like lots of other software projects, Ceph has both build-time and run-time dependencies. Most of time, we are inclined to use the packages prebuilt by the distro. But there are cases where
the necessary dependencies are either missing in the distro, or
their versions are too old, or
they are packaged without some important feature enabled.
we want to ensure that the version of a certain runtime dependency is identical to the one we tested in our lab.
No matter what the reason is, we either need to build them from source, or
to package them as binary packages instead of using the ones shipped by the
distro. Quite a few build-time dependencies are included as git submodules,
but in order to avoid rebuilding these dependencies repeatedly, we pre-built
some of them and uploaded them to our own repos. So, when performing
make check, the building hosts in our CI just pull them from our internal
repos hosting these packages instead of building them.
So far, following packages are prebuilt for ubuntu focal, and then uploaded to chacra:
packages boost. The packages' names are changed from
ceph-libboost-*, and they are instead installed into
/opt/ceph, so they don't interfere with the official
libboostpackages shipped by distro. Its build scripts are hosted at https://github.com/tchaikov/ceph-boost.
tar xjf boost_1_76_0.tar.bz2 git clone https://github.com/ceph/ceph-boost cp -ra ceph-boost/debian boost_1_76_0/ export DEB_BUILD_OPTIONS='parallel=6 nodoc' dpkg-buildpackage -us -uc -b
packages libzbd . The upstream libzbd includes debian packaging already.
packages pmdk . Please note,
ndctlis one of the build dependencies of pmdk, for an updated debian packaging, please see https://github.com/ceph/ceph-ndctl .
please ensure that the package version and the release number of the
packaging are properly updated when updating/upgrading the packaging,
otherwise it would be difficult to tell which version of the package
is installed. We check the package version before trying to upgrade
But in addition to these libraries,
ceph-mgr-dashboard's frontend uses lots of
mention the trouble of testing different combination of versions of these
Also, because our downstream might not want to use the prepackaged binaries when redistributing the precompiled Ceph packages, we also need to include these libraries in our dist tarball. They are
make-dist is a script used by our CI pipeline to create dist tarball so the
tarball can be used to build the Ceph packages in a clean room environment. When
we need to upgrade these third party libraries, we should
update the CMake script
rebuild the prebuilt packages and
update this script to reflect the change.
To ensure that prebuilt packages are available by the jenkins agents, we need to
upload them to either
apt-mirror.front.sepia.ceph.com or chacra. To upload
packages to the former would require the help our our lab administrator, so if we
want to maintain the package repositories on regular basis, a better choice would be
to manage them using chacractl. chacra represents packages repositories using
a resource hierarchy, like:
in general, it is used for denoting a set of related packages. For instance,
branch of project. This mirrors the concept of a Git repo.
a unique id of a given version of a set packages. This id is used to reference the set packages under the
<project>/<branch>. It is a good practice to version the packaging recipes, like the
debiandirectory for building deb packages and the
specfor building rpm packages, and use the sha1 of the packaging receipe for the
ref. But you could also use a random string for
ref, like the tag name of the built source tree.
the distro name for which the packages are built. Currently, following distros are supported:
the version of the distro. For instance, if a package is built on ubuntu focal, the
the architecture of the packages. It could be:
So, for example, we can upload the prebuilt boost packages to chacra like
ls *.deb | chacractl binary create \ libboost/master/099c0fd56b4a54457e288a2eff8fffdc0d416f7a/ubuntu/focal/amd64/flavors/default
We also need to update
install-deps.sh to point the built script to the new
repo. Please refer to the script,
for more details.