Continuous Integration
Updated 855 Days AgoPublic
Actions

What is the Lubuntu CI?

The Lubuntu Continuous Integration system, running at ci.lubuntu.me, is a Jenkins instance which aims to act as a testing ground for packages before they are uploaded to the archive and are generally available for users. Use cases include:

Packagers to hone their abilities in a sandboxed environment.
Ensuring packages do not fail to build from source before they are uploaded.
Running QA tests on packages to ensure they are ready for the archive.
Testing Git master for the purposes of creating upstream bug reports.

What kind of packages are built and from which upstream sources?

There are two tracks available for the Lubuntu CI, stable and unstable. Both are available in separate packaging branches for all of the non-native packages hosted on Launchpad. The ci/stable branch is used for stable, and the ci/unstable branch is used for unstable.

This aims to build most of the applications we ship.

How does it work?

Every night at midnight Central US time (or when new commits are pushed to a repository) the following happens for each package:

A merger job is ran for the package. This simply does a git merge from ubuntu/DEVEL -> ci/stable -> ci/unstable and pushes it to Phabricator. If merge conflicts cannot be solved, this should cause the merger job to fail.
Builds are ran for every single release and the two tracks, stable and unstable. This grabs the packaging, grabs the appropriate upstream source, and uploads it to the PPA. Every five minutes the tooling checks to see if the package has been published, and if there are build failures, it should fail the job. Otherwise, the packages are now available.

Packages are built for five architectures: amd64, arm64, armhf, ppc64el, and s390x. The i386 architecture was recently removed because we can soon expect not to have such an architecture in the archive.

PPAs are available on the Lubuntu CI Launchpad team.

What is the difference between stable and unstable?

There are two key differences between these two tracks:

Where the orig tarball is derived from.
Where the packaging is pulled from.

Example

For example, let's say we have package foo in the archive at version 1.0.2. 1.0.3 was recently released, and we want to package that. However, before we publish it to the archive, we want to test it and ensure it works.

First, we would stage the tentative packaging in ubuntu/DEVEL. This allows us to know exactly what will be uploaded. When that is pushed to Phabricator, the CI will merge those commits into ci/stable and then merge ci/stable into ci/unstable.

As long as the changelog entry is bumped, the ci/stable branch will pull 1.0.3. This is because the stable jobs use uscan --download-current-version to download the orig tarball. However, unless there are other packaging changes, the unstable build should not change at all, because the orig tarball is already using the tip of upstream master. You can then grab the deb produced by the CI in the stable track, test that, and when you're ready, you can push that to the archive.

Now, let's say you need to cherry-pick an upstream patch that will go into 1.0.4, prior to 1.0.4 being released. You would cherry-pick that, push the packaging to ubuntu/DEVEL, and while the stable build should pass, the unstable build should fail. This is because quilt will throw an error due to it not being able to apply the patch. You can then push a commit to ci/unstable removing that patch, perhaps staging changes that you would like to pull in once 1.0.4 is released, and when 1.0.4 is released, you can simply merge ci/unstable into ubuntu/DEVEL.

What's Britney, and what role does it play?

Britney is used in the archive for proposed migration. In the Ubuntu archive, new uploads (in the typical case of the development release) are directed to devel-proposed. When they are there, the packages build, autopkgtests are ran for the package and its reverse-dependencies to ensure tested-for regressions are caught, and installability tests are ran to ensure that it will not break existing packages on the system when installed.

You can think of the Lubuntu CI setup as a miniature version of the archive. When Jenkins (with Lugito in the changelog) does an upload, we have proposed PPAs for both stable and unstable. If any new builds fail and/or any packages are made uninstallable, Britney will not allow the package to migrate.

Note, this is only for regressions, not for all failures, much like the archive. For example, if package foo has a failing arm64 build, and we have accepted that this build will always be failing, we can manually migrate it, and future builds that have arm64 failing will migrate. However, if arm64 starts passing again, and that migrates, builds with arm64 failing will require manual attention prior to being migrated again. This is the same case for installability issues.

Britney runs every four hours, and is a management job on Jenkins. The four hour cycle starts at midnight US Central time.

What's left?

A few things are left, and if someone has spare time, it would be great to implement them.

autopkgtest support, either with our own infra or Ubuntu's.
Proper implementation of hints, like it's done in the archive. This involves changing the default Britney configuration file and creating a hints repository.
Ensuring that when a merge fails, it causes the job to fail.
Native package support.