# What is the Lubuntu CI?
The Lubuntu Continuous Integration system, running at [[ https://ci.lubuntu.me | ci.lubuntu.me ]], is a [Jenkins](https://jenkins.io/) instance which aims to act as a testing ground for packages before they are uploaded to the archive and are generally available for users. Use cases include:
- Packagers to hone their abilities in a sandboxed environment.
- Ensuring packages do not fail to build from source before they are uploaded.
- Running QA tests on packages to ensure they are ready for the archive.
# What kind of packages are built and from which upstream sources?
There are two tracks available for the Lubuntu CI, **stable** and **unstable**. Both are available in separate packaging branches for all of the non-native packages hosted on Launchpad. The `ci/stable` branch is used for stable, and the `ci/unstable` branch is used for unstable.
This aims to build most of the applications we ship.
# How does it work?
Every night at midnight Central US time (or when new commits are pushed to a repository) the following happens for each package:
1. A merger job is ran for the package. This simply does a git merge from `ubuntu/DEVEL` -> `ci/stable` -> `ci/unstable` and pushes it to Phabricator. If merge conflicts cannot be solved, this should cause the merger job to fail.
1. Builds are ran for every single release and the two tracks, stable and unstable. This grabs the packaging, grabs the appropriate upstream source, and uploads it to the PPA. Every five minutes the tooling checks to see if the package has been published, and if there are build failures, it should fail the job. Otherwise, the packages are now available.
Packages are built for five architectures: `amd64`, `arm64`, `armhf`, `ppc64el`, and `s390x`. The `i386` architecture was recently removed because we can soon expect not to have such an architecture in the archive.
# What is the difference between stable and unstable?
There are two key differences between these two tracks:
- Where the orig tarball is derived from.
- Where the packaging is pulled from.
For example, let's say we have package `foo` in the archive at version 1.0.2. 1.0.3 was recently released, and we want to package that. However, before we publish it to the archive, we want to test it and ensure it works.
First, we would stage the tentative packaging in `ubuntu/DEVEL`. This allows us to know exactly what will be uploaded. When that is pushed to Phabricator, the CI will merge those commits into `ci/stable` and then merge `ci/stable` into `ci/unstable`.
As long as the changelog entry is bumped, the `ci/stable` branch will pull 1.0.3. This is because the stable jobs use `uscan --download-current-version` to download the orig tarball. However, unless there are other packaging changes, the unstable build should not change at all, because the orig tarball is already using the tip of upstream master. You can then grab the deb produced by the CI in the stable track, test that, and when you're ready, you can push that to the archive.
Now, let's say you need to cherry-pick an upstream patch that will go into 1.0.4, prior to 1.0.4 being released. You would cherry-pick that, push the packaging to `ubuntu/DEVEL`, and while the stable build should pass, the unstable build should fail. This is because quilt will throw an error due to it not being able to apply the patch. You can then push a commit to `ci/unstable` removing that patch, perhaps staging changes that you would like to pull in once 1.0.4 is released, and when 1.0.4 is released, you can simply merge `ci/unstable` into `ubuntu/DEVEL`.
# What's Britney, and what role does it play?
Britney is used in the archive for [[ https://wiki.ubuntu.com/ProposedMigration | proposed migration ]]. In the Ubuntu archive, new uploads (in the typical case of the development release) are directed to `devel-proposed`. When they are there, the packages build, autopkgtests are ran for the package and its reverse-dependencies to ensure tested-for regressions are caught, and installability tests are ran to ensure that it will not break existing packages on the system when installed.
You can think of the Lubuntu CI setup as a miniature version of the archive. When Jenkins (with Lugito in the changelog) does an upload, we have proposed PPAs for both stable and unstable. If any new builds fail and/or any packages are made uninstallable, Britney will not allow the package to migrate.
Note, this is only for regressions, not for all failures, much like the archive. For example, if package foo has a failing `arm64` build, and we have accepted that this build will always be failing, we can manually migrate it, and future builds that have `arm64` failing will migrate. However, if `arm64` starts passing again, and that migrates, builds with `arm64` failing will require manual attention prior to being migrated again. This is the same case for installability issues.
Britney runs every four hours, and is a management job on Jenkins. The four hour cycle starts at midnight US Central time.
# What's left?
A few things are left, and if someone has spare time, it would be great to implement them.
- autopkgtest support, either with our own infra or Ubuntu's.
- Proper implementation of hints, like it's done in the archive. This involves changing the default Britney configuration file and creating a hints repository.
- Ensuring that when a merge fails, it causes the job to fail.
- Native package support.