We have continuous integration via [Jenkins](https://jenkins.io/)# What is the Lubuntu CI?
The Lubuntu Continuous Integration system, running at [[ https://cici.lubuntu.me | ci.lubuntu.me ]], is a [Jenkins](https://jenkins.io/) instance which aims to act as a testing ground for packages before they are uploaded to the archive and are generally available for users. This results in four PPAs that are available through the [~lubuntu-ci](https://launchpad.net/~lubuntu-ci) team on Launchpad.
Two branches are available:Use cases include:
- Packagers to hone their abilities in a sandboxed environment.
1. **[Stable](https://launchpad.net/~lubuntu-ci/+archive/ubuntu/stable-ci)** builds the latest upstream release as defined in the packaging- Ensuring packages do not fail to build from source before they are uploaded.
1. **[Unstable](https://launchpad.net/~lubuntu-ci/+archive/ubuntu/unstable-ci)** builds from the tip of upstream git, usually the `master` branch.
## How it works
Builds are triggered on Jenkins using the [CI Tooling](https://phab.lubuntu.me/source/ci-tooling/) and [CI Metadata](https://phab.lubuntu.me/source/ci-metadata/). These are published through the proposed PPAs which are not meant to be used. Then they run through [ppa-britney](https://phab.lubuntu.me/source/ppa-britney/) to essentially do the same [tests that Ubuntu does on proposed packages](https://wiki.ubuntu.com/ProposedMigration) before they are suitable for inclusion in the archive and are published to the release PPAs.
## When it happens
Specific Stable and Unstable builds happen when there are new pushes to their packaging repos.- Running QA tests on packages to ensure they are ready for the archive.
# What kind of packages are built and from which upstream sources?
All build configurations are regenerated and all Stable and Unstable builds are ran daily at 0500 UTCThere are two tracks available for the Lubuntu CI, **stable** and **unstable**. Both are available in separate packaging branches for all of the non-native packages hosted on Launchpad. The `ci/stable` branch is used for stable, and the `ci/unstable` branch is used for unstable.
ppa-britney runs 6 times a day: 0100, 0500, 0900, 1300, 1700, and 2100 UTC.
## How it might be useful
This aims to build most of the applications we ship.
# How does it work?
Every night at midnight Central US time (or when new commits are pushed to a repository) the following happens for each package:
* Keep a VM running with the unstable PPA and you'll essentially have a capture of upstream.1. A merger job is ran for the package. This simply does a git merge from `ubuntu/DEVEL` -> `ci/stable` -> `ci/unstable` and pushes it to Phabricator. If merge conflicts cannot be solved, Tthis is great when filing upstream issues as our packages often lag behindshould cause the merger job to fail.
* If there's a big round of packaging (say,1. Builds are ran for every single release and the two tracks, stable and unstable. This grabs the packaging, grabs the appropriate upstream source, and uploads it to the PPA. Every five minutes the tooling checks to see if the package has been published, and if there are build failures, it should fail the job. Otherwise, the packages are now available.
Packages are built for five architectures: `amd64`, `arm64`, `armhf`, `ppc64el`, and `s390x`. a whole new LXQt release),The `i386` architecture was recently removed because we can soon expect not to have such an architecture in the archive.
# What is the difference between stable and unstable?
There are two key differences between these two tracks:
the process necessarily will reveal any issues that might occur in Britney and it will allow for further testing- Where the orig tarball is derived from.
* Pushing to Phabricator doesn't necessarily trigger an upload to the archive- Where the packaging is pulled from.
For example, let's say we have package `foo` in the archive at version 1.0.2. 1.0.3 was recently released, and we want to package that. However, before we publish it to the archive, we want to test it and ensure it works.
First, we would stage the tentative packaging in `ubuntu/DEVEL`. This allows us to know exactly what will be uploaded. If testing hadn't been done previouslyWhen that is pushed to Phabricator, it can be done at this timethe CI will merge those commits into `ci/stable` and makes for a nice alternative to publishing to a PPAthen merge `ci/stable` into `ci/unstable`.
As long as the changelog entry is bumped, the `ci/stable` branch will pull 1.0.3. This is because the stable jobs use `uscan --download-current-version` to download the orig tarball. However, unless there are other packaging changes, the unstable build should not change at all, because the orig tarball is already using the tip of upstream master. You can then grab the deb produced by the CI in the stable track, test that, and when you're ready, you can push that to the archive.
Now, let's say you need to cherry-pick an upstream patch that will go into 1.0.4, prior to 1.0.4 being released. You would cherry-pick that, push the packaging to `ubuntu/DEVEL`, and while the stable build should pass, the unstable build should fail. This is because quilt will throw an error due to it not being able to apply the patch. You can then push a commit to `ci/unstable` removing that patch, perhaps staging changes that you would like to pull in once 1.0.4 is released, and when 1.0.4 is released, you can simply merge `ci/unstable` into `ubuntu/DEVEL`.
# What's Britney, and what role does it play?
Britney is used in the archive for [[ https://wiki.ubuntu.com/ProposedMigration | proposed migration ]]. In the Ubuntu archive, new uploads (in the typical case of the development release) are directed to `devel-proposed`. When they are there, the packages build, autopkgtests are ran for the package and its reverse-dependencies to ensure tested-for regressions are caught, and installability tests are ran to ensure that it will not break existing packages on the system when installed.
You can think of the Lubuntu CI setup as a miniature version of the archive. When Jenkins (with Lugito in the changelog) does an upload, we have proposed PPAs for both stable and unstable. If any new builds fail and/or any packages are made uninstallable, Britney will not allow the package to migrate.
Note, this is only for regressions, not for all failures, much like the archive. For example, if package foo has a failing `arm64` build, and we have accepted that this build will always be failing, we can manually migrate it, and future builds that have `arm64` failing will migrate. However, if `arm64` starts passing again, and that migrates, builds with `arm64` failing will require manual attention prior to being migrated again. This is the same case for installability issues.
Britney runs every four hours, and is a management job on Jenkins. The four hour cycle starts at midnight US Central time.
# What's left?
- autopkgtest support, either with our own infra or Ubuntu's.
- Proper implementation of hints, like it's done in the archive. This involves changing the default Britney configuration file and creating a hints repository.
- Ensuring that when a merge fails, it causes the job to fail.
- Native package support.