Page MenuHomeLubuntu

Packaging Tutorial
Updated 34 Days AgoPublic

Introduction

This is a tutorial which hopes to make it easy for new contributors to figure out how to fix a bug in Lubuntu. Note that this is specific to repositories that Lubuntu hosts on Phabricator, i.e. those components that differentiate Lubuntu from other Ubuntu flavors— most notably, LXQt. Everything else is handled upstream in Ubuntu. The process there will be similar.

Packaging is a vast and complex subject, especially given the many upstreams (Ubuntu, Debian, the developer themselves) and the changing processes/requirements among them. We have our own guide that proves some links to further resources.

Long story short

Anything in caps represents a field you'll need to fill out based on your particular situation.

## WARNING: this is the more high-level overview. 
## If you're new, start with the requirements below.
##
## Also beware the exceptions below.
##
## make a place to contain your work
mkdir WORKDIR
cd WORKDIR
## get the packaging repository from Phabricator
git clone ssh://git@phab.lubuntu.me:2222/source/PACKAGE.git
cd PACKAGE
## make sure you're on the development branch
git checkout ubuntu/CODENAME
## get the upstream source code
uscan --download-current-version
tar -x --strip-components=1 -f ../PACKAGE_*.orig.tar.xz
## make a new patch
mkdir debian/patches # just in case it doesn't exist
quilt push -a  # just in case there are patches, we apply them all
quilt new NAME.patch
# this is where you actually make the changes you want
quilt edit PATH/TO/FILE 
quilt refresh
quilt header --dep3 -e
## update the changelog
dch # -i or -a, depending (see below)
dch -r --distribution CODENAME
## test build the package
sudo apt update && sudo apt build-dep PACKAGE
debuild -d --no-sign
## get rid of source
rm -rf !(debian) .pc/ # requires bash and `shopt -s extglob`
## push your changes
git add -A
arc diff
## don't forget to add @tsimonq2, @wxl as reviewers

Digging deeper

Requirements

  1. Software
    1. apt install ssh git php-curl php-cli devscripts tar quilt
    2. Follow "Installing Arcanist" here.
  2. Configuration
    1. Uncomment the deb-src lines in /etc/apt/sources.list so apt build-dep works correctly.
    2. So you get credit where credit's due:
      1. For git:
        • git config --global user.name "your full name"
        • git config --global user.email "your email address"
      2. For Debian tools, add the following to $HOME/.bashrc:
        • export DEBFULLNAME="your full name"
        • export DEBMAIL="your email address"
        • export EMAIL="your email address"
    3. Set up ssh:
      • Easy way
        1. ssh-keygen
        2. Accept the default name ($HOME/.ssh/id_rsa)
        3. Enter a password ≥ 5 characters
      • Hard way
        1. ssh-keygen -C some-identifying-info -t ed25519 # more secure
        2. Select a unique name e.g. $HOME/.ssh/phab # allows you to have more than one key
        3. Pick an even better password!
        4. Edit $HOME/.ssh/config to include
          • Host phab.lubuntu.me
          • IdentityFile ~/.ssh/phab
    4. Add ssh key to Phab:
      1. Go to Settings
      2. Click on "SSH Public Keys" under "Authentication" on the left side
      3. Click SSH Key ActionsUpload Public Key
      4. Give your key a name (anything works) and then copy and paste the public key from the step above
    5. $HOME/.quiltrc from packaging guide
    6. Final Arcanist tweaks:
      1. arc set-config phabricator.uri "https://phab.lubuntu.me/"
      2. arc install-certificate

Extras

  1. ssh agent with keychain TODO
  2. gpg setup
    1. Get the software: sudo apt install gpg
    2. Make a new key: gpg --full-generate-key
      1. Select an RSA and RSA pair
      2. Select 4096 bits for key size
      3. Select expiration date, or none (note you can always move the date out farther)
      4. Verify and accept
      5. Enter your real name
      6. Enter your email address
      7. A comment is not necessary
      8. Give it a good password
      9. It takes a while to gain enough entropy to finish the creation. Open a new terminal and find / 2>/dev/null and that should help it along
  3. bzr setup TODO

Real world example

Background

We were having some trouble setting the default GTK theme only to discover that upstream had created a patch that could help us out. We only package stable released versions, so we couldn't just repackage it with the new master in git. Instead, we wanted to patch our released version with this change.

Get the upstream changes

So first we needed to pull down the upstream repository. Clicking on the repo got us to the main page:


At that point, we clicked the Clone or download button and copied the clone link. Then we opened up a terminal and cloned the repo:

git clone https://github.com/lxqt/lxqt-config.git

Now we needed to make a patch. So we ran git log, which opens in a pager, like less. You should be able to use this to search by typing / followed by some search terms, and then hitting ENTER. So we did a search for "Set the default GTK theme" and found it at commit 4c3ad403dc14dde4fe41e56cf3272ac11e30346f. To make our patch we remember that git has a shortcut for "previous commit." Using that knowledge we did the following, creating a reasonably-named patch, and putting it in $HOME to keep it out of the way of everything else:

git diff 4c3ad403dc14dde4fe41e56cf3272ac11e30346f 4c3ad403dc14dde4fe41e56cf3272ac11e30346f^ > ~/set-default-gtk-theme.patch

Apply the patch

So now we needed to get our repo. We could search for "lxqt-config" in the search box here and we will eventually find rLXQTCONFIGPACKAGING. Once there, we can clone our repo, by clicking on the Clone button. With that now we can go back to our terminal and clone the repo:

git clone ssh://git@phab.lubuntu.me:2222/source/lxqt-config.git

Then we checked out the development branch:

git checkout ubuntu/cosmic

Then we grabbed the source:

uscan --download-current-version
tar -x --strip-components=1 -f ../lxqt-config_0.13.0.orig.tar.xz

Seeing that we already had patches in debian/patches we proceeded to apply them:

quilt push

And then we imported our patch:

quilt import ~/set-default-gtk-theme.patch

After import, we performed a required quilt push. This is the step that actually applied the patch to the source. Sometimes these don't apply cleanly and some editing is required, but in this case, we had no problems.

Just for safety's safe, we did a quilt refresh and then used quilt header --dep3 -e to edit the header:

Description: Set default GTK theme if rc file doesn't exists.
Author: P.L. Lucas <selairi@gmail.com>
Applied-Upstream: https://github.com/lxqt/lxqt-config/commit/4c3ad403dc14dde4fe41e56cf3272ac11e30346f

Finish it up

We added a new changelog entry with dch -i, incrementing the version number (see below about versioning):

lxqt-config (0.13.0-0ubuntu5) cosmic; urgency=medium

  * Set default GTK theme if rc file doesn't exists. 

 -- Hans P. Möller <hmoller@gmail.com>  Thu, 04 Oct 2018 14:44:03 -0300

Then we built it to ensure there were no issues:

sudo apt update && sudo apt build-dep lxqt-config
debuild -d --no-sign

Next, we got rid of the source and remnants of quilt so the repo was back into a normal state.

rm -rf !(debian) .pc/

And finally, we put it up for review:

git add -A
arc diff

We gave it the commit message:

Set default GTK theme if rc file doesn't exists.

And filled out the template for the differential revision review:

Summary: Set default GTK theme if rc file doesn't exists.

Test Plan: See if default GTK theme is according to 'gsettings get org.gnome.desktop.interface gtk-theme'

Reviewers: @tsimonq2, @wxl

Subscribers:

And that gave us the URL of the differential revision:

https://phab.lubuntu.me/D27

And that point, it was reviewed and changes were requested. We made those changes, using git add and git rm on files where appropriate, and then arc diffing again.

Finally, our revision was accepted and finally landed into our repositories, which in turn synced with the Ubuntu repos.


Exceptions

The above assumes a normal package. If it's native (i.e. doesn't use patches), or if the upstream source isn't well tracked, or if the package isn't already in the Ubuntu repos, etc., things will break.

Packaging-only changes

There's no need to use quilt if your change only affects the packaging and not the source code itself. For example, if you're going to change the debian/rules file, you can just make the change and update the changelog. In that specific example, though, since you are essentially changing the build instructions, you'll still want to do a test build.

Native

Check out rDEFAULTSETTINGS and you'll find that in addition to the normal debian folder, there's also other files. Most notably there is a src folder which contains the code. This should be a clear sign that something is different, as Debian packages are nothing more than metadata that is applied to upstream source code. In this case, we do not use uscan to grab the source, nor quilt to manage patches, but just apply the changes to the code in the src folder and then proceed as normal.

Not a package

A changelog entry is a requirement of every Debian package, but sometimes we're not working on packages. rSEED is a great example of this. No changelog to be found. No debian folder, even. This is because this isn't used to build a package that people install. It's used to build Lubuntu itself! You'll notice if you click Clone that it's not on Phab, but on Launchpad. This will mean arc will be useless both for submitting commits, but also for merging them.

Can't find source

TODO

Not in Phab

Or how to do a Merge Proposal on Launchpad.
TODO

Update an existing Differential revision

or arc patching a fresh git clone to in turn arc diff
TODO

And another thing

A note about version numbers

  • Version number is in the format upstream_version-debian_patch_numberubuntuubuntu_patch_number. You will generally be incrementing ubuntu_patch_number.
    • …unless it's a native package, in which case, there is only a simple version number.
  • Before deciding on the version number to use, check a few things:
      1. git pull or just check the repo to see if there have been additional changes while you've been working on your local copy. If there have been changes, there's likely an updated debian/changelog entry that you'll want to append (dch -a).
      2. Check to see if your repo contains any pending changes. This should be shown by browsing in the repo to the file(s) in question or you can just look at all pending revisions. If there are any, you might want to check in on the status, as you can potentially get your changes rolled in.
    1. Check Launchpad for what's in the archives. The best way to do this is to use !upkg your_source_package in DuckDuckGo.
      1. If the final entry in debian/changelog is equal to the "release" version, increment (dch -i).
      2. If the final entry in debian/changelog is greater than the "release" version, append (dch -a).
      3. If there's something in "proposed," you'll probably want to increment, but check in and see if your change can't be rolled in.

Where is my package?

If you're not finding a package in the archives, it may be because it's stuck in the queue, which can be found at https://launchpad.net/ubuntu/CODENAME/+queue. Here are definitions for the various statuses:

  • New: brand new packages that have never been in the repositories waiting on review.
  • Unapproved: new changes to old packages waiting on review.
  • Rejected: these were for some reason or another rejected (sometimes it's because a new upload supersedes the first one).
  • Accepted: the package is no longer stuck in the queue, but it is not published yet.
  • Done: the package is published.

Note that different teams review different queues, and while there are some team members which overlap, different teams are responsible for different reviews.

New queue reviews are the responsibility of the Ubuntu Package Archive Administrators Team, or Archive Admins for short. You can find the member list here.

Unapproved queue reviews for the development release is done by the Ubuntu Release Team. They also handle migration from devel-proposed to devel[-release], and are the go-to contact for dealing with migration. You can find the member list here.

Unapproved queue reviews for stable releases are done by the Ubuntu Stable Release Updates Team, or SRU Team for short. They are responsible for landing updates as laid out in the documentation. You can find the member list here.

You can find the majority of the people in these teams in #ubuntu-release on freenode IRC, and you are thus encouraged to ask there.

The easiest way to the package is to go to https://duckduckgo.com/?q=!upkg+SOURCE-PACKAGE-NAME.

Package hunting: how to get from upload to distribution

The first step in getting a package uploaded is finding who can upload to the package. This requires an understanding of the archive sections of Ubuntu and ways of getting upload access.

Here are the sections of Ubuntu and what they mean:

  • Main: These packages are a core part of Ubuntu, either as part of the foundation or the GNOME desktop as provided in main Ubuntu. Canonical Ltd. as a company commits to provide support for all of these packages for the full length of the support cycle, including security updates and review from the Ubuntu Security Team. Be careful when touching these packages, because you can break more than just Lubuntu when publishing updates to these.
  • Restricted: These are proprietary drivers such as the NVIDIA drivers which are non-free but can be distributed. It is very rare that you should have to touch any of these packages, and these usually come directly from the distributor.
  • Universe: All free software that is not in Main. The majority of the software here flows down from Debian with little to no changes. All flavors have packages in this section, and it is by far the most populated section in Ubuntu.
  • Multiverse: Anything which could be in Universe but has non-free or "grey area" software which can be distributed but not always easily modified. An example of a package in here is the Microsoft fonts, which need to be downloaded from Microsoft and a license needs to be accepted prior to installing them.

The two main uploading teams in Ubuntu are Ubuntu Core Developers and Ubuntu Masters of the Universe (or MOTU). The latter is a subteam of the former. Ubuntu Core Developers have upload access to many code repositories that are core parts of Ubuntu, and have upload access to all four sections of Ubuntu. However, MOTUs are limited in that they only have upload access to Universe and Multiverse. You can additionally request upload access, or Per Package Upload (PPU) permissions to specific packages, but this is done less frequently than joining one of the above teams.

For Lubuntu packages, you need to find someone with MOTU permissions or higher. When in doubt, it could be helpful to use the ubuntu-upload-permission tool from the ubuntu-dev-tools package to see who can upload to a given package.

If your changes are committed in Git, the uploader simply runs debuild -S -d [-sa] to get a collection of files, much like the PPA instructions below. Once they are satisfied with the upload, they run dput ubuntu ../PACKAGE_*source.changes which uploads the package to Ubuntu. At this point, one of two things could happen.

If the upload is to a stable release, it appears in the Unapproved queue for the release you are uploading to. The bug report which should have been mentioned in the changelog will then get either rejection or acceptance feedback from the SRU Team member processing the package. If the package is accepted, it gets accepted into the CODENAME-proposed pocket for testing. The package then needs to be verified, and if it works, after seven days (unless an exception is made), the package is released into CODENAME-updates for general consumption and phased in over a period of two days, assuming no apport reports are submitted that are interpreted as regressions (if this is the case, it is wound back to 0%). If the verification is failed or the package is rejected, it is removed.

Pending (unverified) SRUs can be found here.
The phased updates for packages can be found here.

Otherwise, the upload is for the development release. If there is no hard freeze in effect, an email will be sent from "Ubuntu Installer" to the uploader confirming the upload, and another copy is sent to the CODENAME-changes mailing list at lists.ubuntu.com, in the format of https://lists.ubuntu.com/CODENAME-changes. It will then go through automated regression testing, which includes testing the packages and its reverse dependencies for autopkgtest regressions, testing if it is installable in the release pocket, and if there are any blocking bugs against it. If all checks pass (or the Release Team overrides), the package is then moved to devel-release (which is an invisible pocket pointing at devel) and an email is sent to the uploader from "Ubuntu Installer" confirming that it has migrated.

TODO: explain pockets.
TODO: explain how to view this all visually at Launchpad (just found from !upkg PACKAGE but blunt pointers might be nice).
TODO: explain the Archive Admin's role in moving packages to/from components, why some binaries are in Universe while the sources are in Main, and how to tell what's published where that doesn't involve manually grepping files from archive.ubuntu.com (Launchpad).

Bonus points

Upload to a PPA

This is a great solution if you'd like people to test a change before you get it uploaded.

Requirements

  1. gpg setup above
  2. Add your key to Launchpad
  3. Create your PPA

Procedure

  1. Start in the folder containing your debian folder of whatever package you're working on
  2. Get the source:
    1. uscan --download-current-version
    2. tar -x --strip-components=1 -f ../PACKAGE_*.orig.tar.xz
  3. dch -i a new changelog entry and make sure the version number ends with ppa# where # is some number. This will make the package essentially a higher version than those in the repos.
  4. Build the source: debuild -S -d
  5. Upload it to Launchpad: dput ppa:your-launchpad-id/your-ppa-name ../PACKAGE_*source.changes

You should immediately get an accepted message by email. Then it starts building. This can take a while. Be patient…

Last Author
wxl
Last Edited
Oct 10 2018, 11:13 AM

Event Timeline

wxl created this document.Aug 17 2018, 11:40 AM
wxl edited the content of this document. (Show Details)
wxl added a project: Development Team.
wxl edited the content of this document. (Show Details)
wxl added subscribers: wxl, tsimonq2.
wxl edited the content of this document. (Show Details)Aug 20 2018, 11:02 PM
wxl edited the content of this document. (Show Details)Aug 20 2018, 11:04 PM
wxl edited the content of this document. (Show Details)
wxl edited the content of this document. (Show Details)Aug 20 2018, 11:19 PM
wxl edited the content of this document. (Show Details)Aug 21 2018, 12:05 AM
wxl edited the content of this document. (Show Details)
wxl edited the content of this document. (Show Details)Aug 21 2018, 12:08 AM
wxl edited the content of this document. (Show Details)Aug 21 2018, 12:40 AM
wxl edited the content of this document. (Show Details)Aug 22 2018, 10:29 AM
wxl edited the content of this document. (Show Details)Aug 23 2018, 12:23 AM
wxl edited the content of this document. (Show Details)Aug 23 2018, 12:33 AM
wxl edited the content of this document. (Show Details)Aug 23 2018, 10:23 AM
wxl edited the content of this document. (Show Details)Aug 24 2018, 10:52 AM
wxl edited the content of this document. (Show Details)Aug 24 2018, 11:54 AM
wxl edited the content of this document. (Show Details)Aug 26 2018, 6:07 PM
wxl edited the content of this document. (Show Details)Aug 30 2018, 12:00 AM
wxl edited the content of this document. (Show Details)Sep 3 2018, 2:40 AM
wxl edited the content of this document. (Show Details)Sep 4 2018, 12:36 AM
wxl edited the content of this document. (Show Details)Sep 19 2018, 1:16 AM
wxl edited the content of this document. (Show Details)
wxl edited the content of this document. (Show Details)Sep 19 2018, 10:15 AM

It would be good to explain how to create teh SSH key. And how to upload. To createit here is a good tutorial:
https://help.launchpad.net/YourAccount/CreatingAnSSHKeyPair
anf for upload the user should go to their settings (click on avatar) and then click on the left "SSH Public keys" to upload the one that was generated in the above steps that's inthe file id_rsa.pub.

tsimonq2 changed the edit policy from "Development Team (Project)" to "All Users".Sep 25 2018, 9:12 AM
hmollercl edited the content of this document. (Show Details)Sep 25 2018, 9:17 AM
hmollercl edited the content of this document. (Show Details)
hmollercl published a new version of this document.
hmollercl edited the content of this document. (Show Details)Sep 26 2018, 2:21 PM
wxl edited the content of this document. (Show Details)Oct 1 2018, 1:56 AM
wxl edited the content of this document. (Show Details)Oct 1 2018, 3:20 AM
wxl edited the content of this document. (Show Details)Oct 1 2018, 11:30 PM
wxl edited the content of this document. (Show Details)Oct 2 2018, 12:19 AM
wxl edited the content of this document. (Show Details)Oct 2 2018, 12:34 AM
wxl edited the content of this document. (Show Details)Oct 3 2018, 2:11 AM
wxl edited the content of this document. (Show Details)Oct 4 2018, 2:59 PM
wxl edited the content of this document. (Show Details)Oct 4 2018, 3:15 PM
wxl edited the content of this document. (Show Details)Oct 6 2018, 5:43 PM
wxl edited the content of this document. (Show Details)
wxl edited the content of this document. (Show Details)Oct 7 2018, 1:13 AM
wxl edited the content of this document. (Show Details)Oct 7 2018, 1:18 AM
wxl edited the content of this document. (Show Details)Oct 7 2018, 10:05 PM
wxl published a new version of this document.
wxl edited the content of this document. (Show Details)Oct 8 2018, 11:17 AM
tsimonq2 edited the content of this document. (Show Details)Oct 8 2018, 2:16 PM
tsimonq2 edited the content of this document. (Show Details)
tsimonq2 edited the content of this document. (Show Details)Oct 8 2018, 2:34 PM
tsimonq2 edited the content of this document. (Show Details)Oct 8 2018, 2:55 PM
tsimonq2 published a new version of this document.
tsimonq2 edited the content of this document. (Show Details)Oct 8 2018, 2:57 PM
tsimonq2 edited the content of this document. (Show Details)

Now that packages in launchpad are moving to git from bzr, many from desktop-team suggested to use git-buildpackage instead of quilt directly. I was wondering why though?

Most developers still use Quilt in Debian, and it's much more stable from what I remember.

wxl edited the content of this document. (Show Details)Oct 10 2018, 11:13 AM