A new Debian package helper: debputy

I have made a new helper for producing Debian packages called debputy. Today, I uploaded it to Debian unstable for the first time. This enables others to migrate their package build using dh +debputy rather than the “classic” dh. Eventually, I hope to remove dh entirely from this equation, so you only need debputy. But for now, debputy still leverages dh support for managing upstream build systems.

The debputy tool takes a radicially different approach to packaging compared to our existing packaging methods by using a single highlevel manifest instead of all the debian/install (etc.) and no “hook targets” in debian/rules.

Here are some of the things that debputy can do or does:

• debputy can perform installation similar to dh_install, dh_installdocs (etc.) plus a bit of the dh-exec support. Notably, debputy supports “install /usr/bin/foo into foo” and “install everything else in /usr/bin into foo-utils” without you having to resort to weird tricks. With debhelper, this would require dh-exec‘s => /usr/bin/foo operator.
• debputy can assign mode to files without needing hooks and static file ownership can be assigned without resorting to fakeroot. If you request Rules-Requires-Root: no, debputy will assemble the deb without using fakeroot. The fragileness of fakeroot may some day just be a horror story we tell our unruly children that they do not really believe is true.
• debputy defaults to all scripts with a “#! /bin/tool” or “#! /usr/bin/tool” to have mode 0755 (a+x) unless they are placed in directories known not to have executable files (such as the perl module dirs). As an example, scripts in the examples directory may now get an automatic executable bit if they have a proper #!-line for /usr/bin or /bin.
• debputy supports the default flow of 48 debhelper tools. If you are using pure dh $@ with no sequence add-ons and no hook targets in debian/rules (or only hook targets for the upstream side), odds are that debputy got your needs covered.

There are also some features that debputy does not support at the moment:

• Almost any debhelper sequence add-on. The debputy tool comes with a migration tool that will auto-detect any unsupported dh add-on from Build-Depends and will flag them as potential problematic. The migration tool works from an list of approved add-ons. Note that manually activated add-ons via dh $@ –with … are not detected.
• Anything that installs or recently installed into /lib or another /usr-merged location. My life is too short to be directly involved in the /usr-merge transition. This means no udev and no systemd unit support at the moment (note tmpfiles and sysusers is supported). For the systemd side, I am also contemplating a different model for how to deal with services. Even without the /usr-merge transition, these would not have been supported. The migration tool will detect problematic file in the debian directory immediately and debputy will detect upstream provided systemd unit files at build time.
• There is also no support for packager provided maintscript files at this time. If you have your own maintscripts, then you will not be able to migrate. The migration tool detects the debhelper based path and warns you (such as debian/postinst).
• Additionally, if you need special cases in tools (such as perl-base dependency with dh_perl) or rely on dh_strip-nondeterminsm for reproducibility, then you cannot or is advised not to migrate at this time.

There are all limitations of the current work in progress. I hope to resolve them all in due time.

Trying debputy

With the limitations aside, lets talk about how you would go about migrating a package:

# Assuming here you have already run: apt install dh-debputy
$ git clone https://salsa.debian.org/rra/kstart
[...]
$ cd kstart
# Add a Build-Dependency on dh-sequence-debputy
$ perl -n -i -e \
   'print; print " dh-sequence-debputy,\n" if m/debhelper-compat/;' \
    debian/control
$ debputy migrate-from-dh --apply-changes
debputy: info: Loading plugin debputy (version: archive/debian/4.3-1) ...
debputy: info: Verifying the generating manifest
debputy: info: Updated manifest debian/debputy.manifest
debputy: info: Removals:
debputy: info:   rm -f "./debian/docs"
debputy: info:   rm -f "./debian/examples"

debputy: info: Migrations performed successfully

debputy: info: Remember to validate the resulting binary packages after rebuilding with debputy
$ cat debian/debputy.manifest 
manifest-version: '0.1'
installations:
- install-docs:
    sources:
    - NEWS
    - README
    - TODO
- install-examples:
    source: examples/krenew-agent
$ git add debian/debputy.manifest
$ git commit --signoff -am"Migrate to debputy"
# Run build tool of choice to verify the output.

This is of course a specific example that works out of the box. If you were to try this on debianutils (from git), the output would look something like this:

$ debputy migrate-from-dh
debputy: info: Loading plugin debputy (version: 5.13-13-g9836721) ...
debputy: error: Unable to migrate automatically due to missing features in debputy.

  * The "debian/triggers" debhelper config file (used by dh_installdeb is currently not supported by debputy.

Use --acceptable-migration-issues=[...] to convert this into a warning [...]

And indeed, debianutils requires at least 4 debhelper features beyond what debputy can support at the moment (all related to maintscripts and triggers).

Rapid feedback

Rapid feedback cycles are important for keeping developers engaged in their work. The debputy tool provides the following features to enable rapid feedback.

Immediate manifest validation

It would be absolutely horrible if you had to do a full-rebuild only to realize you got the manifest syntax wrong. Therefore, debputy has a check-manifest command that checks the manifest for syntactical and semantic issues.

$ cat debian/debputy.manifest
manifest-version: '0.1'
installations:
- install-docs:
    sources:
    - GETTING-STARTED-WITH-dh-debputy.md
    - MANIFEST-FORMAT.md
    - MIGRATING-A-DH-PLUGIN.md
$ debputy check-manifest
debputy: info: Loading plugin debputy (version: 0.1.7-1-gf34bd66) ...
debputy: info: No errors detected.
$ cat <<EOF >> debian/debputy.manifest
- install:
    sourced: foo
    as: usr/bin/foo
EOF
# Did I typo anything?
$ debputy check-manifest
debputy: info: Loading plugin debputy (version: 0.1.7-2-g4ef8c2f) ...
debputy: warning: Possible typo: The key "sourced" at "installations[1].install" should probably have been 'source'
debputy: error: Unknown keys "{'sourced'}" at installations[1].install".  Keys that could be used here are: sources, when, dest-dir, source, into.
debputy: info: Loading plugin debputy (version: 0.1.7-2-g4ef8c2f) ...
$ sed -i s/sourced:/source:/ debian/debputy.manifest
$ debputy check-manifest
debputy: info: Loading plugin debputy (version: 0.1.7-2-g4ef8c2f) ...
debputy: info: No errors detected.

The debputy check-manifest command is limited to the manifest itself and does not warn about foo not existing as it could be produced as apart of the upstream build system. Therefore, there are still issues that can only be detected at package build time. But where debputy can reliably give you immediate feedback, it will do so.

Idempotence: Clean re-runs of dh_debputy without clean/rebuild

If you read the “fine print” of many debhelper commands, you may see the following note their manpage:

This command is not idempotent. dh_prep(1) should be called between

invocations of this command …

Manpage of an anonymous debhelper tool

What this usually means, is that if you run the command twice, you will get its maintscript change (etc.) twice in the final deb. This fits into our “single-use clean throw-away chroot builds” on the buildds and CI as well as dpkg-buildpackage‘s “no-clean” (-nc) option. Single-use throw-away chroots are not very helpful for debugging though, so I rarely use them when doing the majority of my packaging work as I do not want to wait for the chroot initialization (including installing of build-depends).

But even then, I have found that dpkg-buildpackage -nc has been useless for me in many cases as I am stuck between two options:

• With -nc, you often still interact with the upstream build system. As an example, debhelper will do a dh_prep followed by dh_auto_install, so now we are waiting for upstream’s install target to run again. What should have taken seconds now easily take 0.5-1 minute extra per attempt.
• If you want to by-pass this, you have to manually call the helpers needed (in correct order) and every run accumulates cruft from previous runs to the point that cruft drowns out the actual change you want to see. Also, I am rarely in the mood to play human dh, when I am debugging an issue that I failed to fix in my first, second and third try.

As you can probably tell, neither option has worked that well for me. But with dh_debputy, I have made it a goal that it will not “self-taint” the final output. If dh_debputy fails, you should be able to tweak the manifest and re-run dh_debputy with the same arguments.

• No waiting for dpkg-buildpackage -nc nor anything implied by that.
• No “self-tainting” of the final deb. The result you get, is the result you would have gotten if the previous dh_debputy run never happened.
• Because dh_debputy produces the final result, I do not have to run multiple tools in “the right” order.

Obviously, this is currently a lot easier, because debputy is not involved in the upstream build system at all. If this feature is useful to you, please do let me know and I will try to preserve it as debputy progresses in features.

Packager provided files

On a different topic, have you ever wondered what kind of files you can place into the debian directory that debhelper automatically picks up or reacts too? I do not have an answer to that beyond it is over 80 files and that as the maintainer of debhelper, I am not willing to manually maintain such a list manually.

However, I do know what the answer is in debputy, because I can just ask debputy:

$ debputy plugin list packager-provided-files
+-----------------------------+---------------------------------------------[...]
| Stem                        | Installed As                                [...]
+-----------------------------+---------------------------------------------[...]
| NEWS                        | /usr/share/doc/{name}/NEWS.Debian           [...]
| README.Debian               | /usr/share/doc/{name}/README.Debian         [...]
| TODO                        | /usr/share/doc/{name}/TODO.Debian           [...]
| bug-control                 | /usr/share/bug/{name}/control               [...]
| bug-presubj                 | /usr/share/bug/{name}/presubj               [...]
| bug-script                  | /usr/share/bug/{name}/script                [...]
| changelog                   | /usr/share/doc/{name}/changelog.Debian      [...]
| copyright                   | /usr/share/doc/{name}/copyright             [...]
[...]

This will list all file types (Stem column) that debputy knows about and it accounts for any plugin that debputy can find. Note to be deterministic, debputy will not auto-load plugins that have not been explicitly requested during package builds. So this list could list files that are available but not active for your current package.

Note the output is not intended to be machine readable. That may come in later version. Feel free to chime in if you have a concrete use-case.

Take it for a spin

As I started this blog post with, debputy is now available in unstable. I hope you will take it for a spin on some of your simpler packages and provide feedback on it. 🙂

For documentation, please have a look at:

• GETTING-STARTED-WITH-dh-debputy.md (how-to guide)
• MANIFEST-FORMAT.md (reference documentation)

Thanks for considering

PS: My deepest respect to the fakeroot maintainers. That game of whack-a-mole is not something I would have been willing to maintain. I think fakeroot is like the Python GIL in the sense that it has been important in getting Debian to where it is today. But at the same time, I feel it is time to let go of the “crutch” and find a proper solution.

wrap-and-sort with experimental support for comments in devscripts/2.22.2

In the devscripts package currently in Debian testing (2.22.2), wrap-and-sort has opt-in support for preserving comments in deb822 control files such as debian/control and debian/tests/control. Currently, this is an opt-in feature to provide some exposure without breaking anything.

To use the feature, add --experimental-rts-parser to the command line. A concrete example being (adjust to your relevant style):

wrap-and-sort --experimental-rts-parser -tabk

Please provide relevant feedback to #820625 if you have any. If you experience issues, please remember to provide the original control file along with the concrete command line used.

As hinted above, the option is a temporary measure and will be removed again once the testing phase is over, so please do not put it into scripts or packages. For the same reason, wrap-and-sort will emit a slightly annoying warning when using the option.

Enjoy. 🙂

Improvements to IntelliJ/PyCharm support for Debian packaging files

I have updated my debpkg plugin for IDEA (e.g. IntelliJ, PyCharm, Android Studios) to v0.0.8. Here are some of the changes since last time I wrote about the plugin.

New file types supported

• .dsc (as a generic deb822 file)
• .buildinfo and .changes (as a generic deb822 provided the base name looks like it follows the “package_version_arch” pattern)
• debian/copyright (as a DEP-5 / machine-readable copyright file if the Format field exists).

Links for URLs and bug closes

There are often links in deb822 files or the debian/changelog and as of v0.0.8, the plugin will now highlight them and able you to easily open them via your browser. In the deb822 case, they generally appear in the Homepage field, the Vcs-* fields or the Format field of the debian/copyright field. For the changelog file, they often appear in the form of bug Closes statements – such as the #123456 in “Closes: #123456”, which is a reference to https://bugs.debian.org/123456.

Improvements to debian/control

The dependency validator now has “per-field” knowledge. This enables it to flag dependency relations in the Provides field that uses operators other than “=” (which is the only operator that is supported in that field). It also knows which fields support build-profile restrictions. It in theory also do Architecture restrictions, but I have not added it – among other because it gets a bit spicy around binary packages. (Fun fact, you can have “Depends: foo [amd64]” but only for arch:any packages.)

The plugin now suggests adding a Rules-Requires-Root field to the Source stanza along with a “quick fix” for adding the field. Admittedly, it was mostly done as exercise for me to learn how to do that kind of feature.

Support for machine-readable debian/copyright

The plugin now has a dedicated file type for debian/copyright that follows the machine-readable format. It should auto-detect it automatically based on the presence of the “Format” field being set to https://www.debian.org/doc/packaging-manuals/copyright-format/1.0. Sadly, I have not found the detection reliable in all cases, so you may have to apply it manually.

With the copyright format, the plugin now scans the “Files” fields for common issues like pointing on non-existing paths and invalid escape sequences. When the plugin discovers a path that does not match anything, it highlights the part of the path that cannot be found. As an example, consider the pattern “src/foo/data.c” and that “src/foo” exist but “data.c” does not exist, then the plugin will only flag the “data.c” part of “src/foo/data.c” as invalid.

The plugin will also suggest a quick fix if you a directory into the Files field to replace it with a directory wildcard (e.g. “src/foo” -> “src/foo/*”), which is how the spec wants you to reference every file beneath a given directory.

Finally, when the plugin can identify part of the path, then it will turn it into a link (reference in IDEA lingo). This means that you can CTRL + click on it to jump to the file. As a side-effect, it also provides refactoring assistance for renaming files, where renaming a file will often be automatically reflected in debian/copyright. This use case is admittedly mostly relevant people, who are both upstream and downstream maintainer.

Folding support improvement for .dsc/.changes/.buildinfo files

The new field types appeared with two cases, where I decided to improve the folding support logic.

The first was the GPG signature (if present), which consists of two parts. The top part with is mostly a single line marker but often followed by a GPG armor header (e.g. “Hash: SHA512”) and then the signature blob with related marker lines around it. Both cases are folded into a single marker line by default to reduce their impact on content in the editor view.

The second case was the following special-case pattern:

Files:
 <md5> <size> filename
Checksums-Sha256:
 <sha256> <size> filename

In the above example, where there is exactly on file name, those fields will by default now be folded into:

Files: <md5> <size> filename
Checksums-Sha256: <sha256> <size> filename

For all other multi-line fields, the plugin still falls back to a list of known fields to fold by default as in previous versions.

Spellchecking improvements

The plugin already supported selective spell checking in v0.0.3, where it often omitted spell checking for fields (in deb822 files) where it did not make sense. The spell check feature has been improved by providing a list of known packaging terms/jargo used by many contributors (so “autopkgtests” is no longer considered a typo). This applies to all file types (probably also those not handled by the plugin as it is just a dictionary).

Furthermore, the plugin also attempts discover common patterns (e.g. file names or command arguments) and exempt these from spell checking in the debian/changelog. This also includes manpage references such as “foo.1” or “foo(1)”. It is far from perfect and relies on common patterns to exclude spell checking. Nonetheless, it should reduce the number of false positive considerably.

Feedback welcome

Please let me know if you run into bugs or would like a particular feature implemented. You can submit bug reports and feature requests in the issue tracker on github.

Support for Debian packaging files in IDEA (IntelliJ/PyCharm)

I have been using the community editions of IntelliJ and PyCharm for a while now for Python or Perl projects. But it started to annoy me that for Debian packaging bits it would “revert” into a fancy version of notepad. Being fed up with it, I set down and spent the last week studying how to write a plugin to “fix” this.

After a few prototypes, I have now released IDEA-debpkg v0.0.3 (Link to JetBrain’s official plugin listing with screenshots). It provides a set of basic features for debian/control like syntax highlighting, various degree of content validation, folding of long fields, code completion and “CTRL + hover” documentation. For debian/changelog, it is mostly just syntax highlighting with a bit of fancy linking for now. I have not done anything for debian/rules as I noted there is a Makefile plugin, which will have to do for now.

The code is available from github and licensed under Apache-2.0. Contributors, issues/feature requests and pull requests are very welcome. Among things I could help with are:

• Icons – both for the plugin and for the file types. Currently it is just colored text, which is as far as my artistic skills got with the space provided.
• Color and text formatting for syntax highlighting.
• Reports of papercut or features that would be very useful to prioritize.
• Review of the “CTRL + hover” documentation. I am hoping for something that is help for new contributors but I am very unlikely to have gotten it right (among other because I wrote most of it to “get it done” rather than “getting it right”)

I hope you will take it for spin if you have been looking for a bit of Debian packaging support to your PyCharm or other IDEA IDE. 🙂 Please do file bugs/issues if you run into issues, rough edges or unhelpful documentation, etc.

A decline in the use of hints in the release team

While we working the release, I had a look at how many hints we have deployed during buster like I did for wheezy a few years back.  As it seemed we were using a lot fewer hints than previously and I decided to take a detour into “stats-land”. 🙂

When I surfaced from “stats-land”, I confirmed that we have a clear decline in hints in the past two releases[1].

wheezy: 3301
jessie: 3699 (+398)
stretch: 2408 (-1291)
buster: 1478 (-930)

While it is certainly interesting, the number of hints on its own is not really an indicator of how much work we put into the release.  Notably, it says very little about the time spent on evaluating unblock requests before adding the hint.

 

[1]

Disclaimer: This are very rough estimate based on same method as from the previous blog post using entire complete months as smallest time unit. It is apparently not very accurate either.  The observant reader will note that the number for wheezy does not match the number I posted years ago (3254 vs 3301).  I am not entirely sure what causes the difference as I am using the same regex for wheezy and mtime for the files look unchanged.

 

Making debug symbols discoverable and fetchable

Michael wrote a few days ago about the experience of debugging programs on Debian.  And he is certainly not the only one, who found it more difficult to find debug symbols on Linux systems in general.

But fortunately, it is a fixable problem.  Basically, we just need a service to map a build-id to a downloadable file containing that build-id.  You can find the source code to my (prototype) of such a dbgsym service on salsa.debian.org.

It exposes one API endpoint, “/api/v1/find-dbgsym”, which accepts a build-id and returns some data about that build-id (or HTTP 404 if we do not know the build-id).  An example:

$ curl --silent http://127.0.0.1:8000/api/v1/find-dbgsym/5e625512829cfa9b98a8d475092658cb561ad0c8/ | python -m json.tool
{
    "package": {
        "architecture": "amd64",
        "build_ids": [
            "5e625512829cfa9b98a8d475092658cb561ad0c8"
        ],
        "checksums": {
            "sha1": "a7a38b49689031dc506790548cd09789769cfad3",
            "sha256": "3706bbdecd0975e8c55b4ba14a481d4326746f1f18adcd1bd8abc7b5a075679b"
        },
        "download_size": 18032,
        "download_urls": [
            "https://snapshot.debian.org/archive/debian-debug/20161028T101957Z/pool/main/6/6tunnel/6tunnel-dbgsym_0.12-1_amd64.deb"
        ],
        "name": "6tunnel-dbgsym",
        "version": "1:0.12-1"
    }
}

Notice how it includes a download URL and a SHA256 checksum, so with this you can download the package containing the build-id directly from this and verify the download. The sample_client.py included in the repo does that and might be a useful basis for others interested in developing a client for this service.

 

To seed the database, so it can actually answer these queries, there is a bulk importer that parses Packages files from the Debian archive (for people testing: the ones from debian-debug archive are usually more interesting as they have more build-ids).

Possible improvements

• Have this service deployed somewhere on the internet rather than the loopback interface on my machine.
• The concept is basically distribution agnostic (Michael’s post in fact links to a similar service) and this could be a standard service/tool for all Linux distributions (or even just organizations).  I am happy to work with people outside Debian to make the code useful for their distribution (including non-Debian based distros).
  • The prototype was primarily based around Debian because it was my point of reference (plus what I had data for).
• The bulk importer could (hopefully) be faster on new entries.
• The bulk importer could import the download urls as well, so we do not have to fetch the relevant data online the first time when people are waiting.
• Most of the django code / setup could probably have been done better as this has been an excuse to learn django as well.
  • Like some basic content on the service to help people get started.
  • And a test suite to avoid regressions.

Patches and help welcome.

Kudos

This prototype would not have been possible without python3, django, django’s restframework, python’s APT module, Postgresql, PoWA and, of course, snapshot.debian.org (including its API).

Improvements to apt-file since stretch

The list of changes for apt-file in buster is rather short, but I would still like to mention a few of them in this post.

New list-indices command:

In stretch, apt-file migrated to use apt’s new acquire system and that meant a lot of changes to apt-file’s interface.  Among other, you could no longer search for source packages via the “-a source” but had to use “-Idsc” instead.  While the manpage has some runes about these names, but not everyone finds them helpful.

To do a bit better, we have included a new list-indices command in apt-file that enables you see exactly which indices that apt-file recognises and whether we can find any data in them.  This looks like the following:

$ apt-file list-indices
+-----------------+-----------------------------+-----------------+
| Index Name (-I) | DefaultEnabled (Apt config) | Index Status    |
+-----------------+-----------------------------+-----------------+
| deb             | <unset>                     | Ok              |
| udeb            | false                       | Empty (code: 4) |
| dsc             | false                       | Empty (code: 4) |
+-----------------+-----------------------------+-----------------+

At the moment, the command outputs three columns:

• “Index Name (-I)” – This is the value you pass to -I/–index-names to search in that index.
• “DefaultEnabled (Apt config)” – This is the value apt-file found while asking apt-config about the index configuration.  As you can disable fetching of Contents files on a “per sources.list” basis, so the above is only the default for URIs you fetch from.
• “Index Status” – This tells you whether apt-file can actually search for anything in that index.  It comes in 3 basic status: Ok (i.e. some searchable data found), Empty (no Contents files found) and Error (an unexpected error happened while checking the status but it did not stop apt-file from generating the table).

Note: It is intended to be human-readable and is not (intended to be) machine readable.  If you need this data in a script or for automation purposes, please use apt-config plus apt-get indextargets directly and extra the data you want via those tools.  The output format of this command may change without notice (if another format is found better suited, etc.).

Status feedback (terminal-only):

The new version of apt-file also provides some feedback to your terminal while you wait for your search.  It comes in the form of a single line showing you what apt-file is doing. Below is the output from apt-file (being interrupted) at various stages.

$ apt-file show lintian
Searching for contents of the packages ...^C
$ apt-file show lintian
Searching, found 370 results so far ...^C

(note: The number of “results” here is prior to de-duplicating the results.  You may see a much larger number here than in the final result)

The output is disabled when you redirect stdout, so most scripts are unaffected by this change:

$ apt-file show lintian | head -n1
lintian: /etc/lintianrc

Faster by default:

Historically, Contents files had a human-readable header that described what the files contented, how to read the file and column headers.  Unfortunately, they made apt-file’s life harder as we had to reliably ignore these headers without losing actual data from them.  However, both dak and reprepro has stopped adding the headers, so apt-file can now skip the header filtering by default.

If you still need to search for files in Contents files with the headers, you can re-enable the check (with related performance hit) by setting apt-file::Parser::Check-For-Description-Header to true in your configuration.  E.g.

$ time apt-file search nothing-else-matches
                                          
real    0m1.853s
[...]
$ time apt-file search -o apt-file::Parser::Check-For-Description-Header=true nothing-else-matches
                                          
real    0m7.875s

Actual performance benefit largely depends on cache size and (apparently) also what you search for.  🙂

“debhelper-compat (= 12)” is now released

A few days ago, we released debhelper/12 and yesterday uploaded it to stretch-backports (as debhelper/12~bpo9+1).  We deliberately released debhelper/12 so it would be included in buster for the people, who backport their packages to older releases via stable-backports.  That said, we would like to remind people to please be careful with bumping the debhelper compat level at this point of the release cycle.  We generally recommand you defer migrating to compat 12 until bullseye (to avoid having to revert that change in case you need an unblock for the buster release).

Upgrading to compat 12

If/when you upgrade the compat level, please consider using the recently added debhelper-compat (= 12) build-dependency form.  It reduces redundancy, centralizes your debhelper relations to debian/control (replacing debian/compat) and avoids having you remember that you need a ~ in your build-dependency to support backports.

As usual, you can read about which behavioural changes are introduced by compat 12 in the debhelper(7) manpage.  About one third of the changes are removing deprecated features and the rest are mostly about tweaking “minor” defaults.  Though, please be careful around systemd unit files for two reasons:

1. debhelper cleanly separates the handling of systemd unit files, so dh_installsystemd now 100% manages these while dh_installinit takes care of the sysvinit scripts.  In particular, if you have something like dh_installinit –no-start then you almost certainly also want a dh_installsystemd –no-start if you have systemd unit shadowing the sysvinit file.
2. Caveat for stretch-backports support: Due to the above, we need a versioned Pre-Depends on init-system-helpers.  That version is unfortunately not available in stretch nor stretch-backports and therefore packages relying on this cannot be safely backported to stretch (but they will be backportable to buster).

If you target stretch-backports and ship services, we recommend you to stay with compat 11 for now.

General changes since stretch (i.e. 10.2.5 to 12):

The following are some of the changes that have been introduced in debhelper since the stretch release (available via stretch-backports):

• dh_missing was added to take over dh_install –list-missing/–fail-missing while reducing some of the issues that dh_install had with those options.
• debhelper now supports the meson+ninja and cmake+ninja build system.
• Improved or added cross-compilation support (via the dh_auto_* tools) for:
  • meson build system
  • cmake build system
  • (“plain”) makefile system (parts only in compat 11+)
  • qmake build system (qt5)
• Experimental support for cross-building for TARGET rather than HOST (for the about less than 5 source packages in total that might find this useful).
• Improved bulk performance in various parts of debhelper.
• Support for “nodoc” profile/build option plus the “terse” build option.
• Correctly handle systemd units with \x2d (escaped “-“) in their name.
• Rules-Requires-Root support when the field is set and dpkg-dev notifies debhelper that it supports the feature (requires dpkg-dev from Debian buster).  Besides removing the need for (fake)root it can also remove about 3 invocations of debian/rules.
• Reduced dbgsym files via dh_dwz (use either manually, with dh –with dwz or dh + compat 12).
• Enable dh to skip more no-op commands including dh_auto_* and to a minor extend also even when dh is passed arguments that it should pass on to the underlying tools.
• Support for setting debhelper compat level via debhelper-compat (= X) build-dependency and load dh add-on sequences via dh-sequence-foo build-dependency (as an alternative to the original methods).
• Support for derivatives and custom/local builds using DH_EXTRA_ADDONS to enable derivative or custom add-ons for debhelper.  Note: Packagers should keep using –with foo or the new dh-sequence-foo build-dependency – this interface is intended to enable a particular add-on without changing the package.
• Improved maintscript snippet ordering to ensure that service enable + start related snippets always run last in postinst (and first in prerm etc.) in the code inserted via the #DEBHELPER# token.  This ensures that all other scripts (e.g. configuration file management via dh_ucf or the debian/maintscript file) is complete by the time the service (re)starts.
• Improved “rollback” handling of maintscripts generated by debhelper. Among other, debhelper autoscripts now handle cases like abort-deconfigure and abort-upgrade.  In general, they are handled like configure and replays the setup to ensure that services are correctly running as expected.
• The autoscript snippet for loading systemd tmpfiles now simply uses the basename of the tmpfiles configuration, which enables the administrator to override the package provided tmpfiles configuration by placing their own in /etc/tmpfiles.d.
• The new dh_installinitramfs tool now installs maintainer provided initramfs hooks and generates autosnippets for all hooks installed in /usr/share/initramfs-tools/hooks.  Enable via dh –with installinitramfs or dh + compat 12 or call it manually.
• The new dh_installsystemduser which manages system units per user rather than for the system.  Enable via dh + compat 12 or call it manually.

The above is by no means complete and among other excludes many things that is introduced in compat 11 or compat 12.

Thanks

Many thanks to the following people, who contributed to debhelper since stretch release with one or more patches (based on git log debian/10.2.5..debian/12 | git shortlog):

Adam Conrad, Américo Monteiro, Axel Beckert, Baptiste Jammet, Chris Lamb, Chris Leick, Christoph Biedl, Clément Hermann, Colin Watson, Daniele Nicolodi, Dmitry Shachnev, Fabian Grünbichler, Fabian Wolff, Felipe Sateler, Geoffrey Thomas, Helmut Grohne, Hideki Yamane, Iain Lane, Isaac Jurado, Jakub Wilk, Johannes Schauer, Josh Triplett, Juhani Numminen, Lisandro Damián Nicanor Pérez Meyer, Luca Boccassi, Mattia Rizzolo, Michael Biebl, Michael Stapelberg, Nicholas Guriev, Nicolas Boulenguez, Niels Thykier, Olly Betts, Paul Tagliamonte, Peter Pentchev, Roberto C. Sánchez, Steven Chamberlain, Sven Joachim, Ville Skyttä, gregor herrmann

Also, many thanks to the people reporting bugs, regressions and feature suggestions via the Debian BTS.

Buster is headed for a long hard freeze

We are getting better and better accumulating RC bugs in testing. This is unfortunate because the length of the freeze is strongly correlated with the number of open RC bugs affecting testing. If you believe that Debian should have short freezes, then it will require putting effort behind that belief and fix some RC bugs – even in packages that are not maintained directly by you or your team and especially in key packages.

The introduction of key packages have been interesting. On the plus side, we can use it to auto-remove RC buggy non-key packages from testing which has been very helpful. On the flip-side, it also makes it painfully obvious that over 50% of all RC bugs in testing are now filed against key packages (for the lazy; we are talking about 475 RC bugs in testing filed against key packages; about 25 of these appear to be fixed in unstable).

Below are some observations from the list of RC bugs in key packages (affecting both testing and unstable – based on a glance over all of the titles).

• About 85 RC bugs related to (now) defunct maintainer addresses caused by the shutdown of Alioth. From a quick glance, it appears that the Debian Xfce Maintainers has the largest backlog – maybe they could use another team member.  Note they are certainly not the only team with this issue.
• Over 100 RC bugs are FTBFS for various reasons..  Some of these are related to transitions (e.g. new major versions of GCC, LLVM and OpenJDK).

Those three points alone accounts for 40% of the RC bugs affecting both testing and unstable.

We also have several contributors that want to remove unmaintained, obsolete or old versions of  packages (older versions of compilers such as GCC and LLVM, flash-players/tooling, etc.).  If you are working on this kind of removal, please remember to follow through on it (even if it means NMU packages).  The freeze is not the right time to remove obsolete key packages as it tends to involve non-trivial changes of features or produced binaries.  As much of this as entirely possible ought to be fixed before 2019-01-12 (transition freeze).

 

In summary: If you want Debian Buster released in early 2019 or short Debian freezes in general, then put your effort where your wish/belief is and fix RC bugs today.  Props for fixes to FTBFS bugs, things that hold back transitions or keep old/unmaintained/unsupportable key packages in Buster (testing).

Build system changes in debhelper

Since debhelper/11.2.1[1], we now support using cmake for configure and ninja for build + test as an alternative to cmake for configure and make for build + test.  This change was proposed by Kyle Edwards in Debian bug #895044. You can try this new combination by specifying “cmake+ninja” as build system.

To facilitate this change, the cmake and meson debhelper buildsystems had to change their (canonical) name.  As such you may have noticed that the “–list” option for dh_auto_* now produces a slightly different output:

 

$ dh_auto_configure --list | grep +
cmake+makefile       CMake (CMakeLists.txt) combined with simple Makefile
cmake+ninja          CMake (CMakeLists.txt) combined with Ninja (build.ninja)
meson+ninja          Meson (meson.build) combined with Ninja (build.ninja)

 

You might also notice that “cmake” and “meson” is no longer listed in the full list of build systems.  To retain backwards compatibility, the names “cmake” and “meson” are handled as “cmake+makefile” and “meson+ninja”.  This can be seen if we specify a build system:

 

$ dh_auto_configure --buildsystem cmake --list | tail -n1
Specified: cmake+makefile (default for cmake)
$ dh_auto_configure --buildsystem cmake+makefile --list | tail -n1
Specified: cmake+makefile
$ dh_auto_configure --buildsystem cmake+ninja --list | tail -n1
Specified: cmake+ninja

 

If your package uses cmake, please give it a try and see what works and what does not.  So far, the only known issue is that cmake+ninja may fail if the package has no tests while it success with cmake+makefile.  I believe this is because the makefile build system checks whether the “test” or “check” targets exist before calling make.

Enjoy. 🙂

 

Footnotes:

[1] Strictly speaking, it was version 11.2.  However, version 11.2 had a severe regression that made it mostly useless.