A decline in the use of hints in the release team

While we working the release, I had a look at how many hints we have deployed during buster like I did for wheezy a few years back.  As it seemed we were using a lot fewer hints than previously and I decided to take a detour into “stats-land”. 🙂

When I surfaced from “stats-land”, I confirmed that we have a clear decline in hints in the past two releases[1].

wheezy: 3301
jessie: 3699 (+398)
stretch: 2408 (-1291)
buster: 1478 (-930)

While it is certainly interesting, the number of hints on its own is not really an indicator of how much work we put into the release.  Notably, it says very little about the time spent on evaluating unblock requests before adding the hint.



Disclaimer: This are very rough estimate based on same method as from the previous blog post using entire complete months as smallest time unit. It is apparently not very accurate either.  The observant reader will note that the number for wheezy does not match the number I posted years ago (3254 vs 3301).  I am not entirely sure what causes the difference as I am using the same regex for wheezy and mtime for the files look unchanged.


Posted in Debian, Release-Team | Leave a comment

Making debug symbols discoverable and fetchable

Michael wrote a few days ago about the experience of debugging programs on Debian.  And he is certainly not the only one, who found it more difficult to find debug symbols on Linux systems in general.

But fortunately, it is a fixable problem.  Basically, we just need a service to map a build-id to a downloadable file containing that build-id.  You can find the source code to my (prototype) of such a dbgsym service on salsa.debian.org.

It exposes one API endpoint, “/api/v1/find-dbgsym”, which accepts a build-id and returns some data about that build-id (or HTTP 404 if we do not know the build-id).  An example:

$ curl --silent | python -m json.tool
    "package": {
        "architecture": "amd64",
        "build_ids": [
        "checksums": {
            "sha1": "a7a38b49689031dc506790548cd09789769cfad3",
            "sha256": "3706bbdecd0975e8c55b4ba14a481d4326746f1f18adcd1bd8abc7b5a075679b"
        "download_size": 18032,
        "download_urls": [
        "name": "6tunnel-dbgsym",
        "version": "1:0.12-1"

Notice how it includes a download URL and a SHA256 checksum, so with this you can download the package containing the build-id directly from this and verify the download. The sample_client.py included in the repo does that and might be a useful basis for others interested in developing a client for this service.


To seed the database, so it can actually answer these queries, there is a bulk importer that parses Packages files from the Debian archive (for people testing: the ones from debian-debug archive are usually more interesting as they have more build-ids).

Possible improvements

  • Have this service deployed somewhere on the internet rather than the loopback interface on my machine.
  • The concept is basically distribution agnostic (Michael’s post in fact links to a similar service) and this could be a standard service/tool for all Linux distributions (or even just organizations).  I am happy to work with people outside Debian to make the code useful for their distribution (including non-Debian based distros).
    • The prototype was primarily based around Debian because it was my point of reference (plus what I had data for).
  • The bulk importer could (hopefully) be faster on new entries.
  • The bulk importer could import the download urls as well, so we do not have to fetch the relevant data online the first time when people are waiting.
  • Most of the django code / setup could probably have been done better as this has been an excuse to learn django as well.

Patches and help welcome.


This prototype would not have been possible without python3, django, django’s restframework, python’s APT module, Postgresql, PoWA and, of course, snapshot.debian.org (including its API).

Posted in Debian | Leave a comment

Improvements to apt-file since stretch

The list of changes for apt-file in buster is rather short, but I would still like to mention a few of them in this post.

New list-indices command:

In stretch, apt-file migrated to use apt’s new acquire system and that meant a lot of changes to apt-file’s interface.  Among other, you could no longer search for source packages via the “-a source” but had to use “-Idsc” instead.  While the manpage has some runes about these names, but not everyone finds them helpful.

To do a bit better, we have included a new list-indices command in apt-file that enables you see exactly which indices that apt-file recognises and whether we can find any data in them.  This looks like the following:

$ apt-file list-indices
| Index Name (-I) | DefaultEnabled (Apt config) | Index Status    |
| deb             | <unset>                     | Ok              |
| udeb            | false                       | Empty (code: 4) |
| dsc             | false                       | Empty (code: 4) |

At the moment, the command outputs three columns:

  • “Index Name (-I)” – This is the value you pass to -I/–index-names to search in that index.
  • “DefaultEnabled (Apt config)” – This is the value apt-file found while asking apt-config about the index configuration.  As you can disable fetching of Contents files on a “per sources.list” basis, so the above is only the default for URIs you fetch from.
  • “Index Status” – This tells you whether apt-file can actually search for anything in that index.  It comes in 3 basic status: Ok (i.e. some searchable data found), Empty (no Contents files found) and Error (an unexpected error happened while checking the status but it did not stop apt-file from generating the table).

Note: It is intended to be human-readable and is not (intended to be) machine readable.  If you need this data in a script or for automation purposes, please use apt-config plus apt-get indextargets directly and extra the data you want via those tools.  The output format of this command may change without notice (if another format is found better suited, etc.).

Status feedback (terminal-only):

The new version of apt-file also provides some feedback to your terminal while you wait for your search.  It comes in the form of a single line showing you what apt-file is doing. Below is the output from apt-file (being interrupted) at various stages.

$ apt-file show lintian
Searching for contents of the packages ...^C
$ apt-file show lintian
Searching, found 370 results so far ...^C

(note: The number of “results” here is prior to de-duplicating the results.  You may see a much larger number here than in the final result)

The output is disabled when you redirect stdout, so most scripts are unaffected by this change:

$ apt-file show lintian | head -n1
lintian: /etc/lintianrc

Faster by default:

Historically, Contents files had a human-readable header that described what the files contented, how to read the file and column headers.  Unfortunately, they made apt-file’s life harder as we had to reliably ignore these headers without losing actual data from them.  However, both dak and reprepro has stopped adding the headers, so apt-file can now skip the header filtering by default.

If you still need to search for files in Contents files with the headers, you can re-enable the check (with related performance hit) by setting apt-file::Parser::Check-For-Description-Header to true in your configuration.  E.g.

$ time apt-file search nothing-else-matches
real    0m1.853s
$ time apt-file search -o apt-file::Parser::Check-For-Description-Header=true nothing-else-matches
real    0m7.875s

Actual performance benefit largely depends on cache size and (apparently) also what you search for.  🙂

Posted in apt-file, Debian | Leave a comment

“debhelper-compat (= 12)” is now released

A few days ago, we released debhelper/12 and yesterday uploaded it to stretch-backports (as debhelper/12~bpo9+1).  We deliberately released debhelper/12 so it would be included in buster for the people, who backport their packages to older releases via stable-backports.  That said, we would like to remind people to please be careful with bumping the debhelper compat level at this point of the release cycle.  We generally recommand you defer migrating to compat 12 until bullseye (to avoid having to revert that change in case you need an unblock for the buster release).

Upgrading to compat 12

If/when you upgrade the compat level, please consider using the recently added debhelper-compat (= 12) build-dependency form.  It reduces redundancy, centralizes your debhelper relations to debian/control (replacing debian/compat) and avoids having you remember that you need a ~ in your build-dependency to support backports.

As usual, you can read about which behavioural changes are introduced by compat 12 in the debhelper(7) manpage.  About one third of the changes are removing deprecated features and the rest are mostly about tweaking “minor” defaults.  Though, please be careful around systemd unit files for two reasons:

  1. debhelper cleanly separates the handling of systemd unit files, so dh_installsystemd now 100% manages these while dh_installinit takes care of the sysvinit scripts.  In particular, if you have something like dh_installinit –no-start then you almost certainly also want a dh_installsystemd –no-start if you have systemd unit shadowing the sysvinit file.
  2. Caveat for stretch-backports support: Due to the above, we need a versioned Pre-Depends on init-system-helpers.  That version is unfortunately not available in stretch nor stretch-backports and therefore packages relying on this cannot be safely backported to stretch (but they will be backportable to buster).

If you target stretch-backports and ship services, we recommend you to stay with compat 11 for now.

General changes since stretch (i.e. 10.2.5 to 12):

The following are some of the changes that have been introduced in debhelper since the stretch release (available via stretch-backports):

  • dh_missing was added to take over dh_install –list-missing/–fail-missing while reducing some of the issues that dh_install had with those options.
  • debhelper now supports the meson+ninja and cmake+ninja build system.
  • Improved or added cross-compilation support (via the dh_auto_* tools) for:
    • meson build system
    • cmake build system
    • (“plain”) makefile system (parts only in compat 11+)
    • qmake build system (qt5)
  • Experimental support for cross-building for TARGET rather than HOST (for the about less than 5 source packages in total that might find this useful).
  • Improved bulk performance in various parts of debhelper.
  • Support for “nodoc” profile/build option plus the “terse” build option.
  • Correctly handle systemd units with \x2d (escaped “-“) in their name.
  • Rules-Requires-Root support when the field is set and dpkg-dev notifies debhelper that it supports the feature (requires dpkg-dev from Debian buster).  Besides removing the need for (fake)root it can also remove about 3 invocations of debian/rules.
  • Reduced dbgsym files via dh_dwz (use either manually, with dh –with dwz or dh + compat 12).
  • Enable dh to skip more no-op commands including dh_auto_* and to a minor extend also even when dh is passed arguments that it should pass on to the underlying tools.
  • Support for setting debhelper compat level via debhelper-compat (= X) build-dependency and load dh add-on sequences via dh-sequence-foo build-dependency (as an alternative to the original methods).
  • Support for derivatives and custom/local builds using DH_EXTRA_ADDONS to enable derivative or custom add-ons for debhelper.  Note: Packagers should keep using –with foo or the new dh-sequence-foo build-dependency – this interface is intended to enable a particular add-on without changing the package.
  • Improved maintscript snippet ordering to ensure that service enable + start related snippets always run last in postinst (and first in prerm etc.) in the code inserted via the #DEBHELPER# token.  This ensures that all other scripts (e.g. configuration file management via dh_ucf or the debian/maintscript file) is complete by the time the service (re)starts.
  • Improved “rollback” handling of maintscripts generated by debhelper. Among other, debhelper autoscripts now handle cases like abort-deconfigure and abort-upgrade.  In general, they are handled like configure and replays the setup to ensure that services are correctly running as expected.
  • The autoscript snippet for loading systemd tmpfiles now simply uses the basename of the tmpfiles configuration, which enables the administrator to override the package provided tmpfiles configuration by placing their own in /etc/tmpfiles.d.
  • The new dh_installinitramfs tool now installs maintainer provided initramfs hooks and generates autosnippets for all hooks installed in /usr/share/initramfs-tools/hooks.  Enable via dh –with installinitramfs or dh + compat 12 or call it manually.
  • The new dh_installsystemduser which manages system units per user rather than for the system.  Enable via dh + compat 12 or call it manually.

The above is by no means complete and among other excludes many things that is introduced in compat 11 or compat 12.


Many thanks to the following people, who contributed to debhelper since stretch release with one or more patches (based on git log debian/10.2.5..debian/12 | git shortlog):

Adam Conrad, Américo Monteiro, Axel Beckert, Baptiste Jammet, Chris Lamb, Chris Leick, Christoph Biedl, Clément Hermann, Colin Watson, Daniele Nicolodi, Dmitry Shachnev, Fabian Grünbichler, Fabian Wolff, Felipe Sateler, Geoffrey Thomas, Helmut Grohne, Hideki Yamane, Iain Lane, Isaac Jurado, Jakub Wilk, Johannes Schauer, Josh Triplett, Juhani Numminen, Lisandro Damián Nicanor Pérez Meyer, Luca Boccassi, Mattia Rizzolo, Michael Biebl, Michael Stapelberg, Nicholas Guriev, Nicolas Boulenguez, Niels Thykier, Olly Betts, Paul Tagliamonte, Peter Pentchev, Roberto C. Sánchez, Steven Chamberlain, Sven Joachim, Ville Skyttä, gregor herrmann

Also, many thanks to the people reporting bugs, regressions and feature suggestions via the Debian BTS.

Posted in Debhelper, Debian | Leave a comment

Buster is headed for a long hard freeze

We are getting better and better accumulating RC bugs in testing. This is unfortunate because the length of the freeze is strongly correlated with the number of open RC bugs affecting testing. If you believe that Debian should have short freezes, then it will require putting effort behind that belief and fix some RC bugs – even in packages that are not maintained directly by you or your team and especially in key packages.

The introduction of key packages have been interesting. On the plus side, we can use it to auto-remove RC buggy non-key packages from testing which has been very helpful. On the flip-side, it also makes it painfully obvious that over 50% of all RC bugs in testing are now filed against key packages (for the lazy; we are talking about 475 RC bugs in testing filed against key packages; about 25 of these appear to be fixed in unstable).

Below are some observations from the list of RC bugs in key packages (affecting both testing and unstable – based on a glance over all of the titles).

  • About 85 RC bugs related to (now) defunct maintainer addresses caused by the shutdown of Alioth. From a quick glance, it appears that the Debian Xfce Maintainers has the largest backlog – maybe they could use another team member.  Note they are certainly not the only team with this issue.
  • Over 100 RC bugs are FTBFS for various reasons..  Some of these are related to transitions (e.g. new major versions of GCC, LLVM and OpenJDK).

Those three points alone accounts for 40% of the RC bugs affecting both testing and unstable.

We also have several contributors that want to remove unmaintained, obsolete or old versions of  packages (older versions of compilers such as GCC and LLVM, flash-players/tooling, etc.).  If you are working on this kind of removal, please remember to follow through on it (even if it means NMU packages).  The freeze is not the right time to remove obsolete key packages as it tends to involve non-trivial changes of features or produced binaries.  As much of this as entirely possible ought to be fixed before 2019-01-12 (transition freeze).


In summary: If you want Debian Buster released in early 2019 or short Debian freezes in general, then put your effort where your wish/belief is and fix RC bugs today.  Props for fixes to FTBFS bugs, things that hold back transitions or keep old/unmaintained/unsupportable key packages in Buster (testing).

Posted in Debian, Release-Team | 2 Comments

Build system changes in debhelper

Since debhelper/11.2.1[1], we now support using cmake for configure and ninja for build + test as an alternative to cmake for configure and make for build + test.  This change was proposed by Kyle Edwards in Debian bug #895044. You can try this new combination by specifying “cmake+ninja” as build system.

To facilitate this change, the cmake and meson debhelper buildsystems had to change their (canonical) name.  As such you may have noticed that the “–list” option for dh_auto_* now produces a slightly different output:


$ dh_auto_configure --list | grep +
cmake+makefile       CMake (CMakeLists.txt) combined with simple Makefile
cmake+ninja          CMake (CMakeLists.txt) combined with Ninja (build.ninja)
meson+ninja          Meson (meson.build) combined with Ninja (build.ninja)


You might also notice that “cmake” and “meson” is no longer listed in the full list of build systems.  To retain backwards compatibility, the names “cmake” and “meson” are handled as “cmake+makefile” and “meson+ninja”.  This can be seen if we specify a build system:


$ dh_auto_configure --buildsystem cmake --list | tail -n1
Specified: cmake+makefile (default for cmake)
$ dh_auto_configure --buildsystem cmake+makefile --list | tail -n1
Specified: cmake+makefile
$ dh_auto_configure --buildsystem cmake+ninja --list | tail -n1
Specified: cmake+ninja


If your package uses cmake, please give it a try and see what works and what does not.  So far, the only known issue is that cmake+ninja may fail if the package has no tests while it success with cmake+makefile.  I believe this is because the makefile build system checks whether the “test” or “check” targets exist before calling make.

Enjoy. 🙂



[1] Strictly speaking, it was version 11.2.  However, version 11.2 had a severe regression that made it mostly useless.

Posted in Debhelper, Debian | Leave a comment

Prototyping a new packaging papercut fix – DRY for the debhelper compat level

Have you ever looked at packaging and felt it is a long exercise in repeating yourself?  If you have, you are certainly not alone.  You can find examples of this on the Debian mailing lists (among other places).  Such as when Russ Allbery pointed out that the debhelper compat level vs. the version in the debhelper build-dependency that is very often but not always the same.

Russ suggests two ways of solving the problem:

  1. The first proposal is to generate the build-dependency from the compat file. However, generating the control file is (as Russ puts it) “fraught with peril”.  Probably because we do not have good or standardized tooling for it – creating such tooling and deploying it will take years.  Not to mention that most contributors appear to be uncomfortable with handling the debian/control as a generated file.
  2. The alternative proposal from Russ is to assume that the major version of the build-dependency should mark the compat level (assuming no compat file exist).  However, Russ again points out an issue here that solution might be “too magical”.  Indeed, this solution have the problem that you implicitly change compat level as soon as you bump the versioned dependency beyond a major version of debhelper.  But only if you do not have a compat file.

Looking at these two options, the concept behind second one is most likely to be deployable in the near future.  However, the solution would need some tweaking and I have spend my time coming up with an alternative.

The third alternative:

My current alternative to Russ’s second proposal is to make debhelper provide multiple versions of “debhelper-compat” and have packages use a Build-Depends on “debhelper-compat (= X)”, where X denotes the desired compat level.  The build-dependency will then replace the “debhelper (>= X~)” relation when the package does not require a specific version of debhelper (beyond what is required for the compat level).

On top of this, debhelper implements some safe-guards to ensure that it can reliably determine the compat level from the build-dependencies.  Notably, there must be exactly one debhelper compat relation, it must be in the “Build-Depends” field and it must have a “strictly equal version” as version constraint.  Furthermore, it must not have any Build-Profile or architecture restrictions and so on.


With all of this in place:

  1. We have no repetition when it is not required.
  2. debhelper can reliable auto-detect which debhelper-compat level you wanted.  Otherwise, debhelper will ensure the build fails (or apt/aptitude, if you end up misspelling the package name or using an invalid version).
  3. Bumping the debhelper compat level is still explicit and separate from bumping the debhelper dependency when you need a  feature or bug fix from a later version.

Testing the prototype:

If you want to test the prototype, you can do so in unstable and testing at the moment (caveat: it is an experimental feature and may change or disappear without notice).  However, please note that lintian is completely unaware of this and will spew out several false-positives – including one nonfatal auto-reject, so you will have to apply at least one lintian override.  Also note, I have only added “debhelper-compat” versions for non-deprecated compat levels.  In other words, you will have to use compat 9 or later to test the feature.

You can use “mscgen/0.20-11” as an example for minimum the changes required.  Admittedly, the example cheats and relies on “debhelper-compat (= 10)” implies a “debhelper (>= 11.1.5~alpha1)” as that is the first version with the provides for debhelper-compat.  Going forward, if you need a feature from debhelper that appears in a later version than that, then you will need an explicit “debhelper (>= Y)” relation for that feature on top of the debhelper-compat relation.

Will you remove the support for debian/compat if this prototype works?

I have no immediate plans to remove the debian/compat file even if this prototype is successful.

Will you upload this to stretch-backports?

Yes, although I am waiting for a fix for #889567 to be uploaded to stretch-backports first.

Will this work for backports?

It worked fine on the buildds when I deployed it in experimental and I believe the build-dependency resolution process for experimental is similar (enough) to backports.

Will the third-party debhelper tools need changes to support this?

Probably no; most third-party debhelper tools do not seem to check the compat level directly. Even then, most tools use the “compat” sub from “Debian::Debhelper::Dh_Lib”, which handles all of this automatically.

That said, if you have a third-party tool that wishes or needs to check the debhelper compat level, please file a bug requesting a cross-language API for this and I will happy look at this.

Future work

I am considering to apply this concept to the dh sequence add-ons as well (i.e. the “dh $@ –with foo”).  From my PoV, this is another case needing a DRY fix.  Plus this would also present an opportune method for solving #836699 – though, the hard part for #836699 is actually taming the dh add-on API plus dh’s sequence handling to consistently only affect the “indep” part of the sequences.

Posted in Debhelper, Debian | 1 Comment

Building packages without (fake)root

Turns out that it is surprisingly easy to build most packages without (fake)root.  You just need to basic changes:

  1. A way to set ownership to “root:root” of paths when dpkg-deb –build constructs the binary.
  2. A way to have debhelper not do a bunch of (now) pointless chowns to “root:root”.

The above is sufficient for dpkg, debhelper, lintian, apt-file, mscgen, pbuilder and a long list of other packages that only provide paths owned by “root:root”. Obviously, packages differ and yours might need more tweaks than this (e.g. dh_usrlocal had to change behaviour to support this).

But for me, the best part is that the above is not just some random prototype stuck in two git repos on alioth:

Unfortunately, if you are working with games or core packages like shadow with need for static ownership different from “root:root” (usually with a setuid or setgid bit), then our first implementation does not support your needs at the moment[1].  We are working on a separate way to solve static ownership in a declarative way.


[1] Note regarding “/usr/local”: If your package needs to provide directories there owned by “root:staff” with mode 02775, then dh_usrlocal can handle that. The non-“root:root” ownership here works because the directories are created in a maintainer script run as root during installation.  Unfortunately, it cannot provide different ownership or modes with “R³ != binary-targets” at the moment.


Posted in Debhelper, Debian | Leave a comment

Introducing the debhelper buildlabel prototype for multi-building packages

For most packages, the “dh” short-hand rules (possibly with a few overrides) work great.  It can often auto-detect the buildsystem and handle all the trivial parts.

With one notably exception: What if you need to compile the upstream code twice (or more) with different flags?  This is the case for all source packages building both regular debs and udebs.

In that case, you would previously need to override about 5-6 helpers for this to work at all.  The five dh_auto_* helpers and usually also dh_install (to call it with different –sourcedir for different packages).  This gets even more complex if you want to support Build-Profiles such as “noudeb” and “nodoc”.

The best way to support “nodoc” in debhelper is to move documentation out of dh_install’s config files and use dh_installman, dh_installdocs, and dh_installexamples instead (NB: wait for compat 11 before doing this).  This in turn will mean more overrides with –sourcedir and -p/-N.

And then there is “noudeb”, which currently requires manual handling in debian/rules.  Basically, you need to use make or shell if-statements to conditionally skip the udeb part of the builds.

All of this is needlessly complex.

Improving the situation

In an attempt to make things better, I have made a new prototype feature in debhelper called “buildlabels” in experimental.  The current prototype is designed to deal with part (but not all) of the above problems:

  • It will remove the need for littering your rules file for supporting “noudeb” (and in some cases also other “noX” profiles).
  • It will remove the need for overriding the dh_install* tools just to juggle with –sourcedir and -p/-N.

However, it currently not solve the need for overriding the dh_auto_* tools and I am not sure when/if it will.

The feature relies on being able to relate packages to a given series of calls to dh_auto_*.  In the following example, I will use udebs for the secondary build.  However, this feature is not tied to udebs in any way and can be used any source package that needs to do two or more upstream builds for different packages.

Assume our example source builds the following binary packages:

  • foo
  • libfoo1
  • libfoo-dev
  • foo-udeb
  • libfoo1-udeb

And in the rules file, we would have something like:


    dh_auto_configure -B build-deb -- --with-feature1 --with-feature2
    dh_auto_configure -B build-udeb -- --without-feature1 --without-feature2


What is somewhat obvious to a human is that the first configure line is related to the regular debs and the second configure line is for the udebs.  However, debhelper does not know how to infer this and this is where buildlabels come in.  With buildlabels, you can let debhelper know which packages and builds that belong together.

How to use buildlabels

To use buildlabels, you have to do three things:

  1. Pick a reasonable label name for the secondary build.  In the example, I will use “udeb”.
  2. Add “–buildlabel=$LABEL” to all dh_auto_* calls related to your secondary build.
  3. Tag all packages related to “my-label” with “X-DH-Buildlabel: $LABEL” in debian/control.  (For udeb packages, you may want to add “Build-Profiles: <!noudeb>” while you are at it).

For the example package, we would change the debian/rules snippet to:


    dh_auto_configure -B build-deb -- --with-feature1 --with-feature2
    dh_auto_configure --buildlabel=udeb -B build-udeb -- --without-feature1 --without-feature2


(Remember to update *all* calls to dh_auto_* helpers; the above only lists dh_auto_configure to keep the example short.)  And then add “X-DH-Buildlabel: udeb” in the stanzas for foo-udeb + libfoo1-udeb.

With those two minor changes:

  • debhelper will skip the calls to dh_auto_* with –buildlabel=udeb if the udeb packages are skipped.
  • dh_auto_install will automatically pick a separate destination directory by default for the udeb build (assuming you do not explicitly override it with –destdir).
  • dh_install will now automatically pick up files from the destination directory.that dh_auto_install used for the given package (even if you overwrote it with –destdir).  Note that you have to remove any use of “–sourcedir” first as this disables the auto-detection.  This also works for other dh_install* tools supporting –sourcedir in compat 11 or later.

Real example

Thanks to Michael Biebl, I was able to make an branch in the systemd git repository to play with this feature.  Therefore I have an real example to use as a show case.  The gist of it is in the following three commits:

Full branch can be seen at: https://anonscm.debian.org/git/pkg-systemd/systemd.git/log/?h=wip-dh-prototype-smarter-multi-builds

Request for comments / call for testing

This prototype is now in experimental (debhelper/10.7+exp.buildlabels) and you are very welcome to take it for a spin.  Please let me know if you find the idea useful and feel free to file bugs or feature requests.  If deemed useful, I will merge into master and include in a future release.

If you have any questions or comments about the feature or need help with trying it out, you are also very welcome to mail the debhelper-devel mailing list.

Known issues / the fine print:

  • It is experimental feature and may change without notice.
  • The current prototype may break existing packages as it is not guarded by a compat bump to ease your testing.  I am still very curious to hear about any issues you may experience.
  • The default build directory is re-used even with different buildlabels, so you still need to use explicit build dirs for buildsystems that prefer building in a separate directory (e.g. meson).
  • udebs are not automatically tagged with an “udeb” buildlabel.  This is partly by design as some source packages only build udebs (and no regular debs).  If they were automatically tagged, the existing packages would break.
  • Label names are used in path names, so you may want to refrain from using “too exciting” label names.
  • It is experimental feature and may change without notice. (Yes, I thought it deserved repeating)
Posted in Debhelper, Debian | Leave a comment

Improving bulk performance in debhelper

Since debhelper/10.3, there has been a number of performance related changes.  The vast majority primarily improves bulk performance or only have visible effects at larger “input” sizes.

Most visible cases are:

  • dh + dh_* now scales a lot better for large number of binary packages.  Even more so with parallel builds.
  • Most dh_* tools are now a lot faster when creating many directories or installing files.
  • dh_prep and dh_clean now bulk removals.
  • dh_install can now bulk some installations.  For a concrete corner-case, libssl-doc went from approximately 11 seconds to less than a second.  This optimization is implicitly disabled with –exclude (among other).
  • dh_installman now scales a lot better with many manpages.  Even more so with parallel builds.
  • dh_installman has restored its performance under fakeroot (regression since 10.2.2)


For debhelper, this mostly involved:

  • avoiding fork+exec of commands for things doable natively in perl.  Especially, when each fork+exec only process one file or dir.
  • bulking as many files/dirs into the call as possible, where fork+exec is still used.
  • caching / memorizing slow calls (e.g. in parts of pkgfile inside Dh_Lib)
  • adding an internal API for dh to do bulk check for pkgfiles. This is useful for dh when checking if it should optimize out a helper.
  • and, of course, doing things in parallel where trivially possible.


How to take advantage of these improvements in tools that use Dh_Lib:

  • If you use install_{file,prog,lib,dir}, then it will come out of the box.  These functions are available in Debian/stable.  On a related note, if you use “doit” to call “install” (or “mkdir”), then please consider migrating to these functions instead.
  • If you need to reset owner+mode (chown 0:0 FILE + chmod MODE FILE), consider using reset_perm_and_owner.  This is also available in Debian/stable.
    • CAVEAT: It is not recursive and YMMV if you do not need the chown call (due to fakeroot).
  • If you have a lot of items to be processed by a external tool, consider using xargs().  Since 10.5.1, it is now possible to insert the items anywhere in the command rather than just in the end.
  • If you need to remove files, consider using the new rm_files function.  It removes files and silently ignores if a file does not exist. It is also available since 10.5.1.
  • If you need to create symlinks, please consider using make_symlink (available in Debian/stable) or make_symlink_raw_target (since 10.5.1).  The former creates policy compliant symlinks (e.g. fixup absolute symlinks that should have been relative).  The latter is closer to a “ln -s” call.
  • If you need to rename a file, please consider using rename_path (since 10.5).  It behaves mostly like “mv -f” but requires dest to be a (non-existing) file.
  • Have a look at whether on_pkgs_in_parallel() / on_items_in_parallel() would be suitable for enabling parallelization in your tool.
    • The emphasis for these functions is on making parallelization easy to add with minimal code changes.  It pre-distributes the items which can lead to unbalanced workloads, where some processes are idle while a few keeps working.


I would like to thank the following for reporting performance issues, regressions or/and providing patches.  The list is in no particular order:

  • Helmut Grohne
  • Kurt Roeckx
  • Gianfranco Costamagna
  • Iain Lane
  • Sven Joachim
  • Adrian Bunk
  • Michael Stapelberg

Should I have missed your contribution, please do not hesitate to let me know.


Posted in Debhelper, Debian | 1 Comment