Buster is headed for a long hard freeze

We are getting better and better accumulating RC bugs in testing. This is unfortunate because the length of the freeze is strongly correlated with the number of open RC bugs affecting testing. If you believe that Debian should have short freezes, then it will require putting effort behind that belief and fix some RC bugs – even in packages that are not maintained directly by you or your team and especially in key packages.

The introduction of key packages have been interesting. On the plus side, we can use it to auto-remove RC buggy non-key packages from testing which has been very helpful. On the flip-side, it also makes it painfully obvious that over 50% of all RC bugs in testing are now filed against key packages (for the lazy; we are talking about 475 RC bugs in testing filed against key packages; about 25 of these appear to be fixed in unstable).

Below are some observations from the list of RC bugs in key packages (affecting both testing and unstable – based on a glance over all of the titles).

  • About 85 RC bugs related to (now) defunct maintainer addresses caused by the shutdown of Alioth. From a quick glance, it appears that the Debian Xfce Maintainers has the largest backlog – maybe they could use another team member.  Note they are certainly not the only team with this issue.
  • Over 100 RC bugs are FTBFS for various reasons..  Some of these are related to transitions (e.g. new major versions of GCC, LLVM and OpenJDK).

Those three points alone accounts for 40% of the RC bugs affecting both testing and unstable.

We also have several contributors that want to remove unmaintained, obsolete or old versions of  packages (older versions of compilers such as GCC and LLVM, flash-players/tooling, etc.).  If you are working on this kind of removal, please remember to follow through on it (even if it means NMU packages).  The freeze is not the right time to remove obsolete key packages as it tends to involve non-trivial changes of features or produced binaries.  As much of this as entirely possible ought to be fixed before 2019-01-12 (transition freeze).

 

In summary: If you want Debian Buster released in early 2019 or short Debian freezes in general, then put your effort where your wish/belief is and fix RC bugs today.  Props for fixes to FTBFS bugs, things that hold back transitions or keep old/unmaintained/unsupportable key packages in Buster (testing).

Advertisements
Posted in Debian, Release-Team | 2 Comments

Build system changes in debhelper

Since debhelper/11.2.1[1], we now support using cmake for configure and ninja for build + test as an alternative to cmake for configure and make for build + test.  This change was proposed by Kyle Edwards in Debian bug #895044. You can try this new combination by specifying “cmake+ninja” as build system.

To facilitate this change, the cmake and meson debhelper buildsystems had to change their (canonical) name.  As such you may have noticed that the “–list” option for dh_auto_* now produces a slightly different output:

 

$ dh_auto_configure --list | grep +
cmake+makefile       CMake (CMakeLists.txt) combined with simple Makefile
cmake+ninja          CMake (CMakeLists.txt) combined with Ninja (build.ninja)
meson+ninja          Meson (meson.build) combined with Ninja (build.ninja)

 

You might also notice that “cmake” and “meson” is no longer listed in the full list of build systems.  To retain backwards compatibility, the names “cmake” and “meson” are handled as “cmake+makefile” and “meson+ninja”.  This can be seen if we specify a build system:

 

$ dh_auto_configure --buildsystem cmake --list | tail -n1
Specified: cmake+makefile (default for cmake)
$ dh_auto_configure --buildsystem cmake+makefile --list | tail -n1
Specified: cmake+makefile
$ dh_auto_configure --buildsystem cmake+ninja --list | tail -n1
Specified: cmake+ninja

 

If your package uses cmake, please give it a try and see what works and what does not.  So far, the only known issue is that cmake+ninja may fail if the package has no tests while it success with cmake+makefile.  I believe this is because the makefile build system checks whether the “test” or “check” targets exist before calling make.

Enjoy. 🙂

 

Footnotes:

[1] Strictly speaking, it was version 11.2.  However, version 11.2 had a severe regression that made it mostly useless.

Posted in Debhelper, Debian | Leave a comment

Prototyping a new packaging papercut fix – DRY for the debhelper compat level

Have you ever looked at packaging and felt it is a long exercise in repeating yourself?  If you have, you are certainly not alone.  You can find examples of this on the Debian mailing lists (among other places).  Such as when Russ Allbery pointed out that the debhelper compat level vs. the version in the debhelper build-dependency that is very often but not always the same.

Russ suggests two ways of solving the problem:

  1. The first proposal is to generate the build-dependency from the compat file. However, generating the control file is (as Russ puts it) “fraught with peril”.  Probably because we do not have good or standardized tooling for it – creating such tooling and deploying it will take years.  Not to mention that most contributors appear to be uncomfortable with handling the debian/control as a generated file.
  2. The alternative proposal from Russ is to assume that the major version of the build-dependency should mark the compat level (assuming no compat file exist).  However, Russ again points out an issue here that solution might be “too magical”.  Indeed, this solution have the problem that you implicitly change compat level as soon as you bump the versioned dependency beyond a major version of debhelper.  But only if you do not have a compat file.

Looking at these two options, the concept behind second one is most likely to be deployable in the near future.  However, the solution would need some tweaking and I have spend my time coming up with an alternative.

The third alternative:

My current alternative to Russ’s second proposal is to make debhelper provide multiple versions of “debhelper-compat” and have packages use a Build-Depends on “debhelper-compat (= X)”, where X denotes the desired compat level.  The build-dependency will then replace the “debhelper (>= X~)” relation when the package does not require a specific version of debhelper (beyond what is required for the compat level).

On top of this, debhelper implements some safe-guards to ensure that it can reliably determine the compat level from the build-dependencies.  Notably, there must be exactly one debhelper compat relation, it must be in the “Build-Depends” field and it must have a “strictly equal version” as version constraint.  Furthermore, it must not have any Build-Profile or architecture restrictions and so on.

 

With all of this in place:

  1. We have no repetition when it is not required.
  2. debhelper can reliable auto-detect which debhelper-compat level you wanted.  Otherwise, debhelper will ensure the build fails (or apt/aptitude, if you end up misspelling the package name or using an invalid version).
  3. Bumping the debhelper compat level is still explicit and separate from bumping the debhelper dependency when you need a  feature or bug fix from a later version.

Testing the prototype:

If you want to test the prototype, you can do so in unstable and testing at the moment (caveat: it is an experimental feature and may change or disappear without notice).  However, please note that lintian is completely unaware of this and will spew out several false-positives – including one nonfatal auto-reject, so you will have to apply at least one lintian override.  Also note, I have only added “debhelper-compat” versions for non-deprecated compat levels.  In other words, you will have to use compat 9 or later to test the feature.

You can use “mscgen/0.20-11” as an example for minimum the changes required.  Admittedly, the example cheats and relies on “debhelper-compat (= 10)” implies a “debhelper (>= 11.1.5~alpha1)” as that is the first version with the provides for debhelper-compat.  Going forward, if you need a feature from debhelper that appears in a later version than that, then you will need an explicit “debhelper (>= Y)” relation for that feature on top of the debhelper-compat relation.

Will you remove the support for debian/compat if this prototype works?

I have no immediate plans to remove the debian/compat file even if this prototype is successful.

Will you upload this to stretch-backports?

Yes, although I am waiting for a fix for #889567 to be uploaded to stretch-backports first.

Will this work for backports?

It worked fine on the buildds when I deployed it in experimental and I believe the build-dependency resolution process for experimental is similar (enough) to backports.

Will the third-party debhelper tools need changes to support this?

Probably no; most third-party debhelper tools do not seem to check the compat level directly. Even then, most tools use the “compat” sub from “Debian::Debhelper::Dh_Lib”, which handles all of this automatically.

That said, if you have a third-party tool that wishes or needs to check the debhelper compat level, please file a bug requesting a cross-language API for this and I will happy look at this.

Future work

I am considering to apply this concept to the dh sequence add-ons as well (i.e. the “dh $@ –with foo”).  From my PoV, this is another case needing a DRY fix.  Plus this would also present an opportune method for solving #836699 – though, the hard part for #836699 is actually taming the dh add-on API plus dh’s sequence handling to consistently only affect the “indep” part of the sequences.

Posted in Debhelper, Debian | Leave a comment

Building packages without (fake)root

Turns out that it is surprisingly easy to build most packages without (fake)root.  You just need to basic changes:

  1. A way to set ownership to “root:root” of paths when dpkg-deb –build constructs the binary.
  2. A way to have debhelper not do a bunch of (now) pointless chowns to “root:root”.

The above is sufficient for dpkg, debhelper, lintian, apt-file, mscgen, pbuilder and a long list of other packages that only provide paths owned by “root:root”. Obviously, packages differ and yours might need more tweaks than this (e.g. dh_usrlocal had to change behaviour to support this).

But for me, the best part is that the above is not just some random prototype stuck in two git repos on alioth:

Unfortunately, if you are working with games or core packages like shadow with need for static ownership different from “root:root” (usually with a setuid or setgid bit), then our first implementation does not support your needs at the moment[1].  We are working on a separate way to solve static ownership in a declarative way.

 

[1] Note regarding “/usr/local”: If your package needs to provide directories there owned by “root:staff” with mode 02775, then dh_usrlocal can handle that. The non-“root:root” ownership here works because the directories are created in a maintainer script run as root during installation.  Unfortunately, it cannot provide different ownership or modes with “R³ != binary-targets” at the moment.

 

Posted in Debhelper, Debian | Leave a comment

Introducing the debhelper buildlabel prototype for multi-building packages

For most packages, the “dh” short-hand rules (possibly with a few overrides) work great.  It can often auto-detect the buildsystem and handle all the trivial parts.

With one notably exception: What if you need to compile the upstream code twice (or more) with different flags?  This is the case for all source packages building both regular debs and udebs.

In that case, you would previously need to override about 5-6 helpers for this to work at all.  The five dh_auto_* helpers and usually also dh_install (to call it with different –sourcedir for different packages).  This gets even more complex if you want to support Build-Profiles such as “noudeb” and “nodoc”.

The best way to support “nodoc” in debhelper is to move documentation out of dh_install’s config files and use dh_installman, dh_installdocs, and dh_installexamples instead (NB: wait for compat 11 before doing this).  This in turn will mean more overrides with –sourcedir and -p/-N.

And then there is “noudeb”, which currently requires manual handling in debian/rules.  Basically, you need to use make or shell if-statements to conditionally skip the udeb part of the builds.

All of this is needlessly complex.

Improving the situation

In an attempt to make things better, I have made a new prototype feature in debhelper called “buildlabels” in experimental.  The current prototype is designed to deal with part (but not all) of the above problems:

  • It will remove the need for littering your rules file for supporting “noudeb” (and in some cases also other “noX” profiles).
  • It will remove the need for overriding the dh_install* tools just to juggle with –sourcedir and -p/-N.

However, it currently not solve the need for overriding the dh_auto_* tools and I am not sure when/if it will.

The feature relies on being able to relate packages to a given series of calls to dh_auto_*.  In the following example, I will use udebs for the secondary build.  However, this feature is not tied to udebs in any way and can be used any source package that needs to do two or more upstream builds for different packages.

Assume our example source builds the following binary packages:

  • foo
  • libfoo1
  • libfoo-dev
  • foo-udeb
  • libfoo1-udeb

And in the rules file, we would have something like:

[...]

override_dh_auto_configure:
    dh_auto_configure -B build-deb -- --with-feature1 --with-feature2
    dh_auto_configure -B build-udeb -- --without-feature1 --without-feature2

[...]

What is somewhat obvious to a human is that the first configure line is related to the regular debs and the second configure line is for the udebs.  However, debhelper does not know how to infer this and this is where buildlabels come in.  With buildlabels, you can let debhelper know which packages and builds that belong together.

How to use buildlabels

To use buildlabels, you have to do three things:

  1. Pick a reasonable label name for the secondary build.  In the example, I will use “udeb”.
  2. Add “–buildlabel=$LABEL” to all dh_auto_* calls related to your secondary build.
  3. Tag all packages related to “my-label” with “X-DH-Buildlabel: $LABEL” in debian/control.  (For udeb packages, you may want to add “Build-Profiles: <!noudeb>” while you are at it).

For the example package, we would change the debian/rules snippet to:

[...]

override_dh_auto_configure:
    dh_auto_configure -B build-deb -- --with-feature1 --with-feature2
    dh_auto_configure --buildlabel=udeb -B build-udeb -- --without-feature1 --without-feature2

[...]

(Remember to update *all* calls to dh_auto_* helpers; the above only lists dh_auto_configure to keep the example short.)  And then add “X-DH-Buildlabel: udeb” in the stanzas for foo-udeb + libfoo1-udeb.

With those two minor changes:

  • debhelper will skip the calls to dh_auto_* with –buildlabel=udeb if the udeb packages are skipped.
  • dh_auto_install will automatically pick a separate destination directory by default for the udeb build (assuming you do not explicitly override it with –destdir).
  • dh_install will now automatically pick up files from the destination directory.that dh_auto_install used for the given package (even if you overwrote it with –destdir).  Note that you have to remove any use of “–sourcedir” first as this disables the auto-detection.  This also works for other dh_install* tools supporting –sourcedir in compat 11 or later.

Real example

Thanks to Michael Biebl, I was able to make an branch in the systemd git repository to play with this feature.  Therefore I have an real example to use as a show case.  The gist of it is in the following three commits:

Full branch can be seen at: https://anonscm.debian.org/git/pkg-systemd/systemd.git/log/?h=wip-dh-prototype-smarter-multi-builds

Request for comments / call for testing

This prototype is now in experimental (debhelper/10.7+exp.buildlabels) and you are very welcome to take it for a spin.  Please let me know if you find the idea useful and feel free to file bugs or feature requests.  If deemed useful, I will merge into master and include in a future release.

If you have any questions or comments about the feature or need help with trying it out, you are also very welcome to mail the debhelper-devel mailing list.

Known issues / the fine print:

  • It is experimental feature and may change without notice.
  • The current prototype may break existing packages as it is not guarded by a compat bump to ease your testing.  I am still very curious to hear about any issues you may experience.
  • The default build directory is re-used even with different buildlabels, so you still need to use explicit build dirs for buildsystems that prefer building in a separate directory (e.g. meson).
  • udebs are not automatically tagged with an “udeb” buildlabel.  This is partly by design as some source packages only build udebs (and no regular debs).  If they were automatically tagged, the existing packages would break.
  • Label names are used in path names, so you may want to refrain from using “too exciting” label names.
  • It is experimental feature and may change without notice. (Yes, I thought it deserved repeating)
Posted in Debhelper, Debian | Leave a comment

Improving bulk performance in debhelper

Since debhelper/10.3, there has been a number of performance related changes.  The vast majority primarily improves bulk performance or only have visible effects at larger “input” sizes.

Most visible cases are:

  • dh + dh_* now scales a lot better for large number of binary packages.  Even more so with parallel builds.
  • Most dh_* tools are now a lot faster when creating many directories or installing files.
  • dh_prep and dh_clean now bulk removals.
  • dh_install can now bulk some installations.  For a concrete corner-case, libssl-doc went from approximately 11 seconds to less than a second.  This optimization is implicitly disabled with –exclude (among other).
  • dh_installman now scales a lot better with many manpages.  Even more so with parallel builds.
  • dh_installman has restored its performance under fakeroot (regression since 10.2.2)

 

For debhelper, this mostly involved:

  • avoiding fork+exec of commands for things doable natively in perl.  Especially, when each fork+exec only process one file or dir.
  • bulking as many files/dirs into the call as possible, where fork+exec is still used.
  • caching / memorizing slow calls (e.g. in parts of pkgfile inside Dh_Lib)
  • adding an internal API for dh to do bulk check for pkgfiles. This is useful for dh when checking if it should optimize out a helper.
  • and, of course, doing things in parallel where trivially possible.

 

How to take advantage of these improvements in tools that use Dh_Lib:

  • If you use install_{file,prog,lib,dir}, then it will come out of the box.  These functions are available in Debian/stable.  On a related note, if you use “doit” to call “install” (or “mkdir”), then please consider migrating to these functions instead.
  • If you need to reset owner+mode (chown 0:0 FILE + chmod MODE FILE), consider using reset_perm_and_owner.  This is also available in Debian/stable.
    • CAVEAT: It is not recursive and YMMV if you do not need the chown call (due to fakeroot).
  • If you have a lot of items to be processed by a external tool, consider using xargs().  Since 10.5.1, it is now possible to insert the items anywhere in the command rather than just in the end.
  • If you need to remove files, consider using the new rm_files function.  It removes files and silently ignores if a file does not exist. It is also available since 10.5.1.
  • If you need to create symlinks, please consider using make_symlink (available in Debian/stable) or make_symlink_raw_target (since 10.5.1).  The former creates policy compliant symlinks (e.g. fixup absolute symlinks that should have been relative).  The latter is closer to a “ln -s” call.
  • If you need to rename a file, please consider using rename_path (since 10.5).  It behaves mostly like “mv -f” but requires dest to be a (non-existing) file.
  • Have a look at whether on_pkgs_in_parallel() / on_items_in_parallel() would be suitable for enabling parallelization in your tool.
    • The emphasis for these functions is on making parallelization easy to add with minimal code changes.  It pre-distributes the items which can lead to unbalanced workloads, where some processes are idle while a few keeps working.

Credits:

I would like to thank the following for reporting performance issues, regressions or/and providing patches.  The list is in no particular order:

  • Helmut Grohne
  • Kurt Roeckx
  • Gianfranco Costamagna
  • Iain Lane
  • Sven Joachim
  • Adrian Bunk
  • Michael Stapelberg

Should I have missed your contribution, please do not hesitate to let me know.

 

Posted in Debhelper, Debian | Leave a comment

Approaching the exclusive “sub-minute” build time club

For the first time in at least two years (and probably even longer), debhelper with the 10.6.2 upload broke the 1 minute milestone for build time (by mere 2 seconds – look for “Build needed 00:00:58, […]”).  Sadly, the result it is not deterministic and the 10.6.3 upload needed 1m + 5s to complete on the buildds.

This is not the result of any optimizations I have done in debhelper itself.  Instead, it is the result of “questionable use of developer time” for the sake of meeting an arbitrary milestone. Basically, I made it possible to parallelize more of the debhelper build (10.6.1) and finally made it possible to run the tests in parallel (10.6.2).

In 10.6.2, I also made the most of the tests run against all relevant compat levels.  Previously, it would only run the tests against one compat level (either the current one or a hard-coded older version).

Testing more than one compat turned out to be fairly simple given a proper test library (I wrote a “Test::DH” module for the occasion).  Below is an example, which is the new test case that I wrote for Debian bug #866570.

$ cat t/dh_install/03-866570-dont-install-from-host.t
#!/usr/bin/perl
use strict;
use warnings;
use Test::More;

use File::Basename qw(dirname);
use lib dirname(dirname(__FILE__));
use Test::DH;
use File::Path qw(remove_tree make_path);
use Debian::Debhelper::Dh_Lib qw(!dirname);

plan(tests => 1);

each_compat_subtest {
  my ($compat) = @_;
  # #866570 - leading slashes must *not* pull things from the root FS.
  make_path('bin');
  create_empty_file('bin/grep-i-licious');
  ok(run_dh_tool('dh_install', '/bin/grep*'));
  ok(-e "debian/debhelper/bin/grep-i-licious", "#866570 [${compat}]");
  ok(!-e "debian/debhelper/bin/grep", "#866570 [${compat}]");
  remove_tree('debian/debhelper', 'debian/tmp');
};

I have cheated a bit on the implementation; while the test runs in a temporary directory, the directory is reused between compat levels (accordingly, there is a “clean up” step at the end of the test).

If you want debhelper to maintain this exclusive (and somewhat arbitrary) property (deterministically), you are more than welcome to help me improve the Makefile. 🙂  I am not sure I can squeeze any more out of it with my (lack of) GNU make skills.

Posted in Debhelper, Debian | 8 Comments