Approaching the exclusive “sub-minute” build time club

For the first time in at least two years (and probably even longer), debhelper with the 10.6.2 upload broke the 1 minute milestone for build time (by mere 2 seconds – look for “Build needed 00:00:58, […]”).  Sadly, the result it is not deterministic and the 10.6.3 upload needed 1m + 5s to complete on the buildds.

This is not the result of any optimizations I have done in debhelper itself.  Instead, it is the result of “questionable use of developer time” for the sake of meeting an arbitrary milestone. Basically, I made it possible to parallelize more of the debhelper build (10.6.1) and finally made it possible to run the tests in parallel (10.6.2).

In 10.6.2, I also made the most of the tests run against all relevant compat levels.  Previously, it would only run the tests against one compat level (either the current one or a hard-coded older version).

Testing more than one compat turned out to be fairly simple given a proper test library (I wrote a “Test::DH” module for the occasion).  Below is an example, which is the new test case that I wrote for Debian bug #866570.

$ cat t/dh_install/03-866570-dont-install-from-host.t
#!/usr/bin/perl
use strict;
use warnings;
use Test::More;

use File::Basename qw(dirname);
use lib dirname(dirname(__FILE__));
use Test::DH;
use File::Path qw(remove_tree make_path);
use Debian::Debhelper::Dh_Lib qw(!dirname);

plan(tests => 1);

each_compat_subtest {
  my ($compat) = @_;
  # #866570 - leading slashes must *not* pull things from the root FS.
  make_path('bin');
  create_empty_file('bin/grep-i-licious');
  ok(run_dh_tool('dh_install', '/bin/grep*'));
  ok(-e "debian/debhelper/bin/grep-i-licious", "#866570 [${compat}]");
  ok(!-e "debian/debhelper/bin/grep", "#866570 [${compat}]");
  remove_tree('debian/debhelper', 'debian/tmp');
};

I have cheated a bit on the implementation; while the test runs in a temporary directory, the directory is reused between compat levels (accordingly, there is a “clean up” step at the end of the test).

If you want debhelper to maintain this exclusive (and somewhat arbitrary) property (deterministically), you are more than welcome to help me improve the Makefile. 🙂  I am not sure I can squeeze any more out of it with my (lack of) GNU make skills.

Posted in Debhelper, Debian | 8 Comments

debhelper 10.5.1 now available in unstable

Earlier today, I uploaded debhelper version 10.5.1 to unstable.  The following are some highlights compared to version 10.2.5:

  • debhelper now supports the “meson+ninja” build system. Kudos to Michael Biebl.
  • Better cross building support in the “makefile” build system (PKG_CONFIG is set to the multi-arched version of pkg-config). Kudos to Helmut Grohne.
  • New dh_missing helper to take over dh_install –list-missing/–fail-missing while being able to see files installed from other helpers. Kudos to Michael Stapelberg.
  • dh_installman now logs what files it has installed so the new dh_missing helper can “see” them as installed.
  • Improve documentation (e.g. compare and contrast the dh_link config file with ln(1) to assist people who are familiar with ln(1))
  • Avoid triggering a race-condition with libtool by ensuring that dh_auto_install run make with -j1 when libtool is detected (see Debian bug #861627)
  • Optimizations and parallel processing (more on this later)

There are also some changes to the upcoming compat 11

  • Use “/run” as “run state dir” for autoconf
  • dh_installman will now guess the language of a manpage from the path name before using the extension.

 

Posted in Debhelper, Debian | Leave a comment

On making Britney smarter

Updating Britney often makes our life easier. Like:

Concretely, transitions have become a lot easier.  When I joined the release team in the summer 2011, about the worst thing that could happen was discovering that two transitions had become entangled. You would have to wait for everything to be ready to migrate at the same time and then you usually also had to tell Britney what had to migrate together.

Today, Britney will often (but not always) de-tangle the transitions on her own and very often figure out how to migrate packages without help. The latter is in fact very visible if you know where to look.  Behold, the number of manual “easy” and “hint”-hints by RT members per year[2]:

Year | Total | easy | hint
-----+-------+------+-----
2005 |   53  |   30 |  23 
2006 |  146  |   74 |  72
2007 |   70  |   40 |  30
2008 |  113  |   68 |  45
2009 |  229  |  171 |  58
2010 |  252  |  159 |  93
2011 |  255  |  118 | 137
2012 |   29  |   21 |   8
2013 |   36  |   30 |   6
2014 |   20  |   20 |   0
2015 |   25  |   17 |   8
2016 |   16  |   11 |   5
2017 |    1  |    1 |   0

As can be seen, the number of manual hints drop by factor of ~8.8 between 2011 and 2012. Now, I have not actually done a proper statistical test of the data, but I have a hunch that drop was “significant” (see also [3] for a very short data discussion).

 

In conclusion: Smooth-updates (which was enabled late in 2011) have been a tremendous success. 🙂

 

[1] A very surprising side-effect of that commit was that the (“original”) auto-hinter could now solve a complicated haskell transition. Turns out that it works a lot better, when you give correct information! 🙂

[2] As extracted by the following script and then manually massaged into an ASCII table. Tweak the in-line regex to see different hints.

respighi.d.o$ cd "/home/release/britney/hints" && perl -E '
    my (%years, %hints);
    while(<>) { 
        chomp;
        if (m/^\#\s*(\d{4})(?:-?\d{2}-?\d{2});/ or m/^\#\s*(?:\d+-\d+-\d+\s*[;:]?\s*)?done\s*[;:]?\s*(\d{4})(?:-?\d{2}-?\d{2})/) {
             $year = $1; next;
         }
         if (m/^((?:easy|hint) .*)/) {
             my $hint = $1; $years{$year}++ if defined($year) and not $hints{$hint}++;
             next;
         }
         if (m/^\s*$/) { $year = undef; next; }
    };
    for my $year (sort(keys(%years))) { 
        my $count = $years{$year};
        print "$year: $count\n"
    }' * OLD/jessie/* OLD/wheezy/* OLD/Lenny/* OLD/*

[3]  I should probably mention for good measure that extraction is ignoring all hints where it cannot figure out what year it was from or if it is a duplicate.  Notable it is omitting about 100 easy/hint-hints from “OLD/Lenny” (compared to a grep -c), which I think accounts for the low numbers from 2007 (among other).

Furthermore, hints files are not rotated based on year or age, nor am I sure we still have all complete hints files from all members.

Posted in Debian, Release-Team | Leave a comment

The stretch freeze is coming

The soft freeze has been on going for almost a month now and the full stretch freeze will start tomorrow night (UTC).  It has definitely been visible in the number of unblock requests that we have received so far.  Fortunately, we are no where near the rate of the jessie freeze.  At the moment, all unblock requests are waiting for the submitter (either for a clarification or an upload).

Looking at stretch at a glance (items are in no particular order):

Secure boot support

Currently, we are blocked on two items:

  • We do not have signing done yet for the boot packages (not even manual signing).
  • Our shim is not yet signed, so no hardware would be trusting our boot chain out of the box.

After they are done, we are missing a handful of uploads to provide a signed bootloader etc. plus d-i and some infrastructure bits need to be updated. At the moment, we are waiting for a handful of key people/organisations to move on their part. As such, there is not a lot you can do to assist here (unless you are already involved in the work).
On the flip side, if both of these items are resolved soon, there is a good chance that we can support secure boot in stretch.See bug#820036 and blockers for more information on the remaining items.

Where can you help with the release?

At the moment, the best you can do is to:

  • Test (packages, upgrades, etc.) and report bugs
  • File bugs against release-notes for issues that should be documented
  • Fix RC bugs (please see the next section)

 

Release Critical Bug report

The UDD bugs interface currently knows about the following release critical bugs:

  • In Total: 1148 (Including 193 bugs affecting key packages)
    • Affecting stretch: 294 (key packages: 158)
      That’s the number we need to get down to zero before the release. They can be split in two big categories:

      • Affecting stretch and unstable: 232 (key packages: 134)
        Those need someone to find a fix, or to finish the work to upload a fix to unstable:

        • 30 bugs are tagged ‘patch’. (key packages: 21)
          Please help by reviewing the patches, and (if you are a DD) by uploading them.
        • 17 bugs are marked as done, but still affect unstable. (key packages: 5)
          This can happen due to missing builds on some architectures, for example. Help investigate!
        • 185 bugs are neither tagged patch, nor marked done. (key packages: 108)
          Help make a first step towards resolution!
      • Affecting stretch only: 62 (key packages: 24)
        Those are already fixed in unstable, but the fix still needs to migrate to stretch. You can help by submitting unblock requests for fixed packages, by investigating why packages do not migrate, or by reviewing submitted unblock requests.

        • 36 bugs are in packages that are unblocked by the release team. (key packages: 14)
        • 26 bugs are in packages that are not unblocked. (key packages: 10)
Posted in Debian, Release-Team | 7 Comments

Piuparts integration in britney

As of today, britney now fetches reports from piuparts.debian.org and uses it as a part of her evaluation for package migration.  As with her RC bug check, we are only preventing (known) regressions from migrating.

The messages (subject to change) look something like:

  • Piuparts tested OK
  • Rejected due to piuparts regression
  • Ignoring piuparts failure (Not a regression)
  • Cannot be tested by piuparts (not a blocker)

If you want to do machine parsing of the Britney excuses, we also provide an excuses.yaml. In there, you are looking for “excuses[X].policy_info.piuparts.test-results”, which will be one of:

  • pass
  • regression
  • failed
  • cannot-be-tested

Enjoy. 🙂

 

Posted in Debian, Release-Team | Leave a comment

Improvements in apt-file 3.1.2

Yesterday, I just uploaded apt-file 3.1.2 into unstable, which comes with a few things I would like to highlight.

 

  • We fixed an issue where apt-file would not show top-level files in source packages. (bug#676642). Thanks to Paul Wise for the proposed solution.
  • Paul Wise also fixed a bug where apt-file list -I dsc <source-pkg> would fail to list all files in the source package if said file was also in other packages.
  • We added –filter-suites / –filter-origins options that can be used to narrow the search space.  Example: apt-file search --filter-suites unstable lintian/checks/

You can also set defaults in the config file – if you want to always search in unstable, simply do:

# echo 'apt-file::Search-Filter::Suite "unstable";' >> /etc/apt/apt-file.conf

For the suite filter, either a code name (“sid”) or a suite name (“unstable”) will work.  Please note that the filters are case-sensitive – suites/code names generally use all lowercase, whereas origins appear to use title-case (i.e. “unstable” vs. “Debian”).

 

Posted in apt-file, Debian | 1 Comment

Stretch transition freeze in a month

It is the first of October and that means the transition freeze is roughly one month away (Nov 5th 2016). In other words, this is the “final boarding call for transitions”.

 

Other milestone dates:

  • If you want new packages into Stretch, you have about 3 months (Jan 5th 2017).  This includes auto removed packages.
  • Automatic updates of existing packages stop in 4 months (Feb 5th 2017)

 

Posted in Debian, Release-Team | Leave a comment