On making Britney smarter

Updating Britney often makes our life easier. Like:

Concretely, transitions have become a lot easier.  When I joined the release team in the summer 2011, about the worst thing that could happen was discovering that two transitions had become entangled. You would have to wait for everything to be ready to migrate at the same time and then you usually also had to tell Britney what had to migrate together.

Today, Britney will often (but not always) de-tangle the transitions on her own and very often figure out how to migrate packages without help. The latter is in fact very visible if you know where to look.  Behold, the number of manual “easy” and “hint”-hints by RT members per year[2]:

Year | Total | easy | hint
-----+-------+------+-----
2005 |   53  |   30 |  23 
2006 |  146  |   74 |  72
2007 |   70  |   40 |  30
2008 |  113  |   68 |  45
2009 |  229  |  171 |  58
2010 |  252  |  159 |  93
2011 |  255  |  118 | 137
2012 |   29  |   21 |   8
2013 |   36  |   30 |   6
2014 |   20  |   20 |   0
2015 |   25  |   17 |   8
2016 |   16  |   11 |   5
2017 |    1  |    1 |   0

As can be seen, the number of manual hints drop by factor of ~8.8 between 2011 and 2012. Now, I have not actually done a proper statistical test of the data, but I have a hunch that drop was “significant” (see also [3] for a very short data discussion).

 

In conclusion: Smooth-updates (which was enabled late in 2011) have been a tremendous success. 🙂

 

[1] A very surprising side-effect of that commit was that the (“original”) auto-hinter could now solve a complicated haskell transition. Turns out that it works a lot better, when you give correct information! 🙂

[2] As extracted by the following script and then manually massaged into an ASCII table. Tweak the in-line regex to see different hints.

respighi.d.o$ cd "/home/release/britney/hints" && perl -E '
    my (%years, %hints);
    while(<>) { 
        chomp;
        if (m/^\#\s*(\d{4})(?:-?\d{2}-?\d{2});/ or m/^\#\s*(?:\d+-\d+-\d+\s*[;:]?\s*)?done\s*[;:]?\s*(\d{4})(?:-?\d{2}-?\d{2})/) {
             $year = $1; next;
         }
         if (m/^((?:easy|hint) .*)/) {
             my $hint = $1; $years{$year}++ if defined($year) and not $hints{$hint}++;
             next;
         }
         if (m/^\s*$/) { $year = undef; next; }
    };
    for my $year (sort(keys(%years))) { 
        my $count = $years{$year};
        print "$year: $count\n"
    }' * OLD/jessie/* OLD/wheezy/* OLD/Lenny/* OLD/*

[3]  I should probably mention for good measure that extraction is ignoring all hints where it cannot figure out what year it was from or if it is a duplicate.  Notable it is omitting about 100 easy/hint-hints from “OLD/Lenny” (compared to a grep -c), which I think accounts for the low numbers from 2007 (among other).

Furthermore, hints files are not rotated based on year or age, nor am I sure we still have all complete hints files from all members.

Posted in Debian, Release-Team | Leave a comment

The stretch freeze is coming

The soft freeze has been on going for almost a month now and the full stretch freeze will start tomorrow night (UTC).  It has definitely been visible in the number of unblock requests that we have received so far.  Fortunately, we are no where near the rate of the jessie freeze.  At the moment, all unblock requests are waiting for the submitter (either for a clarification or an upload).

Looking at stretch at a glance (items are in no particular order):

Secure boot support

Currently, we are blocked on two items:

  • We do not have signing done yet for the boot packages (not even manual signing).
  • Our shim is not yet signed, so no hardware would be trusting our boot chain out of the box.

After they are done, we are missing a handful of uploads to provide a signed bootloader etc. plus d-i and some infrastructure bits need to be updated. At the moment, we are waiting for a handful of key people/organisations to move on their part. As such, there is not a lot you can do to assist here (unless you are already involved in the work).
On the flip side, if both of these items are resolved soon, there is a good chance that we can support secure boot in stretch.See bug#820036 and blockers for more information on the remaining items.

Where can you help with the release?

At the moment, the best you can do is to:

  • Test (packages, upgrades, etc.) and report bugs
  • File bugs against release-notes for issues that should be documented
  • Fix RC bugs (please see the next section)

 

Release Critical Bug report

The UDD bugs interface currently knows about the following release critical bugs:

  • In Total: 1148 (Including 193 bugs affecting key packages)
    • Affecting stretch: 294 (key packages: 158)
      That’s the number we need to get down to zero before the release. They can be split in two big categories:

      • Affecting stretch and unstable: 232 (key packages: 134)
        Those need someone to find a fix, or to finish the work to upload a fix to unstable:

        • 30 bugs are tagged ‘patch’. (key packages: 21)
          Please help by reviewing the patches, and (if you are a DD) by uploading them.
        • 17 bugs are marked as done, but still affect unstable. (key packages: 5)
          This can happen due to missing builds on some architectures, for example. Help investigate!
        • 185 bugs are neither tagged patch, nor marked done. (key packages: 108)
          Help make a first step towards resolution!
      • Affecting stretch only: 62 (key packages: 24)
        Those are already fixed in unstable, but the fix still needs to migrate to stretch. You can help by submitting unblock requests for fixed packages, by investigating why packages do not migrate, or by reviewing submitted unblock requests.

        • 36 bugs are in packages that are unblocked by the release team. (key packages: 14)
        • 26 bugs are in packages that are not unblocked. (key packages: 10)
Posted in Debian, Release-Team | 7 Comments

Piuparts integration in britney

As of today, britney now fetches reports from piuparts.debian.org and uses it as a part of her evaluation for package migration.  As with her RC bug check, we are only preventing (known) regressions from migrating.

The messages (subject to change) look something like:

  • Piuparts tested OK
  • Rejected due to piuparts regression
  • Ignoring piuparts failure (Not a regression)
  • Cannot be tested by piuparts (not a blocker)

If you want to do machine parsing of the Britney excuses, we also provide an excuses.yaml. In there, you are looking for “excuses[X].policy_info.piuparts.test-results”, which will be one of:

  • pass
  • regression
  • failed
  • cannot-be-tested

Enjoy. 🙂

 

Posted in Debian, Release-Team | Leave a comment

Improvements in apt-file 3.1.2

Yesterday, I just uploaded apt-file 3.1.2 into unstable, which comes with a few things I would like to highlight.

 

  • We fixed an issue where apt-file would not show top-level files in source packages. (bug#676642). Thanks to Paul Wise for the proposed solution.
  • Paul Wise also fixed a bug where apt-file list -I dsc <source-pkg> would fail to list all files in the source package if said file was also in other packages.
  • We added –filter-suites / –filter-origins options that can be used to narrow the search space.  Example: apt-file search --filter-suites unstable lintian/checks/

You can also set defaults in the config file – if you want to always search in unstable, simply do:

# echo 'apt-file::Search-Filter::Suite "unstable";' >> /etc/apt/apt-file.conf

For the suite filter, either a code name (“sid”) or a suite name (“unstable”) will work.  Please note that the filters are case-sensitive – suites/code names generally use all lowercase, whereas origins appear to use title-case (i.e. “unstable” vs. “Debian”).

 

Posted in apt-file, Debian | 1 Comment

Stretch transition freeze in a month

It is the first of October and that means the transition freeze is roughly one month away (Nov 5th 2016). In other words, this is the “final boarding call for transitions”.

 

Other milestone dates:

  • If you want new packages into Stretch, you have about 3 months (Jan 5th 2017).  This includes auto removed packages.
  • Automatic updates of existing packages stop in 4 months (Feb 5th 2017)

 

Posted in Debian, Release-Team | Leave a comment

Unseen changes to lintian.d.o

We have been making a lot of minor changes to lintian.d.o and the underlying report framework. Most of them were hardly noticeable to the naked. In fact, I probably would not have spotted any of them, if I had not been involved in writing them.  Nonetheless, I felt like sharing them, so here goes. 🙂

User “visible” changes:

In case you were wondering, the section title is partly a pun as half of these changes were intended to assist visually impaired users. They were triggered by me running into Sam Hartmann at DebConf16, where I asked him about how easy Debian’s websites were for blind people. Allegedly, we are generally doing quite good in his opinion (with one exception, for which Sam filed Bug#830213), which was a positive surprise for me.

On a related note: Thanks Luke Faraone and Asheesh Laroia for getting helping me started on these changes. 🙂

Reporting framework / “Internal” changes:

With the last change + the “−−no−generate−reports” option, we were able to schedule lintian more frequently. Originally, lintian only ran once a day. With the “−−no−generate−reports“, we added a second run and with the last changes, we bumped it to 4 times a day. Unsurprisingly, it means that we are now reprocessing the archive a lot faster than previously.

All of the above is basically the all the note-worthy changes on the Lintian reporting framework since the Partial rewrite of lintian’s reporting setup (~1½ years ago).

Posted in Debian, Lintian | 2 Comments

debhelper 10 is now available

Today, debhelper 10 was uploaded to unstable and is coming to a mirror near you “really soon now”. The actual changes between version “9.20160814” and version “10” are rather modest. However, it does mark the completion of debhelper compat 10, which has been under way since early 2012.

Some highlights from compat 10 include:

  • The dh sequence in compat 10 automatically regenerate autotools files via dh_autoreconf
  • The dh sequence in compat 10 includes the dh-systemd debhelper utilities
  • dh sequencer based packages now defaults to building in parallel (i.e. “–parallel” is default in compat 10)
  • dh_installdeb now properly shell-escapes maintscript arguments.

For the full list of changes in compat 10, please review the contents of the debhelper(7) manpage. Beyond that, you may also want to upgrade your lintian to 2.5.47 as it is the first version that knows that compat 10 is stable.

 

Posted in Debhelper, Debian | 2 Comments