Release Team unblock queue flushed

At the start of this week, I wrote that we had 58 open unblock requests open (of which 25 were tagged moreinfo).  Thanks to an extra effort from the Release Team, we now down to 25 open unblocks – of which 18 are tagged moreinfo.

We have now resolved 442 unblock requests (out of a total of 467).  The rate has also declined to an average of ~18 new unblock requests a day (over 26 days) and our closing rated increased to ~17.

With all of this awesomeness, some of us are now more than ready to have a well-deserved weekend to recharge our batteries.  Meanwhile, feel free to keep the RC bug fixes flowing into unstable.

Posted in Debian, Release-Team | Leave a comment

The first 12 days and 408 unblock requests into the Jessie freeze

The release team receives an extreme amount of unblock requests right now.  For the past 22 days[1], we have been receiving no less than 408 unblock/ageing requests.  That is an average of ~18.5/day.  In the same period, the release team have closed 350 unblocks requests, averaging 15.9/day.

This number does not account for number of unblocks, we add without a request, when we happen to spot when we look at the list of RC bugs[2]. Nor does it account for unblock requests currently tagged “moreinfo”, of which there are currently 25.

All in all, it has been 3 intensive weeks for the release team.  I am truly proud of my fellow team members for keeping up with this for so long!  Also a thanks to the non-RT members, who help us by triaging and reviewing the unblock requests!  It is much appreciated. 🙂

Random bonus info:

  • d (our diffing tool) finally got colordiff support during the Release Sprint last week.  Prior to that, we got black’n’white diffs!
    • ssh coccia.debian.org -t /srv/release.debian.org/tools/scripts/d
    • Though coccia.debian.org do not have colordiff installed right now.  I have filed a request to have it installed.
  • The release team have about 132 (active) unblock hints deployed right now in our hint files.

[1] We started receiving some in the 10 days before the freeze as people realised that their uploads would need an unblock to make it into Jessie.

[2] Related topics: “what is adsb?” (the answer being: Our top hinter for Wheezy)

Posted in Debian, Release-Team | 3 Comments

Release sprint – Preparing for Jessie

The release team are doing a sprint right now up the mini-DebConf in Cambridge, kindly hosted by ARM.

We have a fairly large agenda of around 10 items, ranging from “simple” things like determine the name for the next release to reviewing the list of RC bugs affecting Jessie.

Personally, I am very pleased with our progress.  We managed to finish 8 of items on the first day.  Furthermore, we spent several hours on the RC bug list and keeping up with the unblock requests.

There is also live status of the release team, courtesy of Jonathan Wiltshire.  Mind you it is manually updated.

We will announce the results of our sprint Sunday morning in our Release update talk.  The announcement will also be posted to debian-devel-announce like our freeze announcement.

Update: fix link to the live status

Posted in Debian, Release-Team | Leave a comment

Lintian – Upcoming API making it easier to write correct and safe code

The upcoming version of Lintian will feature a new set of API that attempts to promote safer code. It is hardly a “ground-breaking discovery”, just a much needed feature.

The primary reason for this API is that writing safe and correct code is simply too complicated that people get it wrong (including yours truly on occasion).   The second reason is that I feel it is a waste having to repeat myself when reviewing patches for Lintian.

Fortunately, the kind of issues this kind of mistake creates are usually minor information leaks, often with no chance of exploiting it remotely without the owner reviewing the output first[0].

Part of the complexity of writing correct code originates from the fact that Lintian must assume Debian packages to be hostile until otherwise proven[1]. Consider a simplified case where we want to read a file (e.g. the copyright file):

package Lintian::cpy_check;
use strict; use warnings; use autodie;
sub run {
  my ($pkg, undef, $info) = @_;
  my $filename = "usr/share/doc/$pkg/copyright";
  # BAD: This is an example of doing it wrong
  open(my $fd, '<', $info->unpacked($filename));
  ...;
  close($fd);
  return;
}

This has two trivial vulnerabilities[2].

  1. Any part of the path (usr,usr/share, …) can be asymlink to “somewhere else” like /
    1. Problem: Access to potentially any file on the system with the credentials of the user running Lintian.  But even then, Lintian generally never write to those files and the user has to (usually manually) disclose the report before any information leak can be completed.
  2. The target path can point to a non-file.
    1. Problem: Minor inconvenience by DoS of Lintian.  Examples include a named pipe, where Lintian will get stuck until a signal kills it.


Of course, we can do this right[3]:

package Lintian::cpy_check;
use strict; use warnings; use autodie;
use Lintian::Util qw(is_ancestor_of);
sub run {
  my ($pkg, undef, $info) = @_;
  my $filename = "usr/share/doc/$pkg/copyright";
  my $root = $info->unpacked
  my $path = $info->unpacked($filename);
  if ( -f $path and is_ancestor_of($root, $path)) {
    open(my $fd, '<', $path);
    ...;
    close($fd);
  }
  return;
}

Where “is_ancestor_of” is the only available utility to assist you currently.  It hides away some 10-12 lines of code to resolve the two paths and correctly asserting that $path is (an ancestor of) $root.  Prior to Lintian 2.5.12, you would have to do that ancestor check by hand in each and every check[4].

In the new version, the correct code would look something like this:

package Lintian::cpy_check;
use strict; use warnings; use autodie;
sub run {
  my ($pkg, undef, $info) = @_;
  my $filename = "usr/share/doc/$pkg/copyright";
  my $path = $info->index_resolved_path($filename);
  if ($path and $path->is_open_ok) {
    my $fd = $path->open;
    ...;
    close($fd);
  }
  return;
}

Now, you may wonder how that promotes safer code.  At first glance, the checking code is not a lot simpler than the previous “correct” example.  However, the new code has the advantage of being safer even if you forget the checks.  The reasons are:

  1. The return value is entirely based on the “file index” of the package (think: tar vtf data.tar.gz).  At no point does it use the file system to resolve the path.  Whether your malicious package trigger an undef warning based on the return value of index_resolved_path leaks nothing about the host machine.
    1. However, it does take safe symlinks into account and resolves them for you.  If you ask for ‘foo/bar’ and ‘foo’ is a symlink to ‘baz’ and ‘baz/bar’ exists in the package, you will get ‘baz/bar’.  If ‘baz/bar’ happens to be a symlink, then it is resolved as well.
    2. Bonus: You are much more likely to trigger the undef warning during regular testing, since it also happens if the file is simply missing.
  2. If you attempt to call “$path->open” without calling “$path->is_open_ok” first, Lintian can now validate the call for you and stop it on unsafe actions.

It also has the advantage of centralising the code for asserting safe access, so bugs in it only needs to be fixed in one place.  Of course, it is still possible to write unsafe code.  But at least, the new API is safer by default and (hopefully) more convenient to use.

[0] Lintian.debian.org being the primary exception here.

[1] This is in contrast to e.g. piuparts, which very much trusts its input packages by handing the package root access (albeit chroot’ed, but still).

[2] And also a bug.  Not all binary packages have a copyright – instead some while have a symlink to another package.

[3] The code is hand-typed into the blog without prior testing (not even compile testing it).  The code may be subject to typos, brown-paper-bag bugs etc. which are all disclaimed (of course).

[4] Fun fact, our documented example for doing it “correctly” prior to implementing is_ancestor_of was in fact not correct.  It used the root path in a regex (without quoting the root path) – fortunately, it just broke lintian when your TMPDIR / LINTIAN_LAB contained certain regex meta-characters (which is pretty rare).

Posted in Debian, Lintian | Leave a comment

Recent improvements to Britney2

As mentioned by Raphaël Hertzog, I have been spending some time on improving Britney2.  Just the other day I submitted a second branch for review that I expect to merge early next week.  I also got another set patches coming up soon.  Currently, none of them are really user visible, so unless you are hosting your own version of Britney, these patches are probably not all that interesting to you.

The highlights:

  1. Reduce the need for backtracking by finding semantically equivalent packages.
  2. Avoid needing to set up a backtrack point in some cases.
    • This has the side-effect of eliminating some O(e^n) runtime cases.
  3. Optimise “installability” testing of packages affected by a hinted migration.
    • This has the side-effect of avoiding some O(e^n) runtime cases when the “post-hint” state does not trigger said behaviour.
    • There is a follow-up patch for this one coming in the third series to fix a possible bug for a corner-case (causing a valid hint to be incorrectly rejected when it removed an “uninstallable” package).
  4. Reduce the number of affected packages to test when migrating items by using knowledge about semantically equivalent packages.
    • In some cases, Britney can now do “free” migrations when all binaries being updated replace semantically equivalent packages.
  5. (Merge pending) Avoid many redundant calls to “sort_actions()”, which exhibits at least O(n^2) runtime in some cases.
    • For the dataset Raphaël submitted, this patch shaves off over 30 minutes runtime.  In the particular case, each call to sort_actions takes 3+ minutes and it was called at least 10 times, where it was not needed.
    • That said, sort_actions have a vastly lower runtime in the runs for Debian (and presumably also Ubuntu, since no one complained from their side so far).

The results so far:

After the first patch series was merged, the Kali dataset (from Raphaël) could be processed in “only” ~2 hours. With the second patch series merged, the dataset will drop by another 30-50 minutes (most of which are thanks to the change mentioned in highlight #5).

The third patch series currently do not have any mention-worthy performance related changes.  It will probably be limited to bug fixes and some refactoring.

Reflections:

The 3 first highlights only affects the “new” installability tester meaning that the Britney2 instances at Ubuntu and Tanglu should be mostly unaffected by the O(n^2) runtime.  Although those cases will probably just fail with several “AIEEE“s. 🙂 The 5th highlight should equally interesting to all Britney2 instances though.

For me, the most interesting part is that we have never observed the O(n^2) behaviour in a daily “sid -> testing” run.  The dataset from Raphaël was basically a “stable -> testing/sid” run, which is a case I do not think we have ever done before.  Despite our current updates, there is still room for improvements on that particular use case.

In particular, I was a bit disheartened at how poorly our auto hinter(s) performed on this dataset.  Combined they only assisted with the migration of something like 28 “items”.  For comparison, the “main run” migrated ~7100 “items” and 9220 items were unable to migrate. Furthermore, the “Original” auto hinter spend the better part of 15 minutes computing hints  – at least it results in 10 “items” migrating.

Links to the patches:

Posted in Debian, Release-Team | Leave a comment

Upcoming changes to Lintian (in 2.5.22)

The next version of Lintian, 2.5.22, is long overdue – mostly because 2.5.21 FTBFS in sid. Besides fixing test failures, 2.5.22 also fixes:

  • False-negative for most/all JAR related tags caused by file(1) changing its output for these files.
  • A performance regression in checks/cruft.pm. For large source packages, this could severely affect performance of Lintian.  This is #738342.
  • A regression where lintian would not parse options appearing after file arguments.
  • Some false-positives in the “GFDL with invariants” check.

Besides bug fixes, we also added a couple of new features/nice changes:

  • The –color option now defaults to “auto” instead of “never”.
  • The homepage now links to the generated API-docs and thus, also the developer tutorial.
    • Patches extending/improving either/both are more than welcome.
  • The test-suite runner now caches test artifacts meaning that subsequent test runs become vastly faster.
  • The test-suite runner can now run tests under “Devel::Cover“, so we can now collect statistics about how much of the code is covered by the test suite.
  • The “html_reports” tool (used e.g. on lintian.d.o) has been optimised. In particular, it only calls gnuplot twice, instead of once per data file (of which there was basically one per tag).
  • The html-pages generated by the “html_reports” tool now uses “content”-based names for resources (e.g. images and the .css file). This allows them to be cached vastly longer, since any content changes will rename the resource and force browser to refetch them.
    • All resources handled this way are now stored in “resources/”.  As a side-effect, this means that images files are moved from “images/” to “resources/”.
    • A notable exception to this is the auto-generated (.svg) graphs.

We just need to do some final tweaks and get the tag descriptions reviewed before we are ready to release the next version.

Posted in Debian, Lintian | Leave a comment

Release architecture meeting 2014-01-26

Today, we held an one-hour IRC-meeting to debate the status of the current architectures in sid.  During that one hour, we brought up all 13 architectures.  We made a decision on 9 of the architectures.  The remaining 4 will be revisited within 2 months.  As for the actual results, we will announce these on debian-devel-announce with our next bits mail.  I suspect the announcement to happen within a couple of days.

I am quite pleased that we managed to keep the meeting to one hour.  Especially considering that it was planned less than a week ago.  One of the things, that I believe helped us, was that we had summarised the status for each architecture prior to the meeting.  In my view, the summary allowed us to finish at least half of the architectures within their allocated 5 minutes.  In fact, we had 20 minutes for the last 2 architectures – partly because amd64 + i386 combined took about a minute.

A second thing that helped us stay on time was cutting debates short.  Obviously, with less than 5 minutes per architecture, there was little time for debate and, I admit, that was by design.  This meant that we had to defer 4 decisions for later.  We are currently waiting for information, which we hope to have soon.  More on that in the announcement later.
I would like to thank the attendees for contributing to a productive and short meeting.  It was a pleasure and I hope we can repeat the success next time. 🙂

 

Posted in Debian, Release-Team | Leave a comment

Jessie finally has less than 500 RC bugs

I am very pleased to say that RC bug counter for Jessie finally dropped below 500 RC bugs.  For comparison, we broke the 700 RC bug curve in the start of November, so that is about 200 RC bugs in about 7-8 weeks (or about 25-28 RC bugs per week).

Today, we have about 162 RC bugs on the auto-removal list.  Though, I suspect many of the affected packages have reverse dependencies, so their removal from testing may be up 30 days away.  Nevertheless, by this time in January, we could be looking at no more than 350 RC bugs left for Jessie.

 

Posted in Debian, Release-Team | 7 Comments

Automated reprocessing of packages on lintian.debian.org

Yesterday, I have managed to finish up an old pet peeve of mine.  I wrote a series of patches to have harness reprocess all packages, which were last checked by an older version of Lintian than the current.  It is not completed to the level I would like, but it is good enough for now.

The implementation works by processing up to 1024 groups after each daily run.  It also have a timer to skip this, if the run has taken more than 4 hours.  Packages to be reprocessed are sorted by the version they were last processed – should a new version of Lintian be upload before all packages have been reprocessed, the oldest results are refreshed first.  🙂

A rough estimate suggests than in about 3-4 weeks, the entire archive will have been reprocessed with Lintian 2.5.20.  It is by no means faster than a full run (which is about a week), but it is automatic and removes the “human” factor in keeping Lintian results up to date.

Posted in Debian, Lintian | 2 Comments

Getting out of the way

I have decided to step down as main maintainer of Lintian and pass the baton to Bastien Roucariès.  This is actually “fairly” old news, since I announced this almost a month ago.  However, I was not very vocal about it until now (so do not be surprised if you had not heard of this before).

In the past month, I never once regretted having stepped down.  If anything, I should probably have done it a bit earlier.  Beyond the lack of motivation, I also realised that I had become an “all talk and no do”-maintainer.  The kind that I have been advocating against publicly.  This may sound weird to a lot of people, who knows me as “the Lintian guy” or “Mr. Lintian” (or whatever Lintian-related nickname people gave me).  But the fact is that Bastien has done more for Lintian in the past month than I have in the past two.

Despite stepping down as main developer, I have not left Lintian completely.  I am still around for helping/explaining, uploading and keeping lintian.debian.org running.

Posted in Debian, Lintian | Leave a comment