Partial rewrite of lintian’s reporting setup

I had the mixed pleasure of doing a partial rewrite of lintian’s reporting framework.  It started as a problem with generating the graphs, which turned out to be “not enough memory”. On the plus side, I am actually quite pleased with the end result.  I managed to “scope-creep” myself quite a bit and I ended up getting rid of a lot of old issues.

The major changes in summary:

  • A lot of logic was moved out of harness, meaning it is now closer to becoming a simple “dumb” task scheduler.  With the logic being moved out in separate processes, harness now hogs vastly less memory that I cannot convince perl to release to the OS.  On “vastly less” is on the order of reducing “700ish MB” to “32 MB”.
  • All important metadata was moved into the “harness state-cache”, which is a simple YAML file. This means that “Lintian laboratory” is no longer a data store. This change causes a lot of very positive side effects.
  • With all metadata now stored in a single file, we can now do atomic updates of the data store. That said, this change itself does not enable us to run multiple lintian’s in parallel.
  • As the lintian laboratory is no longer a data store, we can now do our processing in “throw away laboratories” like the regular lintian user does.  As the permanent laboratory is the primary source of failure, this removes an entire class of possible problems.

There are also some nice minor “features”:

  • Packages can now be “up to date” in the generated reports.  Previously, they would always be listed as “out of date” even if they were up to date.  This is the only end user/website-visitor visible change in all of this (besides the graphs are now working again \o/).
  • The size of the harness work list is no longer based on the number of changes to the archive.
  • The size of the harness work list can now be changed with a command line option and is no longer hard coded to 1024.  However, the “time limit” remains hard coded for now.
  • The “full run” (and “clean run”) now simply marks everything “out-of-date” and processes its (new) backlog over the next (many) harness runs.  Accordingly, a full-run no longer causes lintian to run 5-7 days on lilburn.d.o before getting an update to the website.  Instead we now get incremental updates.
  • The “harness.log” now features status updates from lintian as they happen with “processed X successfully” or “error processing Y” plus a little wall time benchmark.  With this little feature I filed no less than 3 bugs against lintian – 2 of which are fixed in git.  The last remains unfixed but can only be triggered in Debian stable.
  • It is now possible with throw-away labs to terminate the lintian part of a reporting run early with minimal lost processing.  Since the lintian-harness is regular fed status updates from lintian, we can now mark successfully completed entries as done even if lintian does not complete its work list.  Caveat: There may be minor inaccuracies in the generated report for the particular package lintian was processing when it was interrupted.  This will fix itself when the package is reprocessed again.
  • It is now vastly easier to collect new meta data to be used in the reports.  Previously, they had to be included in the laboratory and extracted from there.  Now, we just have to fit it into a YAML file.  In fact, I have been considering to add the “wall time” and make a “top X slowest” page.
  • It is now possible to generate the html pages with only a “state-cache” and the “lintian.log” file.  Previously, it also required a populated lintian laboratory.

As you can probably tell, I am quite pleased with the end result.  The reporting framework lacks behind in development, since it just “sits there and takes care of itself”.  Also with the complete lack of testing, it also suffers from the “if it is not broken, then do not fix it” paradigm (because we will not notice if we broke until it is too late).

Of course, I managed to break the setup a couple of times in the process.  However, a bonus feature of the reporting setup is that if you break it, it simply leaves an outdated report on the website.

Anyway, enjoy. :)


Posted in Debian, Lintian | Leave a comment

Status on Jessie (December 2014)

Here is a slightly overdue status on Jessie.


  • We are not ready to set a date on a Jessie release yet.  Even if we were, it would be unlikely that said date would be in January.  Accordingly, it is safe to assume that the rapid automatic removals clause of the freeze policy will be applied before our release.  If your package (or a package you use) depend on any package in this list, then they are at risk.
  • It is unclear to me whether our CD1s are able to contain all the necessary packages.  This was a major issue for Wheezy.  So far I have only done some minor prodding here.  However, I really want this item done soon as it can easily take a month or more to resolve this.
  • While the release notes have been improved quite a bit, I am certain that there are more cases we can cover.  As an example, I am looking for input on #773386.
  • Despite the declining number of RC bugs, we still have some particular unhealthy ones left.  E.g. #759530.  There is also a view of some of the known unfixed Jessie blockers.
  • We have had a number of issues with APT and dpkg (e.g. trigger cycles, APT breaking under puppet) that caused upgrade failures or were a severe regression.  Most of these have been resolved now.  There are some trigger issues left and I have pushed for a fixed dpkg at the cost of possibly removing some packages. See #772866 (among others).

Stricter freeze policy per January 5th

The next timed change of the freeze policy will apply per January 5th.  After that date, we will only accept RC bugs fixes.  Which means it is final chance for translation updates.

More on RC bugs

In absolute numbers, the RC bugs have declined quite well.  We are below 150 now.  We lost quite a bit of traction in December compared to November. However, November was an extremely efficient month.  However, we still need the final push here.

Debian installer release pending

Yesterday, we received a list of packages that needed to be unblocked for d-i with a remark that a release of d-i might follow.  Based on what we have unblocked previously, it will likely contain some (improved?) UEFI support.

Pending Debian 7.8 release

While not directly relevant to Jessie, we also got a pending Wheezy release planned for the 10th of January. The window for getting changes into the 7.8 release closes this weekend.

Want to help?

  • File bugs against release-notes (source at [RN source]) and installation-guide for missing or outdated documentation.  Patches and drafts especially welcome.
  • Fix RC bugs – especially the known Jessie blockers.
  • Test upgrade paths and installation media.  Although in both cases, you may want to wait a bit (for the new dpkg to migrate and for the new debian-installer release respectively).
  • Consider offering your help to teams such as the CD team or debian-installer team.

Thank you,

[RN source]:

svn co

Git Repo:


Posted in Debian, Release-Team | 3 Comments

Jessie has half the number of RC bugs compared to Wheezy

In the last 24 hours, the number of RC bugs currently affecting Jessie was reduced to just under half of the same number for Wheezy.


There are still a lot of bugs to be fixed, but please keep up the good work. :)


Posted in Debian, Release-Team | Leave a comment

Volume of debian-release mailing list

Page 1 of 5

To be honest, I do not know how many mails it shows “per page” (though I assume it is a fixed number).  So for comparison, I found the month on debian-devel@l.d.o with the  highest volume in the past two years:  May 2013 with “Page 1 of 4“.

I hope you will please forgive us, if we are a bit terse in our replies or slow to reply.  We simply got a lot to deal with.  :)


Posted in Debian, Release-Team | 1 Comment

Release Team unblock queue flushed

At the start of this week, I wrote that we had 58 open unblock requests open (of which 25 were tagged moreinfo).  Thanks to an extra effort from the Release Team, we now down to 25 open unblocks – of which 18 are tagged moreinfo.

We have now resolved 442 unblock requests (out of a total of 467).  The rate has also declined to an average of ~18 new unblock requests a day (over 26 days) and our closing rated increased to ~17.

With all of this awesomeness, some of us are now more than ready to have a well-deserved weekend to recharge our batteries.  Meanwhile, feel free to keep the RC bug fixes flowing into unstable.

Posted in Debian, Release-Team | Leave a comment

The first 12 days and 408 unblock requests into the Jessie freeze

The release team receives an extreme amount of unblock requests right now.  For the past 22 days[1], we have been receiving no less than 408 unblock/ageing requests.  That is an average of ~18.5/day.  In the same period, the release team have closed 350 unblocks requests, averaging 15.9/day.

This number does not account for number of unblocks, we add without a request, when we happen to spot when we look at the list of RC bugs[2]. Nor does it account for unblock requests currently tagged “moreinfo”, of which there are currently 25.

All in all, it has been 3 intensive weeks for the release team.  I am truly proud of my fellow team members for keeping up with this for so long!  Also a thanks to the non-RT members, who help us by triaging and reviewing the unblock requests!  It is much appreciated. :)


Random bonus info:

  • d (our diffing tool) finally got colordiff support during the Release Sprint last week.  Prior to that, we got black’n’white diffs!
    • ssh -t /srv/ <srcpkg>
    • Though do not have colordiff installed right now.  I have filed a request to have it installed.
  • The release team have about 132 (active) unblock hints deployed right now in our hint files.


[1] We started receiving some in the 10 days before the freeze as people realised that their uploads would need an unblock to make it into Jessie.

[2] Related topics: “what is adsb?” (the answer being: Our top hinter for Wheezy)


Posted in Debian, Release-Team | 2 Comments

Release sprint – Preparing for Jessie

The release team are doing a sprint right now up the mini-DebConf in Cambridge, kindly hosted by ARM.

We have a fairly large agenda of around 10 items, ranging from “simple” things like determine the name for the next release to reviewing the list of RC bugs affecting Jessie.

Personally, I am very pleased with our progress.  We managed to finish 8 of items on the first day.  Furthermore, we spent several hours on the RC bug list and keeping up with the unblock requests.

There is also live status of the release team, courtesy of Jonathan Wiltshire.  Mind you it is manually updated.

We will announce the results of our sprint Sunday morning in our Release update talk.  The announcement will also be posted to debian-devel-announce like our freeze announcement.

Update: fix link to the live status

Posted in Debian, Release-Team | Leave a comment

Lintian – Upcoming API making it easier to write correct and safe code

The upcoming version of Lintian will feature a new set of API that attempts to promote safer code. It is hardly a “ground-breaking discovery”, just a much needed feature.

The primary reason for this API is that writing safe and correct code is simply too complicated that people get it wrong (including yours truly on occasion).   The second reason is that I feel it is a waste having to repeat myself when reviewing patches for Lintian.

Fortunately, the kind of issues this kind of mistake creates are usually minor information leaks, often with no chance of exploiting it remotely without the owner reviewing the output first[0].

Part of the complexity of writing correct code originates from the fact that Lintian must assume Debian packages to be hostile until otherwise proven[1]. Consider a simplified case where we want to read a file (e.g. the copyright file):

package Lintian::cpy_check;
use strict; use warnings; use autodie;
sub run {
  my ($pkg, undef, $info) = @_;
  my $filename = "usr/share/doc/$pkg/copyright";
  # BAD: This is an example of doing it wrong
  open(my $fd, '<', $info->unpacked($filename));

This has two trivial vulnerabilities[2].

  1. Any part of the path (usr,usr/share, …) can be asymlink to “somewhere else” like /
    1. Problem: Access to potentially any file on the system with the credentials of the user running Lintian.  But even then, Lintian generally never write to those files and the user has to (usually manually) disclose the report before any information leak can be completed.
  2. The target path can point to a non-file.
    1. Problem: Minor inconvenience by DoS of Lintian.  Examples include a named pipe, where Lintian will get stuck until a signal kills it.

Of course, we can do this right[3]:

package Lintian::cpy_check;
use strict; use warnings; use autodie;
use Lintian::Util qw(is_ancestor_of);
sub run {
  my ($pkg, undef, $info) = @_;
  my $filename = "usr/share/doc/$pkg/copyright";
  my $root = $info->unpacked
  my $path = $info->unpacked($filename);
  if ( -f $path and is_ancestor_of($root, $path)) {
    open(my $fd, '<', $path);

Where “is_ancestor_of” is the only available utility to assist you currently.  It hides away some 10-12 lines of code to resolve the two paths and correctly asserting that $path is (an ancestor of) $root.  Prior to Lintian 2.5.12, you would have to do that ancestor check by hand in each and every check[4].

In the new version, the correct code would look something like this:

package Lintian::cpy_check;
use strict; use warnings; use autodie;
sub run {
  my ($pkg, undef, $info) = @_;
  my $filename = "usr/share/doc/$pkg/copyright";
  my $path = $info->index_resolved_path($filename);
  if ($path and $path->is_open_ok) {
    my $fd = $path->open;

Now, you may wonder how that promotes safer code.  At first glance, the checking code is not a lot simpler than the previous “correct” example.  However, the new code has the advantage of being safer even if you forget the checks.  The reasons are:

  1. The return value is entirely based on the “file index” of the package (think: tar vtf data.tar.gz).  At no point does it use the file system to resolve the path.  Whether your malicious package trigger an undef warning based on the return value of index_resolved_index leaks nothing about the host machine.
    1. However, it does take safe symlinks into account and resolves them for you.  If you ask for ‘foo/bar’ and ‘foo’ is a symlink to ‘baz’ and ‘baz/bar’ exists in the package, you will get ‘baz/bar’.  If ‘baz/bar’ happens to be a symlink, then it is resolved as well.
    2. Bonus: You are much more likely to trigger the undef warning during regular testing, since it also happens if the file is simply missing.
  2. If you attempt to call “$path->open” without calling “$path->is_open_ok” first, Lintian can now validate the call for you and stop it on unsafe actions.

It also has the advantage of centralising the code for asserting safe access, so bugs in it only needs to be fixed in one place.  Of course, it is still possible to write unsafe code.  But at least, the new API is safer by default and (hopefully) more convenient to use.


[0] being the primary exception here.

[1] This is in contrast to e.g. piuparts, which very much trusts its input packages by handing the package root access (albeit chroot’ed, but still).

[2] And also a bug.  Not all binary packages have a copyright – instead some while have a symlink to another package.

[3] The code is hand-typed into the blog without prior testing (not even compile testing it).  The code may be subject to typos, brown-paper-bag bugs etc. which are all disclaimed (of course).

[4] Fun fact, our documented example for doing it “correctly” prior to implementing is_ancestor_of was in fact not correct.  It used the root path in a regex (without quoting the root path) – fortunately, it just broke lintian when your TMPDIR / LINTIAN_LAB contained certain regex meta-characters (which is pretty rare).

Posted in Debian, Lintian | Leave a comment

Recent improvements to Britney2

As mentioned by Raphaël Hertzog, I have been spending some time on improving Britney2.  Just the other day I submitted a second branch for review that I expect to merge early next week.  I also got another set patches coming up soon.  Currently, none of them are really user visible, so unless you are hosting your own version of Britney, these patches are probably not all that interesting to you.


The highlights:

  1. Reduce the need for backtracking by finding semantically equivalent packages.
  2. Avoid needing to set up a backtrack point in some cases.
    • This has the side-effect of eliminating some O(e^n) runtime cases.
  3. Optimise “installability” testing of packages affected by a hinted migration.
    • This has the side-effect of avoiding some O(e^n) runtime cases when the “post-hint” state does not trigger said behaviour.
    • There is a follow-up patch for this one coming in the third series to fix a possible bug for a corner-case (causing a valid hint to be incorrectly rejected when it removed an “uninstallable” package).
  4. Reduce the number of affected packages to test when migrating items by using knowledge about semantically equivalent packages.
    • In some cases, Britney can now do “free” migrations when all binaries being updated replace semantically equivalent packages.
  5. (Merge pending) Avoid many redundant calls to “sort_actions()”, which exhibits at least O(n^2) runtime in some cases.
    • For the dataset Raphaël submitted, this patch shaves off over 30 minutes runtime.  In the particular case, each call to sort_actions takes 3+ minutes and it was called at least 10 times, where it was not needed.
    • That said, sort_actions have a vastly lower runtime in the runs for Debian (and presumably also Ubuntu, since no one complained from their side so far).


The results so far:

After the first patch series was merged, the Kali dataset (from Raphaël) could be processed in “only” ~2 hours. With the second patch series merged, the dataset will drop by another 30-50 minutes (most of which are thanks to the change mentioned in highlight #5).

The third patch series currently do not have any mention-worthy performance related changes.  It will probably be limited to bug fixes and some refactoring.



The 3 first highlights only affects the “new” installability tester meaning that the Britney2 instances at Ubuntu and Tanglu should be mostly unaffected by the O(n^2) runtime.  Although those cases will probably just fail with several “AIEEE“s. :) The 5th highlight should equally interesting to all Britney2 instances though.


For me, the most interesting part is that we have never observed the O(n^2) behaviour in a daily “sid -> testing” run.  The dataset from Raphaël was basically a “stable -> testing/sid” run, which is a case I do not think we have ever done before.  Despite our current updates, there is still room for improvements on that particular use case.

In particular, I was a bit disheartened at how poorly our auto hinter(s) performed on this dataset.  Combined they only assisted with the migration of something like 28 “items”.  For comparison, the “main run” migrated ~7100 “items” and 9220 items were unable to migrate. Furthermore, the “Original” auto hinter spend the better part of 15 minutes computing hints  – at least it results in 10 “items” migrating.


Links to the patches:


Posted in Debian, Release-Team | Leave a comment

Getting out of the way

I have decided to step down as main maintainer of Lintian and pass the baton to Bastien Roucariès.  This is actually “fairly” old news, since I announced this almost a month ago.  However, I was not very vocal about it until now (so do not be surprised if you had not heard of this before).

In the past month, I never once regretted having stepped down.  If anything, I should probably have done it a bit earlier.  Beyond the lack of motivation, I also realised that I had become an “all talk and no do”-maintainer.  The kind that I have been advocating against publicly.  This may sound weird to a lot of people, who knows me as “the Lintian guy” or “Mr. Lintian” (or whatever Lintian-related nickname people gave me).  But the fact is that Bastien has done more for Lintian in the past month than I have in the past two.

Despite stepping down as main developer, I have not left Lintian completely.  I am still around for helping/explaining, uploading and keeping running.

Posted in Debian, Lintian | Leave a comment

Wheezy was brought to you by …

During the Wheezy freeze, the Debian release team deployed 3254 hints[1].  This number may include some duplicates (i.e. where two members of the team hinted the same package), it certainly does not include a lot of rejected requests <insert more disclaimers here>.

The top hinter was *drum roll*… Adam, who did 1799 hints(That is 55% of all hints during the freeze).  For comparison, the second and third runner ups added together did 1023 hints (or 31.4%).  Put in a different way, on average Julien Cristau and I would both add about 1.5 hints each day and Adam would on his own add 5.6 hints a day.

Of course, this is not intended to diminish the work other the rest of the team.  Reviewing and unblocking packages is not all there is to a release.  Particularly, a great thanks to Cyril Brulebois for his hard work on the Debian Installer (without which Debian Wheezy could not be released at all).


[1] Determined by:

  egrep -c 'done 201(3|2-?(07|08|09|10|11|12)) $HINT_FILE

It does not count hints, but the little “header” we add above our hints.  One header can apply to multiple hints (which is good, because udeb unblocks are usually paired with a regular unblock and we don’t want to count them as two separate hints).

Posted in Debian, Release-Team | 5 Comments

Getting space for more packages

In 2011, I wrote about how small files could consume a lot of space. I meant to do a follow-up on the savings but I forgot about it until now.

In 2.5.7, we started compressing some of the collected data files. Some of these are ridiculously compressable (#664794).  Even better, compressing them is sometimes faster than writing them directly to the disk, so in some cases it is a pure win/win.  For lintian.d.o, we also see a vast size reduction in overall size of the laboratory.

I have taken a few samples occasionally. The samples were done with du(1):

$ du -csh [--apparent-size] laboratory/*

Version/date du -csh –apparent-size
N/A – around 20 Mar 2012 (#664794) 16G 13G
2.5.6 (Fri Apr 27 2012) 14GB N/A
2.5.6 (Mon Jun 04 2012)) N/A 12G (Fri Sep 21 2012) 12G 8.3G
2.5.11 (Wed Jan 2 2013) 10G 6.1G

And the most awesome part of this? The comparison is quite biased against the 2.5.11 entry, which is the only entry to also process experimental (approx. 10% extra packages).  Some of the early entries (2.5.6 and “older”) might also have suffered from the “too many links” issue[1].  I only wish I had been better at collecting data points, so I could have made a proper graph of it.  :)

It sounds almost too good to be true, but if you look at the size of one of the linux-image packages[2], the space usage dropped from 27M to 15M between 2.5.5 to 2.5.9.   Currently it is squeezed down to 14M (tested with head of the git master branch).

[1] I believe is about 5-10% less binary packages processed for those runs.

[2] linux-image-3.2.0-2-amd64_3.2.20-1_amd64.deb

Posted in Debian, Lintian | Leave a comment

Performance bottlenecks in Lintian

Thanks to a heads up from Bastian Blank, I learned that Lintian 2.5.7 and 2.5.8 were horribly slow on the Linux binaries.  Bastian had already identified the issue and 2.5.9 fixed the performance regression.

But in light of that, I decided to have a look at a couple of other bottlenecks.  First, I added a simple benchmark support to Lintian 2.5.10 (enabled with -dd) that prints the approximate run time of a given collection.  As an example, when running lintian -dd on lintian 2.5.10, you can see something like:

N: Collecting info: unpacked for source:lintian/2.5.10 ...
N: Collection script unpacked for source:lintian/2.5.10 done (0.699s)

When done on linux-image, the slowest 3 things with 2.5.10 are (in order of appearance):

N: Collection script strings for binary:linux-image-3.2.0-2-amd64/3.2.20-1/amd64 done (12.333s)
N: Collection script objdump-info for binary:linux-image-3.2.0-2-amd64/3.2.20-1/amd64 done (15.915s)
N: Finished check: binaries (5.911s)

(The mileage (and order) probably will vary a bit.)

These 3 things makes up about 22 seconds of a total running time on approximately 28-30s on my machine.  Now if you wondering how 12, 16 and 6 becomes 22 the answer is “parallelization”.  strings and objdump-info are run in parallel so only the “most
expensive” of the two counts in practise (with multiple processing units).

The version of linux-image I have been testing (3.2.20-1, amd64) has over 2800 ELF binaries (kernel modules).  That makes the runtime of strings and objdump-info much more dominating than in “your average package”.   For the fun of it – I have done a small informal benchmark of various Lintian versions on the binary.

I have used the command line:

# time is the bash shell built-in and not /usr/bin/time
$ time lintian -EvIL +pedantic linux-image-3.2.0-2-amd64_3.2.20-1_amd64.deb >/dev/null
# This was used with only versions that did not accept -L +pedantic
$ time lintian -EvI --pedantic linux-image-3.2.0-2-amd64_3.2.20-1_amd64.deb >/dev/null

With older versions of Lintian (<= 2.5.3) Perl starts to emit warnings; these have been manually filtered out.  I used lintian from the git repository (i.e. I didn’t install the packages, but checked out the relevant git tags).  I had libperlio-gzip-perl installed (affects the 2.5.10 run).

Most results are only from a single run, though I ran it twice on the first version (hoping my kernel would cache the deb for the next run). The results are:

real    0m28.836s
user    0m36.982s
sys     0m3.280s

real    1m9.378s
user    0m33.702s
sys     0m11.177s

real    4m54.492s
user    4m0.631s
sys     0m30.466s

2.5.7 (not tested, but probably about same as 2.5.8)

real    1m20s   - 1m22s
user    0m19.0s - 0m20.7s
sys     0m5.1s  - 0m5.6s

I think Bastian’s complaint was warranted for 2.5.{7,8}.  :)

While it would have been easy to attribute the performance gain in 2.5.10 on the new parallelization improvements, it is simply not the case. These improvements only apply to running collections when checking multiple packages.  On my machine, the parallelization limit for a package is effectively determined by the dependencies between the collections on my machine.

Instead the improvements comes from reducing the number of system(3) (or fork+exec) calls Lintian does.  Mostly through using xargs more, even if it meant slightly more complex code.  But also, libperlio-gzip-perl shaved off a couple of seconds on “binaries” check.

But as I said, linux-image is “not your average package”.  Most of the improvements mentioned here are hardly visible on other packages.   So let’s have a look at some more other bottlenecks.  In my experience the following are the “worst offenders”:

  • unpacked (collection)
    • Seen on wesnoth-1.9 source. Here the problem seems to be tar+bzip2, so there is not really a lot to do (on the Lintian side). Though feel free to prove me wrong. :)
  • file-info (collection)
    • Seen in eclipse/eclipse-cdt source. file(1) appears to spend a lot of time classifying some source files. For eclipse-cdt, I experience an approx. 10 second speed up (from 40s to 30s) if file are recompiled with -O2. (That would be #659355).  However, even if file is compiled with -O2, the file-info collection is still the dominating  factor.
  • manpages (check)
    • Running man on manpages can be a dominating factor in certain doc packages. This is #677874 and suggestions for fixing it are more than welcome.

But enough Lintian for now… time to fix some RC bugs!

Posted in Debian, Lintian | 1 Comment

Kudos to Jakub Adam, Miguel Landaeta and James Page

Credit where it is due and I believe it is due for Jakub Adam for packaging eclipse packages.  If you use any of the eclipse packages provided the apt repositories for Wheezy or sid, it is very likely you have Jakub Adam to thank for it.

I also believe that Miguel Landaeta and James Page deserve praise for their work.  Miguel is to thank for the removal of libservlet2.4-java and updating its reverse dependencies – not in that order ;-).  James Page, on the other hand, has been introducing and updating a lot of packages, noticeably the jenkins packages.

Thank you and keep up the good work.

Posted in Debian | 1 Comment

Some sponsors are “evil and pedantic”

If you want to enable all Lintian tags, just remember the phrase:

Some sponsors are “evil and pedantic”

Or on the command-line:

 $ lintian -EvIL +pedantic ...

It works for Lintian 2.5.5 (and newer), which handles “pedantic” like other severities.  If you need help understanding the tags, you can add an extra “i” (-i) to “evil”.  That being said, remember that experimental (-E) and pedantic (-L +pedantic) tags are what they are for a reason. Also quite a few people will probably find verbose (-v) too noisy. However, leaving any of them out would have ruined the mnemonic. :)

Of course, you can also ask Lintian to enable all tags via your lintianrc file.  Here is a quick-start:

display-info = yes # or no
display-experimental = yes # or no
pedantic = yes # or no
verbose = yes # or no

Posted in Debian, Lintian | 2 Comments

Testing testing migration

If you have been following Lintian’s development closely, you will probably have noticed that I have not really done anything there for the past week. Instead I have turned my focus on our testing migration script, britney2. First, I have created a minimal test suite[1]. It started as 4 simple tests and by now it contains about 30 tests.

The size of each test is rather small; the largest tests are about 1600 binary packages in total[2], but most are 2-20 binary packages in total. Thus the test suite is rather fast compared to a “live data sample”, which easily takes more than 10 minutes for a single run. Unfortunately, hand-crafting the test data is somewhat annoying and easy to get wrong.

The test suite has a somewhat unfair focus on “auto-hint”[3] cases, so the current britney2 fails up to 14 tests. Some of these appears to fail because the auto-hinter (for some reason) receives incomplete information about the situation. To my knowledge we not been able to debug the situation, but Adam has a refactor branch that does not seem to have this issue. Personally I am hoping it will soon be merged into the master branch, especially because it seems to simplify a lot of common operations.

Joachim Breitner (who has been working on a SAT-solver based britney) also contributed a couple of test cases[4]. Allegedly, SAT-britney does rather well on the test suite, failing only 2 tests as far as I can tell[5]. On the other hand, it does solve a some of the more interesting cases britney2 does not solve.

On a more mathematical note, the britney2 implementation behaves like a function[6] with an attractive fixed point[7]. This is interesting, because for some cases it may take britney2 a couple of iterations to reach the right solution. This fixed point is somewhat simple to find by using the following steps (pseudo-code):

// Runtime complexity O(n * br * diff), where "n" is the number of iterations until
// a fixed point is reached, "br" is the complexity of "run_britney" and "diff" is
// the runtime of the "last != current" comparison.
function find_fixed_point(initial);
    last = run_britney(initial)
    current = run_britney(last)
    while last != current ; do
        last = current
        current = run_britney(last)
    return current

This gives us a simple way to test if britney will eventually solve the issue herself (and when she will do it). Currently britney2 is automatically run twice a day, so for every 2 iterations (beyond the first) roughly translates to a 24-hours delay. So far the test suite does not have a lot of problems that requires more than one iteration. Personally I would be pleased if it turned out to stay that way as the test suite coverage grows.

If you are interested in playing around with this, you can get sources from:

  • britney2
    • Currently only works in stable (i.e. requires python2.5 and python-apt < 0.8 or so)
    • See the INSTALL file for instructions
    • Adam’s branch
      • use the “p-u” branch.
  • SAT-britney
    • I haven’t tested this one and I do not know the requirements here
  • britney-tests
    • See the README file for instructions



[2] These tests are auto-generated, so it is merely an “up-scaled pattern”.

[3] Basically if two (or more) packages needs to migrate into testing at the exact same time, they need to be hinted in.

[4] Not to mention all the copy-waste errors he pointed out in mine. Apparently, SAT-britney has stricter requirements to the data than britney2. :P

[5] I assume the test called “sat-britney-death” (created by Joachim) was named that way for a reason. The second failure is caused by SAT britney not reading hints (yet?), so the “approve tpu package” test case should fail.

[6] A function that maps an “archive” into another “archive”… erh, I mean, it maps a set of packages into another set of packages… :P


Assuming my claim to be true, the function will have more than one fixed point. The obtained fixed point depends on the initial state of testing.

As an example:
– y depends on x
– x in testing has RC bugs

If x is not in testing, it cannot migrate to testing (due to its RC bugs). If x is not in testing, then y cannot migrate into testing. But if x starts in testing, then y may be able to migrate. This can happen if x migrated to testing before an RC bug was filed against it.

(Dis-)Proving my claim is an exercise left for the reader.

Posted in Debian, Release-Team | 3 Comments

Jessie is coming the 2015-04-25

Indeed, we settled on a release date for Jessie – and pretty quick too.  I sent out a poll on the 28th of March and yesterday, it was clear that the 25th of April was our release date. :)

With that said, we still have some items left that needs to be done.

  • Finishing the release notes.  This is mostly pending myself and a few others.
  • Translation of the release-notes.  I sent out a heads up earlier today about what sections I believe to be done.
  • The d-i team got another release planned as well.
  • All the RC bugs you can manage to fix before the 18th of April. :)
Posted in Debian, Release-Team | 7 Comments

Imminent steep decline in RC bugs affecting Jessie – need more RC bug fixes

Earlier today, I posted a mail to debian-devel about how approximately 25 RC bugs affecting Jessie have been unblocked.  As mentioned, I planned to age some of them.  The expected result is that about 18 of them will migrate tonight and the remaining 7 of them will migrate tomorrow night.

After that, there are no more RC bugs waiting for the RT to unblock them!  The only remaining item on the list is cgmanager, for which we are requesting a t-p-u (maintainer already contacted about it).  If you want a release sooner, please have a look at the list of remaining RC bugs or/and start testing upgrades.

In other news, the glibc regression got fixed.  The new version of glibc has already been approved by us.  It is now waiting for the debian-installer team to testing it and approve it.

Posted in Debian, Release-Team | 1 Comment