Unseen changes to lintian.d.o

We have been making a lot of minor changes to lintian.d.o and the underlying report framework. Most of them were hardly noticeable to the naked. In fact, I probably would not have spotted any of them, if I had not been involved in writing them.  Nonetheless, I felt like sharing them, so here goes.🙂

User “visible” changes:

In case you were wondering, the section title is partly a pun as half of these changes were intended to assist visually impaired users. They were triggered by me running into Sam Hartmann at DebConf16, where I asked him about how easy Debian’s websites were for blind people. Allegedly, we are generally doing quite good in his opinion (with one exception, for which Sam filed Bug#830213), which was a positive surprise for me.

On a related note: Thanks Luke Faraone and Asheesh Laroia for getting helping me started on these changes.🙂

Reporting framework / “Internal” changes:

With the last change + the “−−no−generate−reports” option, we were able to schedule lintian more frequently. Originally, lintian only ran once a day. With the “−−no−generate−reports“, we added a second run and with the last changes, we bumped it to 4 times a day. Unsurprisingly, it means that we are now reprocessing the archive a lot faster than previously.

All of the above is basically the all the note-worthy changes on the Lintian reporting framework since the Partial rewrite of lintian’s reporting setup (~1½ years ago).

Posted in Debian, Lintian | Leave a comment

debhelper 10 is now available

Today, debhelper 10 was uploaded to unstable and is coming to a mirror near you “really soon now”. The actual changes between version “9.20160814” and version “10” are rather modest. However, it does mark the completion of debhelper compat 10, which has been under way since early 2012.

Some highlights from compat 10 include:

  • The dh sequence in compat 10 automatically regenerate autotools files via dh_autoreconf
  • The dh sequence in compat 10 includes the dh-systemd debhelper utilities
  • dh sequencer based packages now defaults to building in parallel (i.e. “–parallel” is default in compat 10)
  • dh_installdeb now properly shell-escapes maintscript arguments.

For the full list of changes in compat 10, please review the contents of the debhelper(7) manpage. Beyond that, you may also want to upgrade your lintian to 2.5.47 as it is the first version that knows that compat 10 is stable.


Posted in Debhelper, Debian | Leave a comment

Selecting key packages via UDD

Thanks to Lucas Nussbaum, we now have a UDD script to filter/select key packages. Some example use cases:

Which key packages used compat 4?

# Data file compat-4-packages (one *source* package per line)
$ curl --silent --data-binary @compat-4-packages \

Also useful for things like bug#830997, which was my excuse for requesting this.🙂

Is package foo a key package (yet)?

$ is-key-pkg() { 
 RES=$(echo "$1" | curl --silent --data-binary @- \
 if [ "$RES" ]; then
   echo yes
   echo no
$ is-key-pkg bash
$ is-key-pkg mscgen
$ is-key-pkg NotAPackage


Above shell snippets might need tweaking for better error handling, etc.

Once again, thanks to Lucas for the server-side UDD script.🙂

Posted in Debian | 1 Comment

mips64el added to Debian testing

Today, we have completed our first Britney run with mips64el enabled in testing. 🙂

At the current time, the set of packages in mips64el are not very connected (and you probably cannot even install build-essential yet[1]). Hopefully this will change over the next few days. For now, Britney does not enforce installability of packages on mips64el in general, so do not expect the architecture to be stable at the moment.

Cheat sheet for package maintainers:

  • Issues with builds (only) on mips64el are not blockers for testing migration (nor RC yet).
    • Such bugs should be filed as “important” for now (unless they also affect a release architecture, in which case you should still make it an RC bug)
  • Your package be an older version on mips64el compared to other architectures.
  • Britney may decide to break your package on mips64el if it means something else can migrate on a release architecture.

We will slowly remove these special cases for mips64el as it matures in testing.


[1]  Update on this: mips64el currently does not have a libc library yet, so build-essential is definitely not installable at the moment.  It will hopefully migrate very soon.

Posted in Debian, Release-Team | 1 Comment

Anti-declarative packaging – top 15 build-helpers inserting maintscripts

Debian packages can run arbitrary code via “maintainer scripts” (sometimes shortened into “maintscripts”) during installation/removal etc. While they certainly have their use cases, their failure modes causes “exciting” bugs like “fails to install” or the dreaded “fails to remove”.

They also have other undesirable effects such as:

  • Bugs in/Updates to auto-generated snippets require a rebuild of all packages (not to mention the obvious code-duplication in all packages).
  • In case of circular dependencies[1] all having “postinst” scripts, dpkg will have to guess which package to configure first.
  • They require forking a shell at least once for each maintscript.
  • They complicate the implementations of e.g. detached chroot creation.

Accordingly, I think we should aim for a more declarative packaging style.  To help facilitate this, I have implemented 3 tracking tags in Lintian.

With these, we were able to learn that 73.5% of all packages do not have any of these scripts.  But I can now also produce a list of helpers that insert the most maintainer script snippets. The current top 15 is:

  1. “dhpython” with 3775 instances
    • This is an umbrella for all helpers using dh-python’s python module, see #827774.
  2. dh_installmenu with 1861 instances
  3. dh_makeshlibs with 1396 remaining instances
  4. dh_installinit with 1224 instances
  5. dh_python2 with 1168 instances
  6. dh_installdebconf with 772 instances
  7. dh_installdeb with 754 instances
    • These are the dpkg-maintscript-helper snippets for “rm_conffile”, “mv_conffile” etc.  Hopefully in the near future, dpkg will support these directly.
  8. dh_systemd_enable with 447 instances
  9. dh_installemacsen with 179 instances
  10. dh_icons with 165 instances
  11. dh_installtex with 137 instances
  12. dh_apache2 with 117 instances
  13. dh_installudev with 98 instances
  14. dh_installxfonts with 87 instances
  15. dh_systemd_start with 79 instances

With this list, it seems to me that some obvious focus areas would be:

  • Replacing the python scripts (I presume it is the byte-code handling, but I have not looked at this at all)
  • Migrating away from menu files
  • Support enabling + starting/stopping/restarting a service declaratively.
    • This might have a “hidden” requirement on declaratively creating service users if we want these packages to become truly “maintscript-less”.

Eventually we will also have to dig through all the “manual” maintainer scripts. But I think we got plenty to start with.🙂


[1] For some, circular dependencies in itself is an issue. I can certainly appreciate them as being suboptimal, but most of the issues we have are probably caused by insufficient tooling rather than a theoretical issue (that is, if we remove all postinst scripts).

Posted in Debhelper, Debian, Lintian | 3 Comments

Another Britney patchset

I just submitted another patch series to improve Britney for review.  If accepted, they will probably be merged into master within 2 weeks. The changes this time are probably most exciting for people that run/maintain Britney.  Key highlights include:

  • Britney will be able to use a regular mirror (without partial suites) as data source
    • Previously you would have to decompress and merge the Packages/Sources for each component.
    • Partial suite support is still not added, but I hope to add it eventually.  I know it is feature used by at least Ubuntu.
    • This change implies renaming some input files around (Dates, Urgency and BugsV files) as Britney expected these next to the Packages files.
  • More machine parsable facts added to “excuses.yaml”.  It will cover almost all excuses currently in use.
  • Britney will support two use cases for “faux packages” natively.
    • I hope to use this to eliminate our need to “injecting” fake packages into Britney’s data source.

I would like to dwell a moment on the “faux packages”.  We have had a helper script generate and inject fake packages into the list of packages (called “faux packages”).  They generally serve two purposes, which Britney will support:

  1. Whitelist of fake packages to satisfy dependencies of other packages.
    • These are generally stand-in for non-free machine configuration packages, where the end-user system would also fetch packages from the vendor’s repository.
    • Packages relying on “faux packages” are generally not in “main” as Debian’s main component is required to be self-contained.
    • These are (still) be called “faux packages” in/after the patch series
  2. Ensuring that certain packages are present and installable in testing.
    • We have a lot of d-i related packages here to avoid accidental breakage of d-i.
    • These are now referred to as a “constraint” (assuming there is no bike-shedding over the name).

Since Britney will now distinguish between these two use cases, I also make Britney enforce the second use case slightly better.  Mind you, it can still be overruled by force-hints and BREAK_ARCHES, so there still enough rope to hang yourself.


The other exciting part of this patch set (for me, at least) is that Britney will hopefully become simpler to deploy. No doubt there are still some missing features and paper cuts left, but I suspect we are not far from a:

  1. Fill out a template config file pointing Britney to your mirror
  2. Run britney -c britney.conf
  3. Make your archive kit update your target suite based on Britney’s output.
  4. Put step 2+3 in crontab/jenkins/task scheduler of choice
  5. Profit

There will certainly be some features that requires extra steps.  An example is the “anti rc-bugs regression” feature, which requires you to feed Britney with the list of RC bugs for your source and target suite. But even without, Britney would still protect your target suite from most installability issues.

Posted in Debian, Release-Team | 1 Comment

auto-decrufter in top 5 after 10 months

About 10 months ago, we enabled an auto-decrufter in dak.  Then after 3 months it had become the top 11th “remover”.  Today, there are only 3 humans left that have removed more packages than the auto-decrufter… impressively enough, one of them is not even an active FTP-master (anymore).  The current score board:

 5371 Luca Falavigna
 5121 Alexander Reichle-Schmehl
 4401 Ansgar Burchardt
 3928 DAK's auto-decrufter
 3257 Scott Kitterman
 2225 Joerg Jaspert
 1983 James Troup
 1793 Torsten Werner
 1025 Jeroen van Wolffelaar
  763 Ryan Murray

For comparison, here is the number removals by year for the past 6 years:

 5103 2011
 2765 2012
 3342 2013
 3394 2014
 3766 2015  (1842 removed by auto-decrufter)
 2845 2016  (2086 removed by auto-decrufter)

Which tells us that in 2015, the FTP masters and the decrufter performed on average over 10 removals a day.  And by the looks of it, 2016 will surpass that.  Of course, the auto-decrufter has a tendency to increase the number of removed items since it is an advocate of “remove early, remove often!”.🙂


Data is from https://ftp-master.debian.org/removals-full.txt.  Scoreboard computed as:

  grep ftpmaster: removals-full.txt | \
   perl -pe 's/.*ftpmaster:\s+//; s/\]$//;' | \
   sort | uniq -c | sort --numeric --reverse | head -n10

Removals by year computed as:

 grep ftpmaster: removals-full.txt | \
   perl -pe 's/.* (\d{4}) \d{2}:\d{2}:\d{2}.*/$1/' | uniq -c | tail -n6

(yes, both could be done with fewer commands)

Posted in Debian | Leave a comment