Selecting key packages via UDD

Thanks to Lucas Nussbaum, we now have a UDD script to filter/select key packages. Some example use cases:

Which key packages used compat 4?

# Data file compat-4-packages (one *source* package per line)
$ curl --silent --data-binary @compat-4-packages \
  https://udd.debian.org/cgi-bin/select-key-packages.cgi
alsamixergui
apg
[...]
sgml-base
wwwconfig-common

Also useful for things like bug#830997, which was my excuse for requesting this.🙂

Is package foo a key package (yet)?

$ is-key-pkg() { 
 RES=$(echo "$1" | curl --silent --data-binary @- \
    https://udd.debian.org/cgi-bin/select-key-packages.cgi)
 if [ "$RES" ]; then
   echo yes
 else
   echo no
 fi
}
$ is-key-pkg bash
yes
$ is-key-pkg mscgen
no
$ is-key-pkg NotAPackage
no

 

Above shell snippets might need tweaking for better error handling, etc.

Once again, thanks to Lucas for the server-side UDD script.🙂

Posted in Debian | 1 Comment

mips64el added to Debian testing

Today, we have completed our first Britney run with mips64el enabled in testing. 🙂

At the current time, the set of packages in mips64el are not very connected (and you probably cannot even install build-essential yet[1]). Hopefully this will change over the next few days. For now, Britney does not enforce installability of packages on mips64el in general, so do not expect the architecture to be stable at the moment.

Cheat sheet for package maintainers:

  • Issues with builds (only) on mips64el are not blockers for testing migration (nor RC yet).
    • Such bugs should be filed as “important” for now (unless they also affect a release architecture, in which case you should still make it an RC bug)
  • Your package be an older version on mips64el compared to other architectures.
  • Britney may decide to break your package on mips64el if it means something else can migrate on a release architecture.

We will slowly remove these special cases for mips64el as it matures in testing.

 

[1]  Update on this: mips64el currently does not have a libc library yet, so build-essential is definitely not installable at the moment.  It will hopefully migrate very soon.

Posted in Debian, Release-Team | 1 Comment

Anti-declarative packaging – top 15 build-helpers inserting maintscripts

Debian packages can run arbitrary code via “maintainer scripts” (sometimes shortened into “maintscripts”) during installation/removal etc. While they certainly have their use cases, their failure modes causes “exciting” bugs like “fails to install” or the dreaded “fails to remove”.

They also have other undesirable effects such as:

  • Bugs in/Updates to auto-generated snippets require a rebuild of all packages (not to mention the obvious code-duplication in all packages).
  • In case of circular dependencies[1] all having “postinst” scripts, dpkg will have to guess which package to configure first.
  • They require forking a shell at least once for each maintscript.
  • They complicate the implementations of e.g. detached chroot creation.

Accordingly, I think we should aim for a more declarative packaging style.  To help facilitate this, I have implemented 3 tracking tags in Lintian.

With these, we were able to learn that 73.5% of all packages do not have any of these scripts.  But I can now also produce a list of helpers that insert the most maintainer script snippets. The current top 15 is:

  1. “dhpython” with 3775 instances
    • This is an umbrella for all helpers using dh-python’s python module, see #827774.
  2. dh_installmenu with 1861 instances
  3. dh_makeshlibs with 1396 remaining instances
  4. dh_installinit with 1224 instances
  5. dh_python2 with 1168 instances
  6. dh_installdebconf with 772 instances
  7. dh_installdeb with 754 instances
    • These are the dpkg-maintscript-helper snippets for “rm_conffile”, “mv_conffile” etc.  Hopefully in the near future, dpkg will support these directly.
  8. dh_systemd_enable with 447 instances
  9. dh_installemacsen with 179 instances
  10. dh_icons with 165 instances
  11. dh_installtex with 137 instances
  12. dh_apache2 with 117 instances
  13. dh_installudev with 98 instances
  14. dh_installxfonts with 87 instances
  15. dh_systemd_start with 79 instances

With this list, it seems to me that some obvious focus areas would be:

  • Replacing the python scripts (I presume it is the byte-code handling, but I have not looked at this at all)
  • Migrating away from menu files
  • Support enabling + starting/stopping/restarting a service declaratively.
    • This might have a “hidden” requirement on declaratively creating service users if we want these packages to become truly “maintscript-less”.

Eventually we will also have to dig through all the “manual” maintainer scripts. But I think we got plenty to start with.🙂

 

[1] For some, circular dependencies in itself is an issue. I can certainly appreciate them as being suboptimal, but most of the issues we have are probably caused by insufficient tooling rather than a theoretical issue (that is, if we remove all postinst scripts).

Posted in Debhelper, Debian, Lintian | 3 Comments

Another Britney patchset

I just submitted another patch series to improve Britney for review.  If accepted, they will probably be merged into master within 2 weeks. The changes this time are probably most exciting for people that run/maintain Britney.  Key highlights include:

  • Britney will be able to use a regular mirror (without partial suites) as data source
    • Previously you would have to decompress and merge the Packages/Sources for each component.
    • Partial suite support is still not added, but I hope to add it eventually.  I know it is feature used by at least Ubuntu.
    • This change implies renaming some input files around (Dates, Urgency and BugsV files) as Britney expected these next to the Packages files.
  • More machine parsable facts added to “excuses.yaml”.  It will cover almost all excuses currently in use.
  • Britney will support two use cases for “faux packages” natively.
    • I hope to use this to eliminate our need to “injecting” fake packages into Britney’s data source.

I would like to dwell a moment on the “faux packages”.  We have had a helper script generate and inject fake packages into the list of packages (called “faux packages”).  They generally serve two purposes, which Britney will support:

  1. Whitelist of fake packages to satisfy dependencies of other packages.
    • These are generally stand-in for non-free machine configuration packages, where the end-user system would also fetch packages from the vendor’s repository.
    • Packages relying on “faux packages” are generally not in “main” as Debian’s main component is required to be self-contained.
    • These are (still) be called “faux packages” in/after the patch series
  2. Ensuring that certain packages are present and installable in testing.
    • We have a lot of d-i related packages here to avoid accidental breakage of d-i.
    • These are now referred to as a “constraint” (assuming there is no bike-shedding over the name).

Since Britney will now distinguish between these two use cases, I also make Britney enforce the second use case slightly better.  Mind you, it can still be overruled by force-hints and BREAK_ARCHES, so there still enough rope to hang yourself.

 

The other exciting part of this patch set (for me, at least) is that Britney will hopefully become simpler to deploy. No doubt there are still some missing features and paper cuts left, but I suspect we are not far from a:

  1. Fill out a template config file pointing Britney to your mirror
  2. Run britney -c britney.conf
  3. Make your archive kit update your target suite based on Britney’s output.
  4. Put step 2+3 in crontab/jenkins/task scheduler of choice
  5. Profit

There will certainly be some features that requires extra steps.  An example is the “anti rc-bugs regression” feature, which requires you to feed Britney with the list of RC bugs for your source and target suite. But even without, Britney would still protect your target suite from most installability issues.

Posted in Debian, Release-Team | 1 Comment

auto-decrufter in top 5 after 10 months

About 10 months ago, we enabled an auto-decrufter in dak.  Then after 3 months it had become the top 11th “remover”.  Today, there are only 3 humans left that have removed more packages than the auto-decrufter… impressively enough, one of them is not even an active FTP-master (anymore).  The current score board:

 5371 Luca Falavigna
 5121 Alexander Reichle-Schmehl
 4401 Ansgar Burchardt
 3928 DAK's auto-decrufter
 3257 Scott Kitterman
 2225 Joerg Jaspert
 1983 James Troup
 1793 Torsten Werner
 1025 Jeroen van Wolffelaar
  763 Ryan Murray

For comparison, here is the number removals by year for the past 6 years:

 5103 2011
 2765 2012
 3342 2013
 3394 2014
 3766 2015  (1842 removed by auto-decrufter)
 2845 2016  (2086 removed by auto-decrufter)

Which tells us that in 2015, the FTP masters and the decrufter performed on average over 10 removals a day.  And by the looks of it, 2016 will surpass that.  Of course, the auto-decrufter has a tendency to increase the number of removed items since it is an advocate of “remove early, remove often!”.🙂

 

Data is from https://ftp-master.debian.org/removals-full.txt.  Scoreboard computed as:

  grep ftpmaster: removals-full.txt | \
   perl -pe 's/.*ftpmaster:\s+//; s/\]$//;' | \
   sort | uniq -c | sort --numeric --reverse | head -n10

Removals by year computed as:

 grep ftpmaster: removals-full.txt | \
   perl -pe 's/.* (\d{4}) \d{2}:\d{2}:\d{2}.*/$1/' | uniq -c | tail -n6

(yes, both could be done with fewer commands)

Posted in Debian | Leave a comment

Putting Debian packages in labelled boxes

Lintian 2.5.44 was released the other day and (to most) the most significant bug fix was probably that Lintian learned about Policy 3.9.8.  I would like to thank Axel Beckert for doing that.  Notably it also made me update the test suite so to make future policy releases less painful.

For others, it might be the fact that Lintian now accepts (valid) versioned provides (which seemed prudent now that Britney accepts them as well).  Newcomers might appreciate that we are giving a much more sensible warning when they have extra spaces in their changelog “sign off” line (rather than pretending it is an improper NMU).  But I digress…

 

What I am here to talk about is that Lintian 2.5.44 started classifying packages based on various “facts” or “properties”, we can determine.  Therefore:

  • Every package will have at least one tag now!
  • These labels are known as something called “classification tags”.
  • The tags are not issues to be fixed!  (I will repeat this later to ensure you get this point!)

Here are some of the “labelled boxes” your packages will be put into[0]:

The tags themselves are (as mentioned) mere classifications and their primary purpose is to classify or measure certain properties.  With them any body can download the data set and come with some bold statement about Debian packages (hopefully without relying too much on “lies, damned lies and statistics“).  Lets try that immediately!

  • Almost 75% of all Debian packages do not need to run arbitrary code doing installation[2]!
  • The “dh-sequencer” with cdbs is the future![3]

In the next release, we will also add tracking of auto-generated snippets from dh_*-tools.  Currently unversioned, but I hope to add versioning to that so we can find and rebuild packages that have been built with buggy autoscripts (like #788098)

If you want to see the classification tags for your package, please run lintian with like this:

# Add classification tags
$ lintian -L +classification <pkg-or-changes>
# Or if you want only classification tags$ lintian -L =classification <pkg-or-changes>

Please keep in mind that classification tags (“C”) are not issues in themselves. Lintian is simply attempting to add a visible indicator about a given “fact” or “property” in the package – nothing more, nothing less.

 

Future work – help (read: patches) welcome:

 

[0] Mind you, the reporting framework’s handling of these tags could certainly be improved.

[1] Please note how it distinguishes 1.0 into native and non-native based on whether the package has a diff.gz.  Presumably that can be exploited somehow …

[2] Disclaimer: At the time of writing, only ~80% of the archive have been processed.  This is computed as: NS / (NS + WS), where NS and WS are the number of unique packages with the tags “no-ctrl-scripts” and “ctrl-script” respectively.

[3] … or maybe not, but we got two packages classified as using both CDBS and the dh-sequencer.  I have not looked at it in detail. For the curious: libmecab-java and ctioga2.

Posted in Debian, Lintian | Leave a comment

Easter patching of Britney

I decided to take a couple of days of vacation next to Easter and obviously ended up with tons of time.  I ended up channelling most of the (productive) time into improving Britney.

In raw results:

  • I wrote about 35 patches during my (extended) Easter holiday + reviewed and merged/cherry-picked 2 patches from others.
    • Today, the “britney-fixes-2016-03” branch had 48 commits not yet in master (8 or so written before Easter).
  • I submitted 33 of the patches for review with the intention of merging them into master soon.
    • The rest will be bundled for a later round.

The most “exciting” items in the patch series are probably:

  • Support for versioned provides (#786803)
    • Admittedly, there is a complete punt on multi-arch’ified provides.
  • Avoid cruft re-entering testing to satisfy dependencies of other packages
  • First step towards supporting packages being read from a standard (dak-built) mirror.
    • Britney still assumes some data files are stored in the “mirror”.  Though it will hopefully work for derivatives/users that disables (read: patches out) the aging and RC bugs policies.
    • Future work include”partial” suite support and self-contained components.
  • A crash fix (#815995) that only occurs with package “hijacking” (i.e. multiple source packages building the same binary).

 

Once reviewed, these will be merged into master and we will have versioned provides support (in Britney).🙂

Posted in Debian, Release-Team | 1 Comment