Performance tuning of lintian

For quite a while, Lintian has been able to create performance logs (--perf-debug --perf-output perf.log) that help diagnose where lintian spends most of its runtime.  I decided to make lintian output these logs on to help us spot performance issues, though I have not been very good at analysing them regularly.

At the beginning of the month, I finally got around to look a bit into one of them.  My findings on IRC triggered Mattia Rizzolo to create this following graph.  It shows the accumulated runtime of each check/collection measured in seconds.  Find these findings, we set out to solve some of the issues.  This lead to the following changes in 2.5.33 (in no particular order):

  • Increased buffer size in check/ to reduce overhead [bc8b3e5] (S)
  • Reduced overhead in strings(1) extraction [b058fef] (P)
  • Reduced overhead in spell-checking [b824170] (S)
    • Also improves the performance of spellintian!
  • Removed a high overhead check that did not work [2c7b922] (P)

Legend: S: run single threaded (1:1 performance improvement).  P: run in parallel.

Overall, I doubt the changes will give a revolutionary change in speed, but it should improve the 3rd, 4th and 5th slowest parts in Lintian.

Beyond runtime performance, we got a few memory optimisations in the pipeline for Lintian 2.5.34:

  • Remove member from “non-dir” nodes in the Lintian path graph (2%) [6365635]
  • Remove two fields at the price of computing them as needed (~5%) [a696197 + 8dacc8e]
  • Merge 4 fields into 1 (~8%) [5d49cd2 + fb074e4]
  • Share some memory between package-based caches (18%) [ffc7174]

Combined these 6 commits reduce memory consumption in caches by ~33% compared to 2.5.33, when lintian processes itself.  In absolute numbers, we are talking about a drop from 12.53MB to 8.48MB.  The mileages can certainly vary depending on the package (mscgen “only” saw an ~25% improvement).  Nevertheless, I was happy to list #715035 as being closed in 2.5.34. 🙂

This entry was posted in Debian, Lintian. Bookmark the permalink.

2 Responses to Performance tuning of lintian

  1. Pingback: Performance tuning of lintian, take 2 | nthykier

  2. Pingback: Performance tuning of lintian, take 3 | nthykier

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s