Mastodon hook for dput-ng

If you use dput-ng, you may be familiar with the Twitter hook that tweets a message when uploading a package.

A similar hook is now available for Mastodon too; if interested, give it a try and comment on the MR

Gmail and tracker.debian.org: add a label to team emails

Recently the Python Team started the migration from Alioth mailing list to a tracker.debian.org team.

Alioth mailing list, being a real ml, included a List-Id header, which is handled by Gmail natively and which you can use to create a filter based on it.

Tracker.d.o instead uses a custom header, X-Distro-Tracker-Team, and Gmail doesn't allow to create filters on custom headers. But there's a solution: Google Apps Scripts.

1. Apps script and its configuration

To create a new apps script, click "New project" from the scripts dashboard: this will open a code editor, so now cut and paste this code (change the label name to match the one you want to use, and the header too if you're going to use this for another team):

function tagPythonTeamEmails() {
  
  // search for all recent threads (hours is the smallest interval searchable)
  var threads = GmailApp.search("newer_than:1h");
  // this is the label i want to apply to the messages
  var label = GmailApp.getUserLabelByName("Debian_Python_Pkgs");  // <-- EDIT THIS

  for (var i = 0; i < threads.length; i++) {
     
    // get all messages in a given thread
    var messages = threads[i].getMessages();

    // for each message in the thread
    for (var j = 0; j < messages.length; j++) {
      var message = messages[j];
      // needed to have access to the full email: headers + body
      var email = message.getRawContent();
      // this is the header i want to check for: if present, add label and archive
      if (email.indexOf("X-Distro-Tracker-Team: python") > -1) {  // <-- EDIT THIS
        threads[i].addLabel(label);
        threads[i].moveToArchive();
      }
    }
  }
}

(This code is largely inspired by this StackOverflow answer.)

Given we're going to perform changes to the emails, we need to add extra permissions to the script; first we need to toggle the option "View > Show manifest file" which will show the appsscript.json file. Once done, edit it and update the scope of the script to include:

  "oauthScopes": [
    ...
    "https://www.googleapis.com/auth/gmail.readonly",
    "https://www.googleapis.com/auth/gmail.modify"
  ],

These new permission will be granted when we deploy the script, which we can actually do right now: from the editor window click on "Publish > Deploy from manifest..." and then "Install add-on"; it will make the app script available to operate on Gmail by opening the usual Google grants authorization popup.

2. Schedule the app script execution

Unlike filters defined on the web UI, apps scripts are not triggered every time a mail is received (there's a pub-sub mechanism, but it doesn't look like it would work for this), so if we want to execute periodically the new script, we need to create a time-driven trigger.

The simplest way is probably from the script editor (but you can also follow this doc): "View > Current project's triggers" and then "Add trigger" from there. For my specific use-case i set:

• event source = time-driven
• type of time based trigger = minutes timer
• minute interval = 5 minutes
• Failure notification settings = notify me immediately

it's not great that you have to wait 5 minutes for getting proper labels, and while it's true you can schedule the trigger for every minute, there are further considerations for that, discussed below.

3. Additional information

Be aware of quotas and limitations for Apps scripts; in particular the "Triggers total runtime" set to "90 min / day" is what led me to set the trigger every 510 mins.

Use the scripts dashboard (doc) to monitor the script execution logs, in the "My Executions" section, and specifically the "Duration" column: the script execution time depends exclusively on the number of emails to process at every execution, so fine-tune the trigger time to stay within your quota.

The newer_than search operator doesn't allow a granularity smaller than hours (although that's not even documented), so that means we're going to scan multiple times the same emails.

UPDATE (2020-12-26): I've already had to bump from every 5 to every 10 minutes, as otherwise i'd get Exception: Service invoked too many times for one day: gmail. (remember the quota!)

Debian source package name from the binary name

it looks like i forgot all the times how to do that, and apparently i'm not able to use google good enough to find it out quickly, let's write down one way to get the source package name from the binary package name:

dpkg-query -W -f='${source:package}\n' <list of bin pkgs>

(since it accepts a list of packages, you can xargs it).

there are probably another million ways to do that, so dont be shy and comment this post if you want to share your method

CFEngine: upgrade Debian packages

say you use CFEngine to install Debian packages on your server, so it's likely you'll have a bundle looking like this:

bundle agent agentname
{
    vars:

        "packages" slist => {
                             "pkg1",
                             "pkg2",
                             "pkg3"
                            };

    packages:

        "$(packages)"
            package_policy => "addupdate",
            package_method => apt_get;

}

this works great to guarantee those packages are installed, but if a newer version is available in the repositories, that wont be installed. If you want CFEngine to do that too, then the web suggests this trick:

    packages:

        "$(packages)"
            package_policy => "addupdate",
            package_version => "999999999",
            package_method => apt_get;

which tweak the install system declaring that you want to install version 999999999 of each package, so if you have available a higher version than the one installed, CFEngine will happily upgrade it for you. It works great.. but sometimes it doesn't. why oh why?

That's because Debian versions can have a epoch: every plain version (like 1.0-1) has an implicit epoch of 0, and same goes for the 999999999 above, that means if any of the installed packages has an epoch, that version will sort higher than 999999999 and the package wont be upgraded. If you want to be sure to upgrade every package, then the right solution is:

    packages:

        "$(packages)"
            package_policy => "addupdate",
            package_version => "9:999999999",
            package_method => apt_get;

Start a MySQL instance to run Debian package build-time tests

I'm the maintainer of MySQL Connector/Python Debian package, and it's shipped with several unittests, but they require a MySQL instance to be run against, so up to know I wasn't able to run them at build-time.

But I want to change that, so I wanted to find a way to spawn a MySQL instance just for testing purposes. I've searched a lot, and then found a package solving the same problem: php5.

The package has a setup-mysql.sh script that takes care of starting/stopping the instances, and debian/rules file sets up instance parameters and then it starts the instance before the tests and then stops it after them.

It's a very interesting solution, so I thought of sharing + taking notes to self, just in case I need it in other situations.

bts-link has a new home!

Since (already) several weeks, bts-link has a new home: busoni.debian.org! That's one of the machines where BTS is running, and it helps to let bts-link complete its job in about 30 mins each time. Thanks goes to Don Armstrong and Peter Palfrader for their support in the migration (from merkel).

Once again, let me encourage you to contact the bts-link developers mailing-list to request your project to be added to the tool in case it's not (we usually do some checks and adds from time to time, but if you notify us, you'll get it faster :) ).

Scribe (& thrift) for Debian Squeeze

Some time ago, I asked what are the common solutions for services remote logging. It was suggested scribe, an application log server developed at Facebook and later released as an open source project, and we finally managed to get this ready for Debian squeeze.

Scribe requires a couple of other tools to work: thrift, a framework for services development (born at Facebook too), and thrift-fb303, a piece of thrift used as a bassline layer for Facebook services.

Searching the web for some work-in-progress of a Debian packaging, I found the wonderful work done by SimpleGeo; since I have some changes to do (and also take the occasion to play a bit with github), I forked their repos in these ones: scribe, thrift and thrift-fb303.

I've prepared a repository with those 3 packages, if you would like to try scribe, give it a look. They are for squeeze because, well, after all I need it for work and since it requires Python 2.6, using squeeze was the easiest choice (backporting 2.6 to lenny could be a mess, even if there are packages out there) and so I could also see how the upgrade from lenny would be.

Let's now dive into the doc, and some examples too, to find out the several configuration options and decide what's best for us.

URL shortcuts in Chromium

If you like the Firefox feature to create special bookmarks that are actual templates for specific urls, well, here's how I got the same for Chromium.

That's actually how WordReference is suggesting, so it's defining new search engines: right click on the URL bar, Edit seach engine, and then Add; now you can define what you like, f.e. :

and so now typing "pts package" in the URL bar you'll get to the package PTS page.

That's one of the core points I had to get fixed in order to migrate from FF to chromium, one step closer then.

select count(bugs) in reportbug where status is open = 91

well, the actual select (against UDD) would be

udd=> select source, all_bugs from bugs_count where source = 'reportbug';
source   | all_bugs
----------+----------
reportbug |       91

but the result is the same: less than 100 bugs against reportbug! Yay!

As you can see from the bugs graph, the Christmas and Easter holidays helped quite a lot to squash those bugs. When I first step into the reportbug world, the bugs count was around 250, and I first imagined in my mind to brake the 200 barrier, than the 150 and finally the less-than-100 one: while the first 2 limits where the hardest one (new to the codebase, other interest) the last one is the one I'm most proud of.

• use SOAP to query the BTS (no comment here please...)
• use the new.822 to query NEW or incoming and all other sub-queues
• a proper unittest (and I hope the talk @ PyCon Italia will help on this) and check suite
• code refactoring and clean-up of the several dead-branches laying in it
• several other smaller things but still interesting

They seem quite nice-to-have stuff but recently I've tried to concentrate on reply fast to new report, fix as many reported crashes as possible and implement some of the long-awaited features.

Additionally, sometimes "No, sorry" is the proper reply to a bug and then it can be closed: it's hypocrite to keep bugs opened while internally you think it's a wontfix but you're not closing them because of a fear of possibly upset the reporters.

There will be more in the future, that's for sure, but I think this is the reportbug we'll see in squeeze, modulo some new crashes or very interesting feature we want it, so expect minor releases from now until squeeze+1

Debian lists spam: ALL messages reviewed!!!

Holy crap! I don't know how I made it, but my mailing-list spam review page is EMPTY!! It was a long way, started on 2009-04-05, but at least the biggest part is done (now there are only the new reports, much much less than the whole backlog).

Again, I'd like to invite anyone to contribute to have (an ideal) lists.d.o spam free:

• any non-DD can report spam messages clicking on the button on the message page on lists.d.o
• any DD can start here and then contribute reviewing the received reports.

I know it seems boring at first (and indeed it is), but we are many people, and anyonw contributes a little, it will be much faster and less-annoying. For example: why not review some message while waiting for your next package to be built? Or while you wait for the dist-upgrade to end? Or in any other situation where you got some minutes of spare time to dedicate to such an easy task?

Some links to those interested:

• Listarchive spam review main page
  
• wiki page
• spam review stats

Have fun!

Reportbug has now less than 150 outstanding bugs!

Yes, finally we made it! Thanks in particular to Carl Chenet and to Luca Bruno we were able to reduce the reportbug bugs count below 150!

It was a very long road (when I joined the team there was 230-something bugs opened) and I hoped to achieve this some time ago, but the fought goals are the most relevant ones.

Of course, the just released version 4.9 contains quite some bug fixes and enhancements: if you have the chance, give it a look and report any regression or feature request you see fit, but not too much: I want to enjoy the barrier just broken :)

Goodbye MIA team

I can't simply put up any longer with all the shit thrown on the face being unable to do punish such bad behavior. There are just so many other things that increase my anger (like, for example, "WTH is python2.6 still not in unstable?!") that I don't need another one.

For sure, I don't advocate or encourage anyone to join the MIA team, and I'm speaking with a lot of experience... There is a proposal, from zack, to restructure how MIA works (another bright example of collaboration...): when will it be implemented? nobody knows. will it help? I don't think so (much).

My work (MIA work in general) was almost invisible, like invisible were all the harsh and aggressive replies I got. The nice part is that those replies came from people with the worst situation. Something like:

• me: "this package has no uploads since 2005", reply: "this package has no upstream release since 1983" - 1. probably is to remove? 2. what about all the bugs filed against the *Debian* package? 3. what about the gazillion of lintian errors and warnings on that package? 4. MIA doesn't contact about only 1 package (so maybe there's something more?)...
• me: "one of your packages has 4 RC bugs", reply: "it's team maintainer!!" - yeah, so let's pretend that since it's team-maintained 'someone else' (that's never you) will work on it.
  

and several other non-sense replies doesn't leave anything but a huge frustration in you.

And when you got really pissed off, and want to see some sort of reaction on such bad and unmotivated attacks and ask the "People with Powers", what do you got? "sorry, we can't do anything to help you, it's your problem". Something should be done to protect people from fuckheads, make them behave, or else kick them out of the project as far away as we can. But what we got is nothing (and nothing seems to be changing on this side).

Well, now it's not my problem anymore, it's someone else: in perfect Debian-style.

Debian Whiteboard

How cool is that? Did you already tried it? if so, you already know, if not, well go and play with it soon!

Kumar introduced me (thanks!) to it, and now when I need to collaboratively edit a text, I do with that.

Things that make me angry

• when people upload to Debian with an @ubuntu.com email address
• even worst, if they do that when they are DDs (so with a @debian.org address)

grrr

Anniversaries

A couple of important anniversaries in the last few days:

• 2008-09-16 - I became a DD! A lot of things were done in this year, but not as much as I'd like to (in particular because of this baby: it's 9 months now :) ), but I hope I'll be able to catch up really soon now;
  
• 2008-09-23 - I had the scooter accident. Even if a year has passed, my knee still gives me pain (in particular when swimming), muscles are still really ipotonic, and the whole articulation is still "strange". They told me that's the way to go, but I hoped it would have been over by now. Never a joy.
  

My dear iceweasel,

WTH are you doing with all this memory?

$ top -b -n 1 | egrep "xulrunner|PID"
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 3907 morph     20   0 2349m 859m  15m S   14 21.7   2013:33 xulrunner-stub

say goodbye, you're going to die; mew

Mid 2009 bts-link update

Several months had passed since I started acting bts-link maintainer and I've never issued a "state of the union" message, so I'm fixing it now :)

I'm going to enlist some of the most relevant facts that have happened:

• bts-link now runs on merkel, that will help it to be identified as an official Debian tool, in particular from upstream projects that doesn't like to be "flooded" by a pick of requests;
• it's executed almost always twice a week (this also reduce load on remote issue tracking systems); now I tried to standardize on Monday and Thursday, but since it's still executed by hand, days may change;
• we now have a log of each execution, and we added to it several additional information to help use drive it the right way;
• the logs are publicly available (even if a bit hidden until now) here;
• we created a scripts to parse those logs and create a couple of graphs (using matplotlib) for each execution, available at the same location of logs; they show us the summary of all logged bts-link executions and (mainly) what are the projects with a lot of forwarded bugs that are still not handled by bts-link (images links are from last run);
• as you can see from the summary graph, the Unconfigured forwarded bugs count is drastically reduced; this is due to mainly 2 reasons: 1. we recently added Roundup (python, gnupg, darcs, mercurial, etc) and Google Code (tesseract, and many small others) issue tracking systems support; 2. we added several missing project for remote BTSes already supported.
• Recap - we are now able to configure packages with remote issue tracking system that uses these tools: berlios, bugzilla, gnats, launchpad, mantis, rt, savane, sourceforge, trac, gforge, googlecode, roundup.
• we have a really active contributor: Olivier Berger. Thanks a lot, Olivier! But we want mooooore;
• we got also a changelog.
  

It's a lot, but there is still more to do:

• add more remote issue tracking systems: php is the next big one, and meebey just told me it has a XMLRPC backend, so it's an easy one; let's hope MySQL has the same XMLRPC enabled, since they based their BTS on php one;
• add more Debian packages under bts-link monitoring: those are packages with forwarded bugs to upstreams with a supported remote BTSes - here we need your help!
• make bts-link more a general purpose tool, not Debian specific - and here Olivier is working really hard, let's hope I can help him achieve this (for example reviewing and merging his changes ;) ).
• have a more suitable summary output, with elapsed time, already grouped up information from results, ready to be parsed and graphed; something is already done, but some code refactoring is needed to complete it;
• [long term] convert it to "some sort of" daemon; run executed by cron is a beginning, but there were talks about a real daemon, continuously checking forwarded bugs, with a reasonable amount of delay to not bother that much on BTSes;
• [long term] integrated with Debian bts;
• we got also a TODO list.
  

What can you do for bts-link? a lot!

• do you maintain a package with forwarded bugs to a supported remote issue tracking system not yet checked by bts-link? let us know, and we'll add it;
• do you maintain a package with forwarded bugs to a not yet supported remote issue tracking system? let us know, and we'll add it (it's welcome if you already contact upstream to alert them of the possibility of automatic tool checking their BTS and to find out if their BTS support some sort of programmatic querying, like CSV exporting, XMLRPC, etc.);
• do you want to help? contact us, there's always something to do :)

Have fun!

Bts-link support Google Code issue tracker

Since this afternoon, bts-link is able to interact with Google Code issue tracker.

So, if you're upstream code & bugs are maintainer there and what to keep them monitored, just drop us an email to the devel mailing list (see link above).

You can also write us if your tool uses another upstream issue tracking still not managed by bts-link and we will see what we can do.

Reportbug: Where we're now, where we're headed

Lately, it's being an active period for Reportbug, and I'm happy about that.

As you might have noticed, this is a quite busy period for me due to non-Debian stuff, so I had to freeze a lot of Debian activities. Due to this, also several projects for Reportbug enhancements have stopped.

In particular for SOAP BTS interrogation. It's a nice topic, and I'd like to guide it myself, but we'll talk about it from August/September ;)

In the meantime, Carl is flooding us with his stream of patches, and it's hard to keep up with that, and Luca is working on making Gtk+ backend more stable and usable and we're thrilled a big number of people tried it and reported bugs (damn, I keep forgetting to disable 'reportbug reportbug' !! ;) ).

So, have fun using reportbug, don't be scare to report bug on it, and if you have some time to devote to it and want to send use patches, you know you'll be welcome :)

UPDATE: I forgot to add that the total bugs count is now at the lower point since I entered reportbug team!

Use a MUA? Can write plugins for it? Hate spam? Read on!

We all hate the spam that come thru the Debian MLs. Wouldn't it be cool to be able to report it directly from our MUAs instead of going to the mail webpage on lists.debian.org ?

Well, Cord kindly created an API (invoked using an URL) to do exactly this. What we are needing now is people that are able to write plugins to MUAs that can call this url.

I've then started a wiki page to keep track of requests/status of those plugins.

If you can write them, and you want Debian lists be a "cleaner" place, please give us your help!