A. Kaihola's Pocket Fluff

YouTube archiver

2008-09-18T22:20:00.005+03:00

Here's a fun experiment with combining various tools to easily make my MythTV box download YouTube videos with one click in the browser on another computer.

Proceeding from client to server, the first component is a Firefox bookmarklet for sending the URL of the current page to the MythTV box:

javascript:(function(){window.open('http://mythtvbox.mydomain.com:12345/'+window.location);})()

On the MythTV box I have a Python script listening for incoming HTTP requests on port 12345. The script uses twisted.web, so apt-get python-twisted-web (on Debian-based distros) is needed.

#!/usr/bin/python

from twisted.web import server, resource
from twisted.internet import reactor

class DownloadRequestHandler(resource.Resource):
  isLeaf = True
  def render_GET(self, request):
      try:
          video_id = request.args['v'][0]
      except:
          msg = "Can't parse URL %r" % request.path
          return msg
      file('/home/akaihola/.download-queue/youtube.com/%s' % video_id, 'w').close()
      return 'Downloading YouTube %s' % video_id

site = server.Site(DownloadRequestHandler())
reactor.listenTCP(12345, site)
reactor.run()

At this point, whenever I select the "Download YouTube" bookmarklet while viewing a YouTube video, a file named after the ID of the video magically appears on the MythTV box.

For actual downloading of the content I use the youtube-dl script. The following bash script downloads content as long as files corresponding to video IDs exist, deletes those files after downloading, and waits for new files to appear. Waiting for new files is implemented efficiently using the inotifywait tool (apt-get inotify-tools on Debian-based distros). Only a small slice of my bandwidth is allowed for downloads during daytime, and in the night it can freely saturate the connection.

#!/bin/bash
QUEUEDIR=/home/akaihola/.download-queue/youtube.com
TEMPLATE="/var/lib/mythtv/videos/%(stitle)s-%(id)s.%(ext)s"
cd $QUEUEDIR
while true; do
while [ "`ls`" ]; do
  VIDEO=`ls -1|head -1`
  RATE=$( [ `date +%H` -gt 6 ] && echo "-r 30k" )
  ~/bin/youtube-dl -b $RATE -o $TEMPLATE http://youtube.com/watch?v=$VIDEO
  rm $VIDEO
done
inotifywait -e create $QUEUEDIR
done

Both scripts are running as services in the background on the MythTV server ready to queue and download videos. The videos appear in the "Watch Videos" menu of the MythTV box ready for enjoyment.

Voilà!* It works! Improvements appreciated.

*or "Viola!" as I've seen many native English speakers happily saluting their favorite string instrument after achieving a desired goal

Update: Insufficient bash experience. Wouldn't work for >1 files in the queue. Fixed now, but [ "`ls`" ] feels like a bit awkward way to test if a directory is not empty.

Reason for Dovecot's memory hunger

2008-09-15T09:25:00.002+03:00

One runs into strange things when using rare combinations of software. I've been wondering why the Dovecot IMAP process on our mailserver grows to 50 or 60 MB when I'm syncing my mailbox with OfflineIMAP. It turns out the reason is in how OfflineIMAP identifies messages and how Dovecot's caching reacts to that.

I won't pretend I understand the technical details, but here's the bug report and some discussion. The quick solution is to delete overgrown dovecot index/cache files for any large folders to be synchronized. Dovecot 1.1's UIDPLUS support may also help, but that version isn't available for Etch even in the backports repository, so I'm not trying it any time soon.

Why Scribus fails to import Sibelius music

2007-10-03T20:23:00.000+03:00

I finally found an explanation for why I haven't been able to import EPS files exported from Sibelius (version 2) into Scribus. The reason is they use Type 3 fonts. See this message thread for more information and some possible solutions.

Goodie for your PDF toolbox: automatic cropping tool

2007-09-26T18:30:00.001+03:00

Ok so I couldn't find a PDF auto-cropping tool for Unix anywhere. Except for a csh shell script which worked fine but which I didn't like too much. So I rolled my sleeves and created my own in Python. Here it is:

#!/usr/bin/python

import re, sys
from subprocess import Popen, PIPE

bbsub = re.compile('%%BoundingBox: [\d ]+\n')

def fixbbox(pdfpath):
badeps = Popen('pdftops -eps %s -' % pdfpath,
               shell=True, stdout=PIPE).stdout.read()
gs = Popen('gs -sDEVICE=bbox -dNOPAUSE -dBATCH -',
           shell=True, stdin=PIPE, stderr=PIPE)
gs.stdin.write(badeps)
gs.stdin.close()
bbox = gs.stderr.read()
goodeps = bbsub.sub(bbox, badeps)
epstopdf = Popen('epstopdf --filter --outfile=%s' % pdfpath,
                 shell=True, stdin=PIPE)
epstopdf.stdin.write(goodeps)
epstopdf.stdin.close()


if __name__ == '__main__':
for pdfpath in sys.argv[1:]:
    fixbbox(pdfpath)

en-dash and em-dash in Emacs

2007-04-14T10:36:00.000+03:00

Finally! I found them both! The em-dash can be typed in Emacs (on Ubuntu Edgy) with the key sequence Compose - - - (that's three hyphens), and the en-dash with Compose - - . (two hyphens and a period).

The en- and em-dashes seem to look identical when using Bitstream Vera Sans Mono, which is annoying but understandable in a monospace font, but at least I can now type them.

And where's the Compose key, you ask. Well, in addition to other peculiar keyboard customizations, I've re-assigned Compose to the context menu key. On my Dell laptop it's above F9, and on most full-size international keyboards it's next to Alt Gr. The XModmap rule for this trick is keycode 117 = Multi_key.

Now, if someone fixed copy-pasting extended characters from terminal windows to Emacs... And where's the full list of compose key sequences?

LaTeX index entries inside footnotes fail

2007-04-10T00:02:00.000+03:00

Inside LaTeX footnotes, \index entries which contain special characters or commands may appear multiple times in the index. This can happen when \makeidx inserts extra spaces in the markup which does not happen with \index in normal text.

I haven't found a good solution nor really understood what's going on. Let this post act as a reminder for myself in case I run into the same problem later and have forgotten details about it. If you know a good solution, please do send a comment!

Here's a simple test case:

% do e.g. the following:
% pdflatex test.tex ; makeindex test.idx ; pdflatex test.tex ; evince test.pdf
% and take a close look at test.idx to see where the problem is

\documentclass{article}
\usepackage{makeidx}
\makeindex
\begin{document}
First index entry is here\index{Index Entry@\emph{Index Entry}}.
\footnote[1]{Second index entry is here\index{Index Entry@\emph{Index Entry}}}
\printindex
\end{document}

And here's what ends up in test.idx:

\indexentry{Index Entry@\emph{Index Entry}}{1}
\indexentry{Index Entry@\emph {Index Entry}}{1}

This is how the index then looks like:

Index Entry, 1
Index Entry, 1

Here's some insight about what is going on. It isn't clear to me what Bernd is suggesting in his "Quick&Dirty" advice.

Django newforms improvement

2007-02-28T10:01:00.000+02:00

It's not the first time during my acquaintance with Django that I've needed functionality not available in the forms (and old manipulators) machinery.

Actually the need is quite simple: Only edit a subset of a model's field in a form, and keep old values for the rest.

For example, I might have a simplified form for changing only the content of a FlatPage, but keeping the title, url and other properties. In this case I'd like to be able to just say

FlatPageForm = form_for_instance(
myflatpage,
include_fields=['content'])

and have Django automagically fill in values from the original myflatpage object when saving form data for all fields except content.

There was a good discussion about this on the #django IRC channel, and eventually I implemented this functionality based on newforms. The module is available in our Subversion repository.

Database conversion Django style

2006-11-30T01:14:00.000+02:00

For various reasons, I wanted to convert one of my Django projects to use SQLite instead of PostgreSQL. After searching for a conversion tool and not finding one, I asked on the #django IRC channel where ubernostrum came up with this plan:

simple process:
1) Select everything.
2) Pickle it.
3) Save to file.
4) Read file.
5) Unpickle.
6) Save to db.
:)
Or something like that.

First I thought it was funny, but then started to think about it and it made perfect sense. Off to coding then.

(fast forward while I'm writing 11 lines of Python code)

Yes, that's right, after eleven lines of Python statements I had a script which pickles the whole Django database into a file, loads it back and saves all the objects!

I actually used more time googling for conversion tools than writing this tool. And since it was so easy, I thought I'll be nice and write some comments, a nice command line interface and sprinkle some debug output for verbose mode. And of course create this blog posting.

The script is available at our Subversion server. Point your browser or svn client to our repository and enjoy.

Update: Now the script handles ForeignKeys as well, and makes sure related objects are saved first. The length of the script went up from 11 lines of course...

Update: I just added many-to-many relation support.

Update: You can now use plugins to hook into the retrieval and saving of objects. This is useful for database schema migration.

Enjoying the web on GPRS while minimizing traffic (or even off-line), trick #1

2006-09-13T09:45:00.000+03:00

This summer I spent quite a lot of time at a summer cottage without broadband, but still had to work on the laptop from time to time. I can get on-line with GPRS on my mobile phone, but that's slow and expensive.

While coding, it's convenient to have on-line documentation and other resources at hand. I could of course download Python, Django and PHP documentation archives and browse them off-line on the hard disk, but often the on-line user comments are very useful, too.

Enter http-replicator.

Http-replicator is a handy Python script which acts as a HTTP proxy and cache. The special thing about it is that it stores all fetched pages as individual files in the filesystem, so it's easy to archive them or clean up some and retain other pages. The files are neatly arranged in directories named after the site domains and organized according to the URLs.

A nice plus is that it's very easy to sneak-peek at and rip CSS and JavaScript used on visited pages since I can immediately open them in an editor.

I keep http-replicator running in the background and run Firefox with a separate profile which uses the local http-replicator as proxy by default.

You can also run http-replicator in "static mode" so files that are present are served from cache directly without contacting the server. I use this to minimize traffic, but I specify some dynamic sites like on-line banking and webmail as exceptions in the Firefox proxy settings.

Here's my script on Linux for launching both http-replicator and Firefox with the special profile:


#!/bin/sh

/usr/bin/http-replicator -p 8086 -d /home/akaihola/http-cache -s --daemon --pid=/tmp/http-replicator.pid
MOZ_NO_REMOTE=1 /usr/bin/firefox -P cachefox
kill `cat /tmp/http-replicator.pid`

The "cachefox" profile simply sets Firefox to use localhost:8086 as the HTTP proxy. I've symlinked all files from my regular Firefox profile directory to the cachefox profile directory exept prefs.js so I don't have to worry about synchronizing bookmarks or extensions between the profiles.

I use this setup exclusively when on GPRS, and if I know beforehand that I'm going to need some pages, I run it already when still on broadband and simply visit them all. It's also possible to slurp whole sites with http_proxy="http://localhost:8086" wget --delete-after -r http://www.site.domain/ (I haven't tested this yet though).

Found use for podcasts today

2006-09-07T18:51:00.000+03:00

I know podcasts are no new thing anymore, but my first chance to try them only came with my new smartphone. It's a very good way to entertain myself while walking a sleeping baby in the park.

As always with new toys, it just doesn't work without technical problems. For some reason OggPlay (S60, Nokia 6680) won't continue playback of the Voice of S60 podcast (MP3) after rewinding or fast forwarding. I already made a note of the problem to the OggPlay forums, so let's see...
Now I just have to find the most interesting podcasts and a handy way to transfer them from my Linux laptop to the phone.

Tags, Thunderbird and Dovecot

2006-09-03T17:03:00.000+03:00

The newest snapshots of the Mozilla Thunderbird e-mail client have at last tag support. It's possible to tag messages with multiple keywords and do live searches based on the tags à la gmail.

There's just still something wrong - if I have a message tagged as "To Do" and try to remove all tags and re-tag it with another tag, all my changes are reverted after a moment. Maybe a problem with the Dovecot IMAP server? Oh well, instead of writing I should check bugzilla and maybe download a newer nightly.

Choosing a PHP web framework

2006-08-18T15:46:00.000+03:00

A client required their website to be done in PHP. Not being a great fan of the language since I discovered Python (and Django), I didn't like the idea too much. But it has to be done, so I set off to see what new tools have been created since I left the PHP world.

Based on other people's comments and without really trying out the many web frameworks available, I narrowed down my choices to Symfony, CakePHP and CodeIgniter. They seemed to represent the best of the breed for complete, medium and lightweight solutions, consecutively.

None of the PHP frameworks seemed to be as mature as Django, and from example code snippets the clumsiness of the language showed clearly compared to Python. But the frameworks seem to be well on their way and offer great benefits compared to bare-bones PHP programming (which I've done my share of in this life, thank you).

Anyway, among the three "finalists", CodeIgniter seemed to lack some essential features I need, namely user authentication and access control lists. Symfony is praised for its documentation but said to be big and difficult. Many good reviews have been written about CakePHP, so I started to work with it.

I'll write more about my impressions on CakePHP soon