Like a Bird on a Wire(shark)...

Over recent years, libferris has been using Qt to mount some Web stuff as a filesystem. I have a subclass of QIODevice which acts as an intermediary to allow one to write to a std::ostream and stream that data to the Web, over a POST for example. For those interested, that code is in Ferris/FerrisQt.cpp of the tarball. It's a bit of a shame that Qt heavy web code isn't in KIO or that the two virtual filesystems are not closer linked, but I digress.

I noticed a little while ago that cp to vimeo://upload didn't work anymore. I had earmarked that for fixing and recently got around to making that happen. It's always fun interacting with these Web APIs. Over the time I've found that Flickr sets the bar for well documented APIs that you can start to use if you have any clue about making GET and POST etc. At one stage google had documented their API in a way that you could never use it. I guess they have fixed that by now, but it did sort out the pretenders from those two could at least sniff HTTP and were determined to win. The vimeo documentation IIRC wasn't too bad when I added support to upload, but the docs have taken a turn for the worst it seems. Oh, one fun tip for the young players, when one API call says "great, thanks, well done, I've accepted your call" and then a subsequent one says "oh, a strange error has happened", you might like to assume that the previous call might not have been so great after all.

So I started tinkering around, adding oauth to the vimeo signup, and getting the getTicket call to work. Having getTicket working meant that my oauth signed call was accepted too. I then was then faced with the upload of the core data (which is normally done with a rather complex streaming POST), and the final I'm done, make it available call. On vimeo that last call seems to be two calls now, first a VerifyChunks call and then a Complete call.

So, first things first. To upload you call getTicket which gives you an endpoint that is an HTTP URL to send the actual video data to, as well as an upload ticket to identify the session. If you try to post to that endpoint URL and the POST converts the CGI parameters using multipart/form-data with boundaries into individual Content-Disposition: form-data elements, you loose. You have to have the ticket_id in the URL after the POST text in order to upload. One little trap.

So then I found that verifyChunks was returning Error 709 Access to the chunk list failed. And that was after the upload had been replied to with "OK. Thanks for the upload.". Oddly, I also noticed that the upload of video data would hang from time to time. So I let the shark out of the pen again, and found that vimeo would return it's "yep were done, all is well" response to the HTTP POST call at about 38-42kb into the data. Not so great.

Mangling the vimeo.php test they supply to upload with my oauth and libferris credentials I found that the POST had a header Expect: 100-continue. Right after the headers were sent vimeo gave the nod to continue, and then the POST body was sent. I assume that just ploughing through and giving the headers followed by the body confused the server end and thus it just said "yep, ok, thanks for the upload" and dropped the line. Then of course forgot the ticket_id because there was no data for it, so the verifyChunks got no chunk list and returned the strange error it did. mmm, hindsight!

So I ended up converting from the POST the newly available PUT method for upload. They call that their "streaming API" even though you can of course stream to a POST endpoint. You just need to frame the parameters and add the MIME tailer to the POST if you want to stream a large file that way. Using PUT I was then able to verify my chunks (or the one single chunk in fact) and the upload complete method worked again.

In the end I've added oauth to my vimeo mounting, many thanks to the creators of the QOAuth library!

KDE, Qt, and the segv

Short story: qt-4.8.4 on Fedora 16 might offer you crashes when clicking the menu in the panel. I had this on my main user account, but not another account. Downgrading to 4.8.3 solved this with the exact same .kde, .local et al in place.

The crashes offered my g_iterator type things in the backtrace. Nothing juicy that I could try to work backward from. Digging around seemed to show theme stuff, pointing at the nvidia drivers (which always comes up in searches), and there was a bug I found along the way mentioning qt 4.8.4, so I reverted to 4.8.3 from a mirror that had the old packages still around.

I decided to update the laptop before a trip on thursday, and this was of course when I discovered this little bundle of joy. I'm not sure which part of my config was causing this, but at least I can now use the laptop with my desktop of choice while away!

Foray into Qt Models with libferris

Version 1.5.2 of libferris, released yesterday, includes both a QAbstractTableModel and QSortFilterProxyModel targeted at exposing libferris to the world of all things Qt. In libferris a great amount of information is exposed as virtual Extended Attributes for a file. As you can see in the below shot, the md5 as well as "ls -lh" size is available directly as attributes. Before you wonder, the md5 is only calculated on demand and is then cached relative to the mtime of the file.



Each file has thousands of attributes on my machine, these include tagging, geospatial information, extracted metadata, and RDF. In my GTK+2 model/view pair I allow the user to "sort" on a column but really have the view sort on another one. For example, as you see below, sorting on size-human-readable either means Qt or some class you provide needs to know how to properly sort on size strings like 14.2k etc. Instead of this, the QSortFilterProxyModel converts some column names into others, so a size-human-readable sort becomes a sort on the "size" column. As the latter is simply a number Qt knows how to do the right thing (TM). Although both columns are shown in the example shot, there is no need to have the size column visible too.



The mtime is actually converted to a QDateTime by the model. It is stored internally as an epoch integer. Thus the mtime-display column was added in libferris which shows you a human readable version. This is handy for command line tools and others which do not want to know about presentation etc and just want something nice to show the user. Most of these attributes in libferris have schema information, which makes it possible to pass them on to Qt in a nice way. Tags should come through as checkboxes etc.

The next logical step is to get at this from QML. I'll wait until I update to Fedora 14 which has Qt 4.7 in it. With the model, and some way to play back audio I should be able to make an audio player in QML. Two things help here; searches can return their results as a virtual filesystem which can then be shown in the QAbstractTableModel, and filesystem "selections" can be built as filesystems which can also be then displayed in the same way. So I have search and playlist support already from that and ffmpeg or some low battery way to make ogg into air oscillation then completes the loop.

n900: The claw arrives!

Clawmotia is a MythTV remote control written using Qt and EFL technology. My repository contains a tarball ready to install clawmotia onto an n900. Desktop shortcut optional :)



Simply extract the tarball into /opt on the device. From there, you currently have to install the desktop file yourself to get it onto the menus. Finally, you'll need to edit start-clawmotia.sh to tell the application where your mythtv server is at and which client you want to control through that server. I plan to add dynamic client selection and a user configuration window in the app itself, but I seem to have more plans than the clock allows.

Looking for some coding work...

If you are on the lookout for a C++, GTK+2, Qt, Linux developer with a PhD and experience in persistent storage and indexing then I'm looking for you!

For those who don't know me, I am "the libferris guy". Which means I have played with virtual filesystems for many many years and implemented a good handful of ways to index and search diverse filesystems. Some of the articles I did for Linux Journal and linux.com will give an impression of the sorts of things libferris can do, and in the next months the Linux Format includes information on Web services as a filesystem, eg, mounting flickr, youtube, and facebook.

After hacking RDF support into KOffice earlier this year, I would very much like to work more with Soprano and RDF technologies. But anything involving indexing and storage is also of huge interest. In fact one of the things I like about RDF at the low level is that it overlaps these desires too, for example, my custom model partial implementation for Soprano which is targeted at great performance on embedded devices.

I have implemented some GIST trees for PostgreSQL and worked with C++ and development on a Linux platform for over 10 years. If you are interested in these things or just want a Qt hacker for a while then please drop me a line using the nick for this blog at gmail or sourceforge as email address.

Libferris: now and then...

Those who have been following libferris developments over the recent years will have noticed that I switched over to using Soprano for RDF support in 2009. I also added support for mounting relational databases (using QtSql) and converted the http and ftp mounting over to use Qt for network IO. This later point also extends to the support for mounting flickr, vimeo, google spreadsheets and docs etc. So Qt is slowly marching closer to the "kernel" of libferris.

I've been toying around with switching over some of the core parts of the libferris kernel to using Qt. Things like using Qt's smart pointers instead of the policy based ones from Modern C++ Design, switching over to using Qt signal/slots, and perhaps offering both iostream and QByteArray data access methods in parallel.

There are a number of "smaller" tasks to support this. For example, creating a stldb4 like library allowing access to Berkeley db4 and other ISAM files with a "Qt native" style API. Stldb4 only supports bdb files, whereas allowing other ISAM like gdbm, tdb, and WhateverDB to be accessed through the hypothetical QISAM library would be nice. If anyone has recommendations along these lines please drop me a line or leave a comment. I mention smaller in quotes above because I know this task is quite difficult to do well, things like ordering and prefix ordering of keys come up as well as support for secondary indexing and other goodies which are really needed to milk performance out of custom ISAM solutions.

The harder part would be porting over the core kernel of libferris to Qt stuff. As it would be extremely hard to do piecewise it would require a contiguous block of free time. It is also a fairly hard thing to try to get financial support for because the only "feature" it adds is the possibility of ports for other platforms. My point of mentioning this is that at some stage, libferris ports for Symbian and other Qt supported platforms will become much simpler. Finally I could "cp" an image directly from the camera of my s60 phone to flickr without the annoying third party apps.

As an aside, I was wondering how long it has been since I started using RDF on the desktop. The November 2003 version 1.1.12 release mentions mounting RDF... so it's been a while. RDF is extremely handy for separating data access from personal metadata. If you want to add an extended attribute to a read-only NFS share, with libferris and RDF you can, and you just use the same API as normal ;)

I give you the CLAW!

Clawmotia, an edje MythTV remote control.
Sources hot off the presses: clawmotia-0.0.1. You'll also notice the diablo binary for clawmotia on there so you don't need a scratchbox to get going. I put the clawmotia.edj up too so you don't even need to compile that if you don't wanna.

Dependencies on the device: Qt, qedje. You probably have most of the requirements for the latter installed if you have canola2 on the device. There are deb files for qedje on its web site. Qt is also packaged for the n810. I don't have debs for clawmotia yet, feel free to send the debian directory to do that.

You'll obviously need a MythTV server (and client) to use the remote. It uses the web interface to mythtv so you'll want that installed on the server machine too. The various configuration settings are passed in using environment variables in version 0.0.1. See the script start-clawmotia.sh for these settings. You will also need to be in the same directly as the .edj file when you start clawmotia as it expects that at this point. I install the script, binary and edj into /usr/local/bin on the device.

Setting CLAWMOTIA_FS=1 will start clawmotia full screen. There is a bug in version 0.0.1 where it will start but not be shown if started full screen. You have to switch to another app and back to clawmotia to see it. I was planning to fix this before release, but it seems some folks want to have a tinker with it so I'll fix that bug soon.

If you want to change the buttons or layout, you only need to edit the clawmotia.edc file. If you are good with the gimp, you might like to reate a theme at 800x480. As long as you use layers it should be a fairly mechanical process to create an edc file for your theme.

To make a button, define its image in the images section and call the three macros COMMAND_BUTTON, COMMAND_BUTTON_ICON, and COMMAND_PROGRAMS. COMMAND_BUTTON sets up the area of the screen that will react to your button with an optional text label for the button as arg2. COMMAND_BUTTON_ICON links a COMMAND_BUTTON with an image file. And COMMAND_PROGRAMS links a button with the exact, HTTP encoded string to send to the MythTV server. HTTP encoding doesn't effect single keys much, but Ctrl+w becomes Ctrl%2Bw. The first argument to all the three macros is the button name and should be the same for the same button. Taking a look at what is there and remembering to use all three macros to give (a) initial setup, (b) image file, (c) action should get you one the way.

You can update the theme by compiling it with edje_cc and copying it to the device. The application itself shouldn't need any changes.

Note that the bevel buttons are separate to the actual button function image. So if you like you can remote the button by setting 255 to 0 in the lines
color, 0 0 0 255
in the COMMAND_BUTTON definition in clawmotia.edc

I'll probably move to cmake or autofools at some point. If, on the odd case that there are many patches flowing in, then I'll throw the code into subversion somewhere.

The Rasterman has recommended cloning a few hardware remotes as themes. I'm not sure when I'll have the time for that but it should be interesting.

Oh yeah, and if you like this then you feel free to contribute toward a memory card or IR transmitter for my future tinkering. Or send any old n900's you have laying around ;) Hey... I can dream right?

I want my MythTV: Clawmotia

I noticed that there are many packages which turn a maemo device into a remote control. Unfortunately, the MythTV ones I saw either didn't install or were not what I was looking for. Thus clawmotia was born yesterday.


I'm using edje for the interface and qedje to actually create the UI. The program logic uses Qt to talk to a MythTV server and send the commands to it. This relies on knowing there your server is, and what client you want to control which are both passed in as environment variables. Really simple stuff, but it works well.

You'll need the Web interface on the MythTV server machine too. The upshot is that you only need a wifi connection and you can control any MythTV client you want. No bluetooth or IR dongles needed.

As you can see, there are some nasty graphical artifacts on the buttons which I'm not sure if its the evas engine or something to do with qedje. The black parts on the buttons are not there on the desktop.

As it uses edje, you can create different themes and layouts, compile the edje on your desktop machine and just scp it to the device to change the layout and button functionality. Including cute little slide in and out panels for the more rarely used controls. Hopefully I can convince somebody with more artistic ability than me to do just that. Source will be released in the next few days.