Welcome to the libferris screencast on copy append.
1) I start a copy from the command line.
2) I decide I want to copy another file to the same place.
For example to the same USB stick.
3) In the old days, I would have to start a parallel copy
or wait for the first copy to finish before copying
the second file.
4) Now, with libferris, I can just append it to the existing copy task for this destination.

Unfortunately the video quality is not so good in the final web version. Still some fun and games to be had getting things up to scratch.

Untitled from Ben Martin on Vimeo.

Automounting for the People

Now that the bad pun in the title has been read, you can't unread it!

So, now libferris has some closer HAL integration. Of course, some might say right away that ferris has been able to mount dbus for ages, so why have an explicit hal:// filesystem. The reason is that the fdo methods in the raw dbus interface for HAL are very coarse, and by explicitly having a hal:// and a volume manager daemon you get some interesting possibilities.

When the daemon can't work out what to do, it runs a nice little wizard:


One you choose how to identify the volume, you can tell libferris what to do:


More actions will arrive in time. I particularly like the auto copy mode, if a usb stick is readable at 20mb/s, a local scratch RAID-10 at 200Mb/s it makes sense to start moving the data to the scratch RAID right away and then pick it apart from there rather than from the USB stick directly.

Libferris summer hackfest

OK, so I now officially have one participant in the libferris summer hackfest! If you've been looking to help out a crazy filesystem project but hesitant as to who to approach, look no further... ;)

But seriously, if you have been using libferris and always wanted it to do X as well then now is a great time to help make it happen. Of course you could send patches, but you might be better off just buying some time and now is a great time to do the latter. And so ends the plugfest.

Unfsck your redland RDF.

Again, for those who are using the unpatched db4 redland backend, if you get issues where libferris apps just eat CPU and seem to hang, if your backtrace includes Ferris::RDF::RedlandIterator::__advance() calling librdf_iterator_next() then there is a fair chance your RDF db4 files are screwed. You can try to recover them with the below.... sorry about the long standing issue folks.

#!/bin/bash

cd ~/.ferris/rdfdb

for if in myrdf-po2s.db myrdf-so2p.db myrdf-sp2o.db
do
echo "Fixing: $if"
db_dump -r $if >| dump
rm -f $if
db_load -f dump $if
rm -f dump
done

No n810 is an island

Ch, Ch, Ch, Ch, Changes.

Yes, maemo has had rsync for ages. Yes, rsync makes dumping stuff to the device really easy. Trouble is, when you are travelling around for a while, and when you have a little keyboard you tend to update some of those files.

Assuming the OOM (Only One Me) rule, Unison is a great tool to make the n810 much less of a data island and more of an extension of the desktop.
Check out my repository for the debs. You'll need to install gcc and binutils, complements of the way ocaml is currently built. So thats another 15mb of downloads or 40mb of card space you'll loose. The gcc thing should be able to be worked around to make a leaner install, but I wanted unison for use before the maemo summit. Kind of handy to sync stuff over and know you can edit it with the device. Of course you could always rsync from the device when you get home again, but unison is just a nicer bidirectional solution.

Oh by the way, which one's pink?

This should be a really interesting conference. Right at a very exciting time for KDE, so I might be able to survive the event without an onslaught of comments on why I coded libferris instead of working on core VFS + Indexing code in KDE instead :-p~~. It will be interesting talking about nepomuk and the existing RDF+Inferencing that is in libferris. Perhaps everyone can benefit from metadata harvesting and cross communication of RDF data between KDE and libferris even if the codebases and projects are completely separate.

Ruby, don't take your love to town...

It seems that compiling a kernel for n810 devices involves a bunch of clean compiles, a little offering to the daimaou and some mules for the purple cat (now that I've included some nice narrow references Lou ;).

Issues don't seem to be related to the kernel image itself being too large, but compiling some functionality just seems to make the kernel kill the boot just before the desktop is displayed. These two pages are a good start along the merry road to a new kernel, even one that actually boots. Anyway, after a bunch of compile, download, try to boot cycles I found that you can have fuse, dm, encrypted filesystems, NFS client + server and xfs in a working kernel. I'm still trying to work out what parts of iptables state based filtering makes the kernel crap out.

But with the fuse and crypt stuff I should be able to have some fun making a nice front end to encrypt the internal 2gb sdcard... it will be interesting to see how much crypto impacts performance. I think /usr can be raw, but a bunch of the stuff in /home should be encrypted just because I don't want the guy who steals my n810 to read my email too ;-p

In other news, it sort of feels a little like the start of fight club, but instead of browsing furniture and other crap its hardware. 10G-BaseT can not arrive soon enough.

Trees, a forrest, some falling and an ear

Well, my desktop machine is now starting to actually work again after updating it to Fedora 9. "A charmer for the ladies" is the best description of the time trap that was... and still shall be for a few weeks I imagine.

I don't recall any previous fedora being as painful to update to. And then there is the kde4 issue, which I knew about before hand but didn't expect the outcome to be as it has become. So, the issue that finally forced me over the edge in a past GNOME release, not being able to fully hide the panel, was in fact the starter to drive me away from KDE, atleast as my desktop shell. I still love the apps of KDE4, I just use something else for my panel and desktop shell (no candy treats for guessing which wm/shell is at the core now).

It is a shame, I updated to the beta1 of 4.1 and somehow managed to get KDE wedged into a state where it just didn't want to start anymore. even with the old .kde that I started off with. hmm. Though the compositing and kwin stuff is very nice... fundamental things about panel interaction are just not there, or are rather hidden from the user, like being able to rearrange the icons in the panel.

And so it goes that another blog post has fallen, another rant was cut short in mercy for the wasted electrons.

A slightly more readable version of the data from the linux-raid mailing list. RAID-6 is the surprising one here, the rewrite performance taking the cake over all the RAID-10 configurations. So much for parity RAID being slow for writes ;-0

How now is soon?

Well, the main libferris site has now moved to libferris.com. And there was singing and dancing etc.

In other news, the maemo port is getting a little stronger. The db4 indexing modules can handle indexing about 15-20 music files / second on the device itself. Of course search is nice and snappy, but the lack of prelinking problem still remains slowing down actual process spawning. Ferris can now be compiled without Xalan-C which brings its footprint down a little on the device. Still a few other things I might trim off to make things a little nicer on limited hardware. The debs still weight in at about 6.3Mb which is quite chunky for a device with an initial filesystem size of 256Mb.

Still, using a forked child process an app can quite nicely fire off many index searches and sidestep the prelink issue after the initial exec() is done and dusted. Still no panel integration stuff but the fun begins once the port itself works acceptably. Mounting the GPS anyone?

libferris for the n810

So the virtual filesystem that mounts XML, db4, postgresql and even emacs now runs on the n810. There are some initial performance issues, Nokia seems to have decided to drop prelink packages which were in earlier maemo distros. This is quite annoying because small apps like ferrisls actually spend more time in the dynamic linker now than running on the 810. Nothing that can't be fixed with prelink on the n810 and maybe GNU Hash linking style as well. But these things are annoying because it means you have to probably replace your binutils on the device and install prelink from a third party repo (mine, most likely).

But the glory of running XUpdate XQueries on the n810 is still there. Federated indexing from the device instead of the "lets index everything in a static manner" that seems to be the MO for maemo apps at the moment.

I've not tested the mounting of flickr/23hq from the maemo build but having ferris on the 810 should let me integrate "send to" flickr items in application context menus. Once I get the performance issues under control then such handy integration will be on the cards.

Now all I need is a mokophone with 32gb of flash for media (I can dream eh?) so I can taint it with libferris as well.

I'll probably start a target for embedded ferris soon so that more server oriented things like XSLT and XQuery can be trimmed out to provide a smaller footprint VFS/Index solution.

The saru paws and the n810

I finally got one sitting on my desk. I've still to install the one true editor.

First impressions are good. It is quite snappy for a 400Mhz ARM chip. Relative to the e61 things are quite nice. The storage quickly becomes an issue. Luckily I have an 8gb microsdhc card in it, so data files can be taken with me but still I would like to have more storage on the go... but many have noted the blow of only one minisd slot relative to the two full sdhc slots on the n800. Luckily the n810's microUSB seems to work with USB OTG stuff. This will be interesting with 16gb usb thumb drives, being able to plug in a card with a bunch of data on it.

I've already done the whole dual boot by reappropriating the internal 2gb card to be the new boot drive. The 256mb root filesystem might be great for users who don't want to treat the n810 as a little linux machine, but for the serious hacker having a /partition that is over 1.5Gb makes for a much more enjoyable system to install software on.

One very surprising thing is that stuff like kernel side encryption of partitions is so hard to do out of the box. Encrypting /home on a laptop is standard practice, but with an n810, much more stealable than a lappy one would really *need* to have the data become unaccessible to any would be thief. Sure you loose a cool piece of hardware, but allowing someone who has just stolen your mobile device access to your little digital world is just a no go. I would have hoped that you could just install an extra kernel package like for NFS to get device mapper and ecryptfs and luks working without much pain. There are precompiled kernels and modules for luks but having nokia offer a bunch of kernel modules in grouped "kernel-extra" debs by default wouldn't be that hard for them to do and would make the whole process much easier.

I still haven't managed to get FUSE going yet. I guess sadly that it might be true that folks don't care too much about funky filesystems while on the move. But that won't stop me forcing libferris onto the device. Mounting postgresql, sqlite and XML from the n810 isn't going to be any less fun than from the desktop ;)

maemo dreaming

It's things like this video on upnp http://base-art.net/Articles/94/ that make me want to get my hands on a n800/n810 right now. Of course I'd have to expose libferris as a upnp server for the path to the dark side to be complete...

Searching using ID3 / Annodex metadata and having your files served up my libferris... eases the pain. If only I could have made it to LCA and scored one of the n810s they were giving away... ah the waiting.

For those using redland for RDF in the wild

Not so much if, but when redland corrupts its db4 files here is a series of commands that you might want to shove into a script. Though you really want to setup daily db_dumps that are archived if you use redland for data you care about.

I have attempted and posted a patch and nagged to get this fixed mainline. But I'm not holding my breath that it will be fixed any time soon. I might have to dump redland, fork it proper, or some other nasty solution to stop libferris users complaining about apps "crashing" which is directly due to redland getting its db4s stuffed.

You get the pleasure of running this on all your redland db4 files to try to recover (at least you can run client apps again without a segv in the redland code).

$ db_dump -r myrdf-so2p.db > myrdf-so2p.dump
db_dump: myrdf-so2p.db: DB_VERIFY_BAD: Database verification failed
$ rm -f myrdf-so2p.db
$ db_load -f myrdf-so2p.dump myrdf-so2p.db
$ rm -f myrdf-so2p.dump

The public bug is here
http://bugs.librdf.org/mantis/view.php?id=104

Manufacturing Descent

The joys of changing one of the base libraries of the libferris suite. I had to release 8 packages at once because the base library had a few cleanups that were not backward compatible. I thought it was better to just break things a little now before the OBS packages for Fedora, openSUSE and Ubuntu are publicised for general consumption.

In other news, you can now get two sets of EA for results from a search on a PostgreSQL EA index. inside either the "idx:" or "index:" namespace are EA versions that are taken directly from the database itself. So for example you can show the MD5 checksum for resulting files by using idx:md5 as the result column. This is very handy for checksums and the like but also handy when you don't actually want to go and touch the original file at all to present results. There are still a few little changes to be made for full, totally offline, removable SATA drive index result presentation but things are very close to that level now.

Also checkout the apps/phpsearchinterface directory in the latest libferris release for two versions of a web interface to libferris searching. The first is a simple example that uses XSLT on client side to present XML search results to the user. The second example uses an AJAX interface together with the dhtmlxSuite grid control (not included) to present a nicer, more modern interface. These two clients are just examples and are in their infancy but they do give (particularly the latter example) an idea of how libferris can be used on a LAN environment for index searching. Think "libferris Intranet search" ;-)