Libferris as a KIO Slave

The libferris virtual filesystem can now be exposed as a KIO Slove too. This allows you to use KDE applications to list, read and write the vast number of virtual filesystems that libferris makes available. For those who don't know, ferris is a little project I've been working on over the last decade to make everything a file: XML, db, relational data, web services, and even applications like XWindow, Amarok, pulseaudio, and gstreamer.

The following will create a Berkeley db4 file from the command line and show it to you. To dig into such files with libferris you can just read the file directly. So in this case I just grab the "base" directory in this db4 file with konq. You should see the sample and file2 files in that directory view and be able to load and save those "files" into kwrite. Sorry about the video being a tad jumpy, I have to work out what part of my desktop is causing that :/


Using libferris through the KIO interface from Ben Martin on Vimeo.

The commands as plain text:

$ cd /tmp
$ db45_load -T -t btree foo.db
base/sample
value here
base/file2
contents

$ db_dump -p foo.db
$ konqueror ferris:/tmp/foo.db/base

little tricks I found during the hacking, your listDir() method might call listEntry( e, false ) with a final call using ( e, true ). The last call is impotent with regard to "e" and just a finalizer call. The get() method uses data() to deliver bytes to the KIO user (application). The put() method uses dataReq() in a loop to grab chunks frrom the KIO user. Currently I have lazy methods, for example the put() just grabs everything from the KIO user and then operates on the data once it has everything. Really bad for 4gb files, but for smallish files to get a feel for things it works quite well.

Also, if you are using library init, in my case using gmodule to dynamically load some plugins from libferris itself, you might be in for a world of fun and games. Currently I spawn processes and interact with them from the KIO slave to get around this issue. I imagine for a more efficient implementation being able to tell KDE to load my KIO Slave as an application with a normal full init leading to a main() but haven't looked at that little technicality yet.

I've mostly been tinkering with kioclient, konq, and kwrite on libferris files at the moment. Things are turning out well, though there are still many glitches to this early days integration. This will be released in the next libferris version once I clean it up a bit more.

Mounting KIO with libferris

I know there are more folks interested in going the other way -- seeing libferris through kio glasses. Not that there are hoards of folks in either camp of course. Nonetheless the first move has been to allow libferris to mount kio.

The groundwork is very similar to what I'm thinking of using to allow kio to mount libferris. Top level URL schemes will appear first, allowing you to dig into each URL scheme. For example, in libferris kio appears under the kio: filesystem. The first directory in that filesystem is the KIO URL scheme to use. So something like the following will work:

$ date >| /tmp/df.txt
$ fcat kio:file:/tmp/df.txt

Some of the more fun KIO slaves to access through this are the man and fonts URLs. The following two commands produce the same man page:

$ fcat kio:man:/man
$ kioclient cat man:/man

Support is preliminary and only allows reading the files but not writing to them through the kio: filesystem as yet. Already though the kio: can be exposed to XQuery, SQLite, and through the libferris REST interface. Yay for cooperating!

I notice some really juicy digikam kio slaves but haven't dug into them enough to use them as yet. Although you can already upload to various web services from digikam, once I get access to digikamalbums through ferris kio mounting I can then 'cp' the images directly from the command line to other things that libferris can already mount such as flickr API sites, printers etc.

REST & Filesystems: A homage to plan9

In my previous post I mentioned, among other things, two things I want to hack together: An improved REST API for libferris, and KDE KIO integration for libferris. It occurred to me that these two, while able to be performed as distinct tasks, are better done as one. Or rather that the later can be coded to rely on the first being available. With a REST API and a Web server, a virtual filesystem inherits one of the great properties plan9 - filesystems as separate processes that can run on another machine and access data on a third machine.

Consider wanting to get at information from a PostgreSQL table from a mobile phone. The REST API might be on the "webserver" machine, while the database runs on dbserver. The phone talks to webserver sending GET/POST and receiving XML/JSON depending on it's preference. Using libferris, webserver talks to dbserver (by mounting it) and queries and returns results back to the phone.

This also lets the phone upload images via the REST API. Instead of setting up software on the phone, just use QML or HTML5 and http POST the image to the REST API using a path of "flickr://me/upload/" and have the server send that image onwards. Anything that libferris can mount and interact with becomes available over the light weight REST API, ready for QML/Javascript thumb interfaces. Other examples would be getting at a mounted Zoneminder over REST, or a webcam using mounted gstreamer.

Using REST like this also fairly nicely sweeps away language binding issues. Almost all languages can access http in some way. Sort of like CORBA without the IDL files and tools to create stubs for you. Hmm, a REST API IDL compiler for Qt with OAuth support... mmm...

Perl io::all, REST, and libferris, wagging the dog

A feature that I've had planned to add to libferris since the UKUUG spring conference earlier this year is Perl io::all support. I've syndicated this because adding an io:all interface to KDE kio stuff would also be very cool IMHO.

I'm digging around for others who think io:all is cool and perhaps that libferris is "plus neutral" and might have some time to help out with the creation of a CPAN module to let the two things work together. In libferris there is a single shot interface to get metadata or file content as a single std::string or an std::iostream based interface allowing seeks, streaming and all that jazz. Ultimately I'd like to make the streaming stuff meet perl::io and also have a fake hash there to allow EA to get read on demand. The C++ and to a fairly large extent the Perl binding side I have knowledge of. But I admit to mostly being a Perl user rather than developer of the language or CPAN modules.

If anyone has pointers to stuff that would help out doing this stuff (tutorials and the like) or would like to support the feature in other ways please drop me a line or comment here. Otherwise, I'll probably dig into this later this year, time permitting.

I'm also thinking about making a REST server for CRUDing libferris. This should make Web clients much simpler to create, for example file managers in Yahoo widget UI. Once again, if this interests you, feel free to jump on in ;-p I'll have a survey of existing REST apis for this sort of stuff, obviously providing a similar interface will help other clients to interface. Luckily I can reuse some of the knowledge I gained when mounting web services as filesystems in the creation of the libferris web service server.