Sunday, February 14, 2010

Libferris: now and then...

Those who have been following libferris developments over the recent years will have noticed that I switched over to using Soprano for RDF support in 2009. I also added support for mounting relational databases (using QtSql) and converted the http and ftp mounting over to use Qt for network IO. This later point also extends to the support for mounting flickr, vimeo, google spreadsheets and docs etc. So Qt is slowly marching closer to the "kernel" of libferris.

I've been toying around with switching over some of the core parts of the libferris kernel to using Qt. Things like using Qt's smart pointers instead of the policy based ones from Modern C++ Design, switching over to using Qt signal/slots, and perhaps offering both iostream and QByteArray data access methods in parallel.

There are a number of "smaller" tasks to support this. For example, creating a stldb4 like library allowing access to Berkeley db4 and other ISAM files with a "Qt native" style API. Stldb4 only supports bdb files, whereas allowing other ISAM like gdbm, tdb, and WhateverDB to be accessed through the hypothetical QISAM library would be nice. If anyone has recommendations along these lines please drop me a line or leave a comment. I mention smaller in quotes above because I know this task is quite difficult to do well, things like ordering and prefix ordering of keys come up as well as support for secondary indexing and other goodies which are really needed to milk performance out of custom ISAM solutions.

The harder part would be porting over the core kernel of libferris to Qt stuff. As it would be extremely hard to do piecewise it would require a contiguous block of free time. It is also a fairly hard thing to try to get financial support for because the only "feature" it adds is the possibility of ports for other platforms. My point of mentioning this is that at some stage, libferris ports for Symbian and other Qt supported platforms will become much simpler. Finally I could "cp" an image directly from the camera of my s60 phone to flickr without the annoying third party apps.

As an aside, I was wondering how long it has been since I started using RDF on the desktop. The November 2003 version 1.1.12 release mentions mounting RDF... so it's been a while. RDF is extremely handy for separating data access from personal metadata. If you want to add an extended attribute to a read-only NFS share, with libferris and RDF you can, and you just use the same API as normal ;)


nidi said...

Hi Ben,
thanks for your update on libferris.

I was wondering the other day what the Nepomuk times will bring to us command line geeks.

Looking at Ferris again ... shows ... that that's exactly what I want!

Now that the idea of a semantic desktop has become popular, I wonder what role libferris could play in the Free Software universe. Some thoughts and questions:

1- The "mounting everything" feature is something basically everybody wants. For sure! KIO can do a couple of those, but this is something which should be desktop independent. And I definitely want to mount databases!!

2- Metadata extraction: Well, we also want that. How do strigi and the corresponding ferris component relate? Are the tasks of crawling / watching file changes, analyzing files and indexing metadata well separated, so components could be interchanged? Would it make sense to use strigi as a crawler for ferris?

3- How could ferris and Nepomuk go together? Could they be different frontends to the same semantic database? Or could Nepomuk be ported to ferris?

4- Other way round: Could I use the ferris command line suite to query a Nepomuk database?

And a couple of technical questions about ferris:

5- What's the overhead of adding ferris (sans GUI) to a standard linux desktop system in terms of
- additional dependencies
- size of ferris itself
- additionally running processes ?

6- Which parts of ferris can be used independently? (translation of the dependency tree to the end user, please ;-))

7- Could you say a couple of words about ferris' stability, performance and code quality?

8- What's the easiest way for seeing ferris in action (distro / vm / build script)?

Thank you,

monkeyiq said...

I'm glad you're interested in libferris :) I recommend using the OBS builds for Fedora 11 or 12... perhaps in a VM if you don't use Fedora by default. I'd love to have packages for Debian/Ubuntu, openSUSE etc but given limited time etc :/

1. Unfortunately some virtual filesystems shy away from this a bit. Some on the grounds that the semantics might be vague or that they can not give POSIX guarantees. If I can do something useful by mounting anything, then I'll either have it supported already or on the TODO list.

2- I have a libstreams plugin for libferris already. So both can use the same metadata extraction. Metadata extraction, indexing, and index maintenance are well separated in libferris.

3 and 4- Both Nepomuk and libferris use Soprano to store and query the RDF. So they already go together right now. There is the issue of ontology mapping, but that will always be present in independently designed systems. Luckily the semantic web guys have worked things out to help mitigate that pain.

5- No additional processes. Some are optional and add additional features. Size is Qt, Soprano, libferrisloki, libferrisstreams, libstldb4, libferris and its various clients. Putting it in perspective, I have builds for maemo, so it works in "limited resource" environments.

6- I've tried to make much of libferris able to be not built. If you don't want to mount Y then it's plugin is not compiled or installed. You can use mounting, metadata, and indexing all somewhat separately, indexing relies on the former two.. If you don't want all the command line tools, don't install them all.

7- IMHO the code quality is good. But asking a developer how good their code is is somewhat like asking how long a piece of string is ;) I try to make the code correct first while not ignoring performance. Some actions you can take will possibly be slow. For example, updating part of an XML file which you have mounted is not an operation that can be made fast easily.

Stability is always going to be better in features I use often. eg. mounting XML, db4, postgresql and much of the RDF support. But in general should be quite good. Patches accepted etc... If you try strange things, run ferris-capplet-logging and turn up debugging for relevant messages.

8- Fedora and OBS packages.

nidi said...

Thanks for your answers!

I'll try out the Fedora packages in a VM soon and try to provide some ebuilds if I succeed compiling libferris on Gentoo.

Paul said...

Just wondering what the status of libferris is? The library hasn't been updated since 2010 on source forge. It looks to be super interesting and I was wondering whether there's a "2.1" version hanging around.


monkeyiq said...

@paul: there have been some regular releases, the last on the 31st of Jan 2012...

Since then there is now identica mounting support which will be in the next release. So things are progressing all the time, of course patches accepted etc, and so forth ;)