OSX Bundling Soprano and other joys

Libferris has been moving to use more Qt/KDE technologies over the years. Ferris is also a fairly substantial software project in it's own right, with many plugins and support for multiple libraries. Years back I moved from using raw redland to using soprano for RDF handling in libferris.

Over recent months, from time to time, I've been working on an OSX bundle for libferris. The idea is to make installation as simple as copying Ferris.app to /Applications. I've done some OSX packaging before, so I've been exposed to the whole library paths inside dylib stuff, and also the freedesktop specs expecting things in /etc or whatever and you really want it to look into /Applications/YouApp/Contents/Resources/.../etc/whatever.

The silver test for packaging is to rename the area that is used to build the source to something unexpected and see if you can still run the tools. The Gold test is obviously to install from the app.dmz onto a fresh machine and see that it runs.

I discovered a few gotchas during silver testing and soprano usage. If you get things half right then you can get to a state that allows the application to run but that does not allow a redland RDF model to ever be created. If your application assumes that it can always create an in memory RDF store, a fairly secure bet really, then bad things will befall the app bundle on osx.

Plugins are found by searching for the desktop files first and then loading the shared libary plugin as needed. The desktop files can be found with the first line below, while the second line allows the plugin shared libraries to be found and loaded.

export SOPRANO_DIRS=/Applications/Ferris.app/Contents/Resources/usr/share
export LD_LIBRARY_PATH=/Applications/Ferris.app/Contents/Resources/usr/local/lib/soprano/

You have to jump through a few more hoops. You'll find that the plugin ./lib/soprano/libsoprano_redlandbackend.so links to lib/librdf.0.dylib and librdf will link to other redland libraries which themselves link to things like libxml2 which you might not have bundled yet.

There are also many cases of things linking to QtCore and other Qt libraries. These links are normally to nested paths like Library/Frameworks/QtCore.framework/Versions/4/QtCore which will not pass the silver test. Actually, links inside dylibs like that tend to cause the show to segv and you are left to work out where and why that happened. My roll by hand solution is to create softlinks to these libraries like QtCore in the .../lib directory and then resolve the dylib links to these softlinks.

In the end I'd also like to make an app bundle for specific KDE apps. Just being able to install okular by drag and drop would be very handy. It is my preferred reader for PDF files and having a binary that doesn't depend on a build environment (homebrew or macports) makes it simpler to ensure I can always have okular even when using an osx machine.

Redland && Fedora 18

It appears that redland might be busted in some respects in Fedora 18. After a fresh install I recompiled libferris as per usual after an upgrade, only to find that the RDF/Soprano engine for metadata was busted. Digging into this, looking at the soprano sources and with many versions of soprano and sopranocmd I thought maybe the issue was one layer closer to the metal. Switching to rdfproc from redland I noticed that I couldn't create a new on disk RDF/db store!

$ rdfproc myrdf add \
  'http://witme.sf.net/libferris-core/0.1/subj' \
  'http://witme.sf.net/libferris-core/0.1/pred' \
  'http://witme.sf.net/libferris-core/0.1/obj1'

rdfproc: Failed to open hashes storage 'myrdf'

After downloading redland's source rpm, recompiling it and installing the resulting rpm files I could then create RDF/db again. Same command, different result:

$ rdfproc myrdf add \
  'http://witme.sf.net/libferris-core/0.1/subj' \
  'http://witme.sf.net/libferris-core/0.1/pred' \
  'http://witme.sf.net/libferris-core/0.1/obj1'
rdfproc: Added triple to the graph

I haven't verified this on a second Fedora 18 machine yet, but it might well mean that anyone trying to use the Berkeley db backends of redland on F18 would need an update to redland or to do some tinkering.

Can't get *there* from here

I've been thinking for a while about an application for mobile devices that knows rail, bus, ferry, and magical flying eagle timetables and stops. The idea is to be able to glance at a mobile device and see it say "hey you are at foobar, its 9pm, I guess you want to go home/to hotel, and here's how you can do it, how regularly that happens and the last time you could do it today".

Of course, n900 guys will want to just use a web service for this. But as mobile data costs kidneys in some parts of the world, I'm more inclined to choose a design that precaches the data when free wifi is available. This also plays well for travelling with roaming charges etc.

If you are travelling, then the machine should already know where you are staying and when, so it can direct you to the metro line of interest from where you are to get there. Again for travelling, being able to wangle a timeline to move "now" forward and see that the app can suggest reasonable options as it goes is also a good idea. You don't want to rely on it to suggest and then find you are late for the jet.

I'm thinking perhaps Qt/Soprano for this, but the exact RDF vocabulary for the bus, train, etc timetables is not jumping out at me. There are many adhoc designs I've thought about, like a series or RDF list of ical entries for each bus run, perhaps with each entry using geo84 or some other ICBM assocation method. If you know of a good RDFS for this, please let me know. I don't know when/if I'll get to hack on the code, but the itch to do so is unlikely to go away by itself.

Five Star General

I've been thinking about proxy patterns a Soprano lately. It seems like a great idea to create a Soprano::Model shim which logs mutable operations to an SQLite database. This would allow intermittent syncs between devices, so a maemo device could have a full local mirror of RDF and communicate with your laptops, servers and desktop machines to keep semantic knowledge in sync.

There are a bunch of nice little edge cases which make this deceptively complex. bnode handling for one changes the rules because identity must be preserved.

I'm not sure if/when I'll hack on such functionality, but I thought I'd blog about it in case somebody else is also kicking around the same ball :)

KOffice & RDF: Who, What, When, Where?

As mentioned in a previous post, ODF documents can contain one or more RDF/XML files. These files allow you to unambiguously encode information for both computer and human consumption. So you can describe a person in a way that tells you their phone number and also lets the computer know that these digits are a specific person's home phone number. Common data formats like vcard and ical have some encodings in RDF and soon a KOffice near you will understand these pieces of data from ODF files.

KOffice currently understands some of the FOAF vocabulary (storing contact data), and the rdfical format (for events). There are a few ways to encode longitude and latitude in RDF. The current patch supports two of them, with optional linking to rdfical. This is one of the major strengths of RDF, you can say who, where and when and also link these things together so an event carries not only a time but its location information too.

The below video shows the new RDF docker in aciton. As you click on text that has associated RDF, the docker shows you the interesting information. Frodo and Sam are assoicated with both traditional contact data and a location. The items in the RDF docker let you import them into your system (into kaddressbook or korganizer) or export them to well known desktop formats like vcard and ical. Editing locations is done with Marble and there is only a minimal set of information for contacts and events currently. Note that mid way through when I edit an event, timezones are respected. If the RDF describes an event as being in Tokyo, that timezone offset from your current localtime is respected.

Towards the end of the video I show that contacts can be simply drag and dropped between koffice and kaddressbook. This also works for events to/from Evolution but I had some issues with korganizer for events. D&D makes KOffice and ODF quite a convenient format for transmitting semantic information to colleagues in a single, self describing file.

KOffice & RDF: Who, What, When, Where? from Ben Martin on Vimeo.

White lightning in triplicate

Recently I started hacking on a memory mapped, multi_index soprano backend. While adding triples, and using listStatements() should work fine, implementing SPARQL is making for interesting times.

I started out allowing a single triple match with a filter(regex()) to restrict results. And this worked rather well, making the first one free as they say. So, noticing the little white rabbit that seemed to disappear into the SPARQL bushes, I decided to join in the high tea and mercury sniffing that so induces sanity. Over the course of version 0.0.1 to 0.0.5 the SPARQL code is becoming better, little by little. The code is up at my sf.net page. But don't blame me if the your SPARQL is not implemented yet or your triples somehow disappear.

Anyway, here is a little benchmark session. I'm using the data set generator and queries found here. To make the data I use

$ cd /usr/local/java/bsbmtools
$ cat run.sh
#!/bin/bash
java -cp bin:lib/ssj.jar benchmark.generator.Generator "$@"
$ ./run.sh -fc -pc 1000 -s nt
$ mv dataset.nt  thousand-prods.nt
$ mkdir -p /tmp/RDFBENCH
$ cd /tmp/RDFBENCH
$ mkdir mmap redland

Queries are run multiple times to ensure a hot disk cache. This is on a 3 disk RAID-5 and an Intel Q6600 with 8gb RAM.
The last query is not optimized properly in boostmmap yet, so its far slower than it rightly should be. For benchmarking the boostmmap backend...

$ cd /tmp/RDFBENCH/mmap
$ time sopranocmd --backend boostmmap \
  --serialization ntriples \
  import /usr/local/java/bsbmtools/thousand-prods.nt >|out 2>&1

real    1m49.642s
210M     triples.mmap*

$ time sopranocmd \
  --backend boostmmap \
  list "" '<http://www.w3.org/1999/02/22-rdf-syntax-ns#type>' \
  '<http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/product>' \
   >| /tmp/out 2>&1

real    0m0.103s
grep Product /tmp/out | wc -l
1001

## based on Query 6
$ time sopranocmd \
  --backend boostmmap query \
"
select ?what ?lab
where
{
  ?what http://www.w3.org/2000/01/rdf-schema#label ?lab .
  filter( regex( str( ?lab ), 'excites' ))
}"
?lab -> <yawned%20excites%20deflower>;
  ?what -> <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/productfeature295>
?lab -> <goofs%20excites%20enigmata>;
  ?what -> <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/productfeature3276>

real    0m0.091s


$ time sopranocmd --backend boostmmap query \
"
prefix bsbm: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/>
prefix xsd: <http://www.w3.org/2001/xmlschema#>
prefix dc: <http://purl.org/dc/elements/1.1/>
select ?offer ?price
where {
    ?offer  bsbm:product http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/dataFromProducer1/Product5 .
    ?offer  bsbm:vendor ?vendor .
    ?vendor bsbm:country http://downlode.org/rdf/iso-3166/countries#ES .
    ?offer  dc:publisher ?vendor .
    ?offer  bsbm:price ?price .
}"
0.93sec

Note that this 0.9seconds is shameful and needs to be optimized back to <0.1sec.

For redland,

$ cd /tmp/RDFBENCH/redland
$ time sopranocmd --backend redland \
 --serialization ntriples \
 import /usr/local/java/bsbmtools/thousand-prods.nt \
 >|/tmp/out 2>&1

real    38m34.735s
480mb

$ time sopranocmd --backend redland \
  list "" \
  '<http://www.w3.org/1999/02/22-rdf-syntax-ns#type>' \
  '<http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/product>'  \
  >| /tmp/out 2>&1

real    0m0.096s
grep Product /tmp/out | wc -l
1000

So for just listStatements() redland and mmap are fairly equal in performance. Which, for a single indexed lookup, you might expect. In libferris I had restricted RDF usage to raw triple probes like this because I used redland directly prior to version 1.4.x of libferrris.

So for SPARQL,

## based on Query 6
$ time sopranocmd --backend redland query \
"
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
select ?what ?lab
where
{
  ?what rdfs:label ?lab .
  filter( regex( str( ?lab ), 'excites' ))
}"
what -> <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/productfeature295>;
   lab -> "yawned excites deflower"
what -> <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/productfeature3276>;
   lab -> "goofs excites enigmata"
real    0m3.855s

Gah, and I didn't slip up and put the 3 on the left side of the dot there. We are talking about 0.1 seconds for boostmmap against 3.86 seconds for redland.

$ time sopranocmd --backend redland query \
"
prefix bsbm: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/>
prefix xsd: <http://www.w3.org/2001/xmlschema#>
prefix dc: <http://purl.org/dc/elements/1.1/>
select ?offer ?price
where {
      ?offer bsbm:product <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/datafromproducer1/product5>
      ?offer bsbm:price ?price .
      ?offer bsbm:vendor ?vendor .
      ?offer dc:publisher ?vendor .
      ?vendor bsbm:country <http://downlode.org/rdf/iso-3166/countries#es> .
}"
real    0m7.134s

Since this query doesn't work well on boostmmap it only goes from 1 to 7 seconds. But I think I can resolve it in much much less time than 1 second. This is not meant to make redland look bad, it's SPARQL implementation is much more complete than boostmmap will likely be any time soon. Creating an optimal query plan for the full SPARQL language will be an interesting challenge.

Development might be bursty as I don't know what time I can spare for improving the SPARQL completeness in the short term.

RDF on the device

I've started work on a memory mapped soprano RDF backend. Given that mobile devices use flash for their permanent storage, an RDF backend designed using primary storage algorithms should work well on maemo.

While the SPARQL implementation is "far from complete", it can already evaluate some common queries very very quickly. I have some triple matching and the ability to have multiple regex filter statements. Other filters and more complete SPARQL lanugage support should follow in time, as time permits... patches accepted etc and so on.

For those interested, see soprano-boostmmapbackend on my main sf.net page.