Friday, December 18, 2009
This blog post message was filtered by the Australian Government.
This blog post message was filtered by the Australian Government.
Saturday, December 5, 2009
Office documents that mean something?
I've been hacking on the development branch of KOffice to add RDF support. Thanks to KO GmbH for this chance! This means that while a document can posit a linear progression of characters and words like normal, the software can start to understand a bit of what you are talking about too.
The ODF document specification allows RDF to be included in the document zip file either inline in the main content.xml file, or using manifest.rdf and other RDF/XML files in the zip. The RDF is free to refer to XML elements in the content.xml, so you can add metadata to names, places, times etc so that a computer can work out unambiguously what you are talking about. This makes an ODF file a very powerful container format for transmitting meaning to people.
At it's heart, RDF represents all information in triples. Bob knows Alice, etc. See lwn.net or other sources if you want to know more about RDF. But you don't have to know it to use it with KOffice ;)
Below you can see that I've hacked the "Document Information" window to let you see the raw RDF if you want to know exactly what is going on:
If on the other hand, you are not a developer and/or don't really care about triples or RDF, there is the Semantic view. Right now I only have support for Contact information which is drawn from the FOAF RDF vocabulary. If the later doesn't mean anything to you, its just a way to describe people and their relationships using RDF. As you can see, you don't have to care about RDF here, you just see people. Right clicking on a person lets you do things like import them into your contacts database, or, in the future, other things like email them or phone them.
While it is nice to be able to have contact information extracted from the document, it is much nicer to have KOffice know exactly where that information relates in the document. For this I have a new Docker for RDF as seen in the below video. First you see that the addressbook is empty. Then I start moving the cursor around in the document. Notice that the docker picks up when the cursor is on the name of a person that has RDF metadata. You see more information about Frodo, and then I import Sam. The contact entry in the addressbook created for the import brings his phone number in too, which was part of the RDF but not explicitly shown in the document.
I hope to add support for other things like times, locations, and relationships. This way you can send somebody a document describing a meeting and they can import the time into their smart phone directly from the document... can we make retyping such information a vague memory like punch cards?
The ODF document specification allows RDF to be included in the document zip file either inline in the main content.xml file, or using manifest.rdf and other RDF/XML files in the zip. The RDF is free to refer to XML elements in the content.xml, so you can add metadata to names, places, times etc so that a computer can work out unambiguously what you are talking about. This makes an ODF file a very powerful container format for transmitting meaning to people.
At it's heart, RDF represents all information in triples. Bob knows Alice, etc. See lwn.net or other sources if you want to know more about RDF. But you don't have to know it to use it with KOffice ;)
Below you can see that I've hacked the "Document Information" window to let you see the raw RDF if you want to know exactly what is going on:
If on the other hand, you are not a developer and/or don't really care about triples or RDF, there is the Semantic view. Right now I only have support for Contact information which is drawn from the FOAF RDF vocabulary. If the later doesn't mean anything to you, its just a way to describe people and their relationships using RDF. As you can see, you don't have to care about RDF here, you just see people. Right clicking on a person lets you do things like import them into your contacts database, or, in the future, other things like email them or phone them.
While it is nice to be able to have contact information extracted from the document, it is much nicer to have KOffice know exactly where that information relates in the document. For this I have a new Docker for RDF as seen in the below video. First you see that the addressbook is empty. Then I start moving the cursor around in the document. Notice that the docker picks up when the cursor is on the name of a person that has RDF metadata. You see more information about Frodo, and then I import Sam. The contact entry in the addressbook created for the import brings his phone number in too, which was part of the RDF but not explicitly shown in the document.
KOffice starts getting RDF from Ben Martin on Vimeo.
I hope to add support for other things like times, locations, and relationships. This way you can send somebody a document describing a meeting and they can import the time into their smart phone directly from the document... can we make retyping such information a vague memory like punch cards?
Thursday, November 12, 2009
Clawmotia, look and feel and portrait mode
Clawmotia is a remote control for MythTV using Qt and qedje. The claw works on maemo and desktop systems. I have updated submenus to fade better, added hotkey support, and improved images and icons since 0.2. Grab it at my maemo repository.
In the shot of it running on the device shown below, the euro sign is for commercial skip forward and back. Having the skip back in the top right of the device makes it very easy to hit if an auto skip has gone too far. It is no accident that pause, mute, and cancel are placed in the other corners. The icons below change the volume up and down. In the middle of the top is an icon to bring up a submenu letting you set the aspect ratio and stretch the video in various ways to get maximal use from your screen.
I have also been tinkering with allowing switching between landscape and portrait mode, the edje is ready, there are just a few things to iron out before it works as expected. Of course, such switching will be far more useful on an n900 where it can happen automatically as you rotate the device. But for n8x0 users a button will be available trigger a rotate...
In the shot of it running on the device shown below, the euro sign is for commercial skip forward and back. Having the skip back in the top right of the device makes it very easy to hit if an auto skip has gone too far. It is no accident that pause, mute, and cancel are placed in the other corners. The icons below change the volume up and down. In the middle of the top is an icon to bring up a submenu letting you set the aspect ratio and stretch the video in various ways to get maximal use from your screen.
I have also been tinkering with allowing switching between landscape and portrait mode, the edje is ready, there are just a few things to iron out before it works as expected. Of course, such switching will be far more useful on an n900 where it can happen automatically as you rotate the device. But for n8x0 users a button will be available trigger a rotate...
Sunday, November 8, 2009
Improving the CLAW!
Clawmotia is a MythTV remote using edje and Qt. It works well on maemo, desktop and other devices. It requires a MythTV Web setup to talk to. See previous posts for how to set it up.
I got rid of the rendering artifacts on maemo, added hotkey support (volume, fullscreen) and some of the buttons on the main remote screen now bring up submenus. This paves the way for more advanced remote control configurations where intricate but rarely used controls are still available on the remote. For example, aspect ratio and image fill are unlikely to be needed on a well setup MythTV installation, but they are in a submenu of the remote should they be needed. I know the backgound for the submenu panel is not really nice, but with the drop shadow and pan in effect (which also needs tweaking) you can easily see it's a submenu.
Grab it here. I've included the x86_64 and armel binaries as well as the edj "edje" file. You'll need QtCore and qedje and you're in business. For the mameo you'll still have to make a menu shortcut yourself, and of course, see the README.
Better artwork very welcome. If you have skills, I should be able to take an xcf file and turn out a theme. Conversely, take a look at the edc file and you shouldn't have much trouble adding/removing buttons and changing the images.
I got rid of the rendering artifacts on maemo, added hotkey support (volume, fullscreen) and some of the buttons on the main remote screen now bring up submenus. This paves the way for more advanced remote control configurations where intricate but rarely used controls are still available on the remote. For example, aspect ratio and image fill are unlikely to be needed on a well setup MythTV installation, but they are in a submenu of the remote should they be needed. I know the backgound for the submenu panel is not really nice, but with the drop shadow and pan in effect (which also needs tweaking) you can easily see it's a submenu.
Grab it here. I've included the x86_64 and armel binaries as well as the edj "edje" file. You'll need QtCore and qedje and you're in business. For the mameo you'll still have to make a menu shortcut yourself, and of course, see the README.
Better artwork very welcome. If you have skills, I should be able to take an xcf file and turn out a theme. Conversely, take a look at the edc file and you shouldn't have much trouble adding/removing buttons and changing the images.
Saturday, October 31, 2009
White lightning in triplicate
Recently I started hacking on a memory mapped, multi_index soprano backend. While adding triples, and using listStatements() should work fine, implementing SPARQL is making for interesting times.
I started out allowing a single triple match with a filter(regex()) to restrict results. And this worked rather well, making the first one free as they say. So, noticing the little white rabbit that seemed to disappear into the SPARQL bushes, I decided to join in the high tea and mercury sniffing that so induces sanity. Over the course of version 0.0.1 to 0.0.5 the SPARQL code is becoming better, little by little. The code is up at my sf.net page. But don't blame me if the your SPARQL is not implemented yet or your triples somehow disappear.
Anyway, here is a little benchmark session. I'm using the data set generator and queries found here. To make the data I use
Queries are run multiple times to ensure a hot disk cache. This is on a 3 disk RAID-5 and an Intel Q6600 with 8gb RAM.
The last query is not optimized properly in boostmmap yet, so its far slower than it rightly should be. For benchmarking the boostmmap backend...
Note that this 0.9seconds is shameful and needs to be optimized back to <0.1sec.
For redland,
So for just listStatements() redland and mmap are fairly equal in performance. Which, for a single indexed lookup, you might expect. In libferris I had restricted RDF usage to raw triple probes like this because I used redland directly prior to version 1.4.x of libferrris.
So for SPARQL,
Gah, and I didn't slip up and put the 3 on the left side of the dot there. We are talking about 0.1 seconds for boostmmap against 3.86 seconds for redland.
Since this query doesn't work well on boostmmap it only goes from 1 to 7 seconds. But I think I can resolve it in much much less time than 1 second. This is not meant to make redland look bad, it's SPARQL implementation is much more complete than boostmmap will likely be any time soon. Creating an optimal query plan for the full SPARQL language will be an interesting challenge.
Development might be bursty as I don't know what time I can spare for improving the SPARQL completeness in the short term.
I started out allowing a single triple match with a filter(regex()) to restrict results. And this worked rather well, making the first one free as they say. So, noticing the little white rabbit that seemed to disappear into the SPARQL bushes, I decided to join in the high tea and mercury sniffing that so induces sanity. Over the course of version 0.0.1 to 0.0.5 the SPARQL code is becoming better, little by little. The code is up at my sf.net page. But don't blame me if the your SPARQL is not implemented yet or your triples somehow disappear.
Anyway, here is a little benchmark session. I'm using the data set generator and queries found here. To make the data I use
$ cd /usr/local/java/bsbmtools
$ cat run.sh
#!/bin/bash
java -cp bin:lib/ssj.jar benchmark.generator.Generator "$@"
$ ./run.sh -fc -pc 1000 -s nt
$ mv dataset.nt thousand-prods.nt
$ mkdir -p /tmp/RDFBENCH
$ cd /tmp/RDFBENCH
$ mkdir mmap redland
Queries are run multiple times to ensure a hot disk cache. This is on a 3 disk RAID-5 and an Intel Q6600 with 8gb RAM.
The last query is not optimized properly in boostmmap yet, so its far slower than it rightly should be. For benchmarking the boostmmap backend...
$ cd /tmp/RDFBENCH/mmap
$ time sopranocmd --backend boostmmap \
--serialization ntriples \
import /usr/local/java/bsbmtools/thousand-prods.nt >|out 2>&1
real 1m49.642s
210M triples.mmap*
$ time sopranocmd \
--backend boostmmap \
list "" '<http://www.w3.org/1999/02/22-rdf-syntax-ns#type>' \
'<http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/product>' \
>| /tmp/out 2>&1
real 0m0.103s
grep Product /tmp/out | wc -l
1001
## based on Query 6
$ time sopranocmd \
--backend boostmmap query \
"
select ?what ?lab
where
{
?what http://www.w3.org/2000/01/rdf-schema#label ?lab .
filter( regex( str( ?lab ), 'excites' ))
}"
?lab -> <yawned%20excites%20deflower>;
?what -> <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/productfeature295>
?lab -> <goofs%20excites%20enigmata>;
?what -> <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/productfeature3276>
real 0m0.091s
$ time sopranocmd --backend boostmmap query \
"
prefix bsbm: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/>
prefix xsd: <http://www.w3.org/2001/xmlschema#>
prefix dc: <http://purl.org/dc/elements/1.1/>
select ?offer ?price
where {
?offer bsbm:product http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/dataFromProducer1/Product5 .
?offer bsbm:vendor ?vendor .
?vendor bsbm:country http://downlode.org/rdf/iso-3166/countries#ES .
?offer dc:publisher ?vendor .
?offer bsbm:price ?price .
}"
0.93sec
Note that this 0.9seconds is shameful and needs to be optimized back to <0.1sec.
For redland,
$ cd /tmp/RDFBENCH/redland
$ time sopranocmd --backend redland \
--serialization ntriples \
import /usr/local/java/bsbmtools/thousand-prods.nt \
>|/tmp/out 2>&1
real 38m34.735s
480mb
$ time sopranocmd --backend redland \
list "" \
'<http://www.w3.org/1999/02/22-rdf-syntax-ns#type>' \
'<http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/product>' \
>| /tmp/out 2>&1
real 0m0.096s
grep Product /tmp/out | wc -l
1000
So for just listStatements() redland and mmap are fairly equal in performance. Which, for a single indexed lookup, you might expect. In libferris I had restricted RDF usage to raw triple probes like this because I used redland directly prior to version 1.4.x of libferrris.
So for SPARQL,
## based on Query 6
$ time sopranocmd --backend redland query \
"
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
select ?what ?lab
where
{
?what rdfs:label ?lab .
filter( regex( str( ?lab ), 'excites' ))
}"
what -> <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/productfeature295>;
lab -> "yawned excites deflower"
what -> <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/productfeature3276>;
lab -> "goofs excites enigmata"
real 0m3.855s
Gah, and I didn't slip up and put the 3 on the left side of the dot there. We are talking about 0.1 seconds for boostmmap against 3.86 seconds for redland.
$ time sopranocmd --backend redland query \
"
prefix bsbm: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/>
prefix xsd: <http://www.w3.org/2001/xmlschema#>
prefix dc: <http://purl.org/dc/elements/1.1/>
select ?offer ?price
where {
?offer bsbm:product <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/datafromproducer1/product5>
?offer bsbm:price ?price .
?offer bsbm:vendor ?vendor .
?offer dc:publisher ?vendor .
?vendor bsbm:country <http://downlode.org/rdf/iso-3166/countries#es> .
}"
real 0m7.134s
Since this query doesn't work well on boostmmap it only goes from 1 to 7 seconds. But I think I can resolve it in much much less time than 1 second. This is not meant to make redland look bad, it's SPARQL implementation is much more complete than boostmmap will likely be any time soon. Creating an optimal query plan for the full SPARQL language will be an interesting challenge.
Development might be bursty as I don't know what time I can spare for improving the SPARQL completeness in the short term.
I give you the CLAW!
Clawmotia, an edje MythTV remote control.
Sources hot off the presses: clawmotia-0.0.1. You'll also notice the diablo binary for clawmotia on there so you don't need a scratchbox to get going. I put the clawmotia.edj up too so you don't even need to compile that if you don't wanna.
Dependencies on the device: Qt, qedje. You probably have most of the requirements for the latter installed if you have canola2 on the device. There are deb files for qedje on its web site. Qt is also packaged for the n810. I don't have debs for clawmotia yet, feel free to send the debian directory to do that.
You'll obviously need a MythTV server (and client) to use the remote. It uses the web interface to mythtv so you'll want that installed on the server machine too. The various configuration settings are passed in using environment variables in version 0.0.1. See the script start-clawmotia.sh for these settings. You will also need to be in the same directly as the .edj file when you start clawmotia as it expects that at this point. I install the script, binary and edj into /usr/local/bin on the device.
Setting CLAWMOTIA_FS=1 will start clawmotia full screen. There is a bug in version 0.0.1 where it will start but not be shown if started full screen. You have to switch to another app and back to clawmotia to see it. I was planning to fix this before release, but it seems some folks want to have a tinker with it so I'll fix that bug soon.
If you want to change the buttons or layout, you only need to edit the clawmotia.edc file. If you are good with the gimp, you might like to reate a theme at 800x480. As long as you use layers it should be a fairly mechanical process to create an edc file for your theme.
To make a button, define its image in the images section and call the three macros COMMAND_BUTTON, COMMAND_BUTTON_ICON, and COMMAND_PROGRAMS. COMMAND_BUTTON sets up the area of the screen that will react to your button with an optional text label for the button as arg2. COMMAND_BUTTON_ICON links a COMMAND_BUTTON with an image file. And COMMAND_PROGRAMS links a button with the exact, HTTP encoded string to send to the MythTV server. HTTP encoding doesn't effect single keys much, but Ctrl+w becomes Ctrl%2Bw. The first argument to all the three macros is the button name and should be the same for the same button. Taking a look at what is there and remembering to use all three macros to give (a) initial setup, (b) image file, (c) action should get you one the way.
You can update the theme by compiling it with edje_cc and copying it to the device. The application itself shouldn't need any changes.
Note that the bevel buttons are separate to the actual button function image. So if you like you can remote the button by setting 255 to 0 in the lines
color, 0 0 0 255
in the COMMAND_BUTTON definition in clawmotia.edc
I'll probably move to cmake or autofools at some point. If, on the odd case that there are many patches flowing in, then I'll throw the code into subversion somewhere.
The Rasterman has recommended cloning a few hardware remotes as themes. I'm not sure when I'll have the time for that but it should be interesting.
Oh yeah, and if you like this then you feel free to contribute toward a memory card or IR transmitter for my future tinkering. Or send any old n900's you have laying around ;) Hey... I can dream right?
Sources hot off the presses: clawmotia-0.0.1. You'll also notice the diablo binary for clawmotia on there so you don't need a scratchbox to get going. I put the clawmotia.edj up too so you don't even need to compile that if you don't wanna.
Dependencies on the device: Qt, qedje. You probably have most of the requirements for the latter installed if you have canola2 on the device. There are deb files for qedje on its web site. Qt is also packaged for the n810. I don't have debs for clawmotia yet, feel free to send the debian directory to do that.
You'll obviously need a MythTV server (and client) to use the remote. It uses the web interface to mythtv so you'll want that installed on the server machine too. The various configuration settings are passed in using environment variables in version 0.0.1. See the script start-clawmotia.sh for these settings. You will also need to be in the same directly as the .edj file when you start clawmotia as it expects that at this point. I install the script, binary and edj into /usr/local/bin on the device.
Setting CLAWMOTIA_FS=1 will start clawmotia full screen. There is a bug in version 0.0.1 where it will start but not be shown if started full screen. You have to switch to another app and back to clawmotia to see it. I was planning to fix this before release, but it seems some folks want to have a tinker with it so I'll fix that bug soon.
If you want to change the buttons or layout, you only need to edit the clawmotia.edc file. If you are good with the gimp, you might like to reate a theme at 800x480. As long as you use layers it should be a fairly mechanical process to create an edc file for your theme.
To make a button, define its image in the images section and call the three macros COMMAND_BUTTON, COMMAND_BUTTON_ICON, and COMMAND_PROGRAMS. COMMAND_BUTTON sets up the area of the screen that will react to your button with an optional text label for the button as arg2. COMMAND_BUTTON_ICON links a COMMAND_BUTTON with an image file. And COMMAND_PROGRAMS links a button with the exact, HTTP encoded string to send to the MythTV server. HTTP encoding doesn't effect single keys much, but Ctrl+w becomes Ctrl%2Bw. The first argument to all the three macros is the button name and should be the same for the same button. Taking a look at what is there and remembering to use all three macros to give (a) initial setup, (b) image file, (c) action should get you one the way.
You can update the theme by compiling it with edje_cc and copying it to the device. The application itself shouldn't need any changes.
Note that the bevel buttons are separate to the actual button function image. So if you like you can remote the button by setting 255 to 0 in the lines
color, 0 0 0 255
in the COMMAND_BUTTON definition in clawmotia.edc
I'll probably move to cmake or autofools at some point. If, on the odd case that there are many patches flowing in, then I'll throw the code into subversion somewhere.
The Rasterman has recommended cloning a few hardware remotes as themes. I'm not sure when I'll have the time for that but it should be interesting.
Oh yeah, and if you like this then you feel free to contribute toward a memory card or IR transmitter for my future tinkering. Or send any old n900's you have laying around ;) Hey... I can dream right?
Thursday, October 29, 2009
I want my MythTV: Clawmotia
I noticed that there are many packages which turn a maemo device into a remote control. Unfortunately, the MythTV ones I saw either didn't install or were not what I was looking for. Thus clawmotia was born yesterday.
I'm using edje for the interface and qedje to actually create the UI. The program logic uses Qt to talk to a MythTV server and send the commands to it. This relies on knowing there your server is, and what client you want to control which are both passed in as environment variables. Really simple stuff, but it works well.
You'll need the Web interface on the MythTV server machine too. The upshot is that you only need a wifi connection and you can control any MythTV client you want. No bluetooth or IR dongles needed.
As you can see, there are some nasty graphical artifacts on the buttons which I'm not sure if its the evas engine or something to do with qedje. The black parts on the buttons are not there on the desktop.
As it uses edje, you can create different themes and layouts, compile the edje on your desktop machine and just scp it to the device to change the layout and button functionality. Including cute little slide in and out panels for the more rarely used controls. Hopefully I can convince somebody with more artistic ability than me to do just that. Source will be released in the next few days.
I'm using edje for the interface and qedje to actually create the UI. The program logic uses Qt to talk to a MythTV server and send the commands to it. This relies on knowing there your server is, and what client you want to control which are both passed in as environment variables. Really simple stuff, but it works well.
You'll need the Web interface on the MythTV server machine too. The upshot is that you only need a wifi connection and you can control any MythTV client you want. No bluetooth or IR dongles needed.
As you can see, there are some nasty graphical artifacts on the buttons which I'm not sure if its the evas engine or something to do with qedje. The black parts on the buttons are not there on the desktop.
As it uses edje, you can create different themes and layouts, compile the edje on your desktop machine and just scp it to the device to change the layout and button functionality. Including cute little slide in and out panels for the more rarely used controls. Hopefully I can convince somebody with more artistic ability than me to do just that. Source will be released in the next few days.
Welcome to the machine...
Warning: this is a hello p.kde post, so now I can read minds; the page down key never looked so appealing!
OK, so I've been working on filemanagers for "a long time now". Back in the days when the Amiga was hot and hardware memory managers were a wonderful thing that would make a segv not take down the whole machine. Over the last roughly ten years I've been hacking on libferris. In that time it has expanded from a virtual filesystem to include index and search, virtualized extended attributes, RDF support, FUSE and rsync support and other party tricks.
Those with a keen eye will be asking why I am on this KDE planet when libferris "competes" with KIO, Strigi, Soprano etc. I have quoted compete because there is no clear winner and looser in projects that you do mainly for technical enjoyment. And there is no reason that libferris can not be used in a mixed fashion with other projects.
The last major release of libferris moved its RDF handling from redland to soprano, so at least the triples can be freely intermixed between projects now. On that note, I've recently created a new backend for soprano to use a memory mapped file with multi indexing as the triple store. See the ferris sf.net page for the boostmmapbackend tarball. The backend is in early days, but I'll take it for a walk on the n810 soon to see how well it works in that environment.
I am planning on adding spatial indexing and other tricks to the soprano backend which I've used for my maemo libferris metadata index module:
OK, so I've been working on filemanagers for "a long time now". Back in the days when the Amiga was hot and hardware memory managers were a wonderful thing that would make a segv not take down the whole machine. Over the last roughly ten years I've been hacking on libferris. In that time it has expanded from a virtual filesystem to include index and search, virtualized extended attributes, RDF support, FUSE and rsync support and other party tricks.
Those with a keen eye will be asking why I am on this KDE planet when libferris "competes" with KIO, Strigi, Soprano etc. I have quoted compete because there is no clear winner and looser in projects that you do mainly for technical enjoyment. And there is no reason that libferris can not be used in a mixed fashion with other projects.
The last major release of libferris moved its RDF handling from redland to soprano, so at least the triples can be freely intermixed between projects now. On that note, I've recently created a new backend for soprano to use a memory mapped file with multi indexing as the triple store. See the ferris sf.net page for the boostmmapbackend tarball. The backend is in early days, but I'll take it for a walk on the n810 soon to see how well it works in that environment.
I am planning on adding spatial indexing and other tricks to the soprano backend which I've used for my maemo libferris metadata index module:
libferris maemo audio search by regex on URL from Ben Martin on Vimeo.
Friday, October 23, 2009
RDF on the device
I've started work on a memory mapped soprano RDF backend. Given that mobile devices use flash for their permanent storage, an RDF backend designed using primary storage algorithms should work well on maemo.
While the SPARQL implementation is "far from complete", it can already evaluate some common queries very very quickly. I have some triple matching and the ability to have multiple regex filter statements. Other filters and more complete SPARQL lanugage support should follow in time, as time permits... patches accepted etc and so on.
For those interested, see soprano-boostmmapbackend on my main sf.net page.
While the SPARQL implementation is "far from complete", it can already evaluate some common queries very very quickly. I have some triple matching and the ability to have multiple regex filter statements. Other filters and more complete SPARQL lanugage support should follow in time, as time permits... patches accepted etc and so on.
For those interested, see soprano-boostmmapbackend on my main sf.net page.
Wednesday, October 14, 2009
Libferris, Soprano, Extended Attributes... the Ménage à trois
If you want to store metadata in a filesystem, there are Extended Attributes (EA). The kernel interface allows you to store key=value metadata in these EAs for each file on your filesystem. The catch is that kernel EA are limited in size, sometimes performance is poor, and some systems do not support EA either natively or in the kernels that Linux distributions ship. The classic example of the latter is NFS, which can have EA support patched in, but many distros do not do that.
In libferris the EA interface is virtualized along with the filesystem. So if you are using XFS (or ext3/4 with the right options) then libferris will let you read and write EA to the kernel filesystem. For other filesystems, libferris stores the EA behind the scenes in RDF for you. The difference is no seen by applications, its all just EA... magically every filesystem supports read/write EA.
Version 1.4.0 and above of libferris use Soprano and optionally Nepomuk for RDF support. To take RDF for a spin from the filesystem lets use FerrisFUSE and the normal console tools...
$ mkdir -p /tmp/RDFTESTING/backing /tmp/RDFTESTING/fs
$ date >| /tmp/RDFTESTING/backing/df1.txt
$ ferrisfs -u /tmp/RDFTESTING/backing /tmp/RDFTESTING/fs
As you can see, backing is where the filesystem is and fs is where you can access backing through libferris. You could just as easily use a HTTP server or emacs as your backing filesystem, anything that libferris can see is up for grabs.
The below uses the attr command to set and get and Extended Attribute. Assuming that the /tmp kernel filesystem does not allow EA to be set by users. If in doubt, use an NFS directory and you'll almost certainly not be able to attr -s directly on the backing filesystem.
$ cd /tmp/RDFTESTING/fs
$ cat df1.txt
Wed Oct 14 22:33:04 EST 2009
$ attr -s foo -V bar df1.txt
Attribute "foo" set to a 3 byte value for df1.txt:
bar
$ attr -g foo df1.txt
Attribute "foo" had a 3 byte value for df1.txt:
bar
So, you might ask where does all this metadata go and come from. And what does the RDF schema look like... The best solution would be to use SPARQL to query the data, but the default store is still a redland one with libferris 1.4.0 and its sparql is very, very slow. 1+ minutes for a simple query on this data vs <2 seconds using the sesame2 soprano backend. So the fastest way to explore the redland RDF store is to export the whole RDF store and grep it for now. Hopefully virtuoso and/or my own soprano backend will save the day in the future :/
$ cd ~/.ferris/rdfdb
$ time sopranocmd --backend redland \
--settings name=myrdf export t
I have changed the URIs to use prefixes in the grep output... as you would expect, there is data attached to the df1.txt URL.
$ grep RDFTE t ferris:uuid ferris:93f22bd8-b8be-11de-8e06-001bfc4f043c .
That UUID node has an mtime and a out-of-band-ea bnode.
$ grep 93f22bd8-b8be-11de-8e06-001bfc4f043c t
ferris:93f22bd8-b8be-11de-8e06-001bfc4f043c ferris:mtime "1255523953"^^ .
ferris:93f22bd8-b8be-11de-8e06-001bfc4f043c ferris:out-of-band-ea _:r1255523601r5448r1 .
And the bnode has the EA foo=bar set on it.
$ grep r1255523601r5448r1 t
_:r1255523601r5448r1 ferris:user.foo "bar"^^ .
As you see, the UUID node has a mtime assoicated with it, this way libferris can tell if you have updated any RDF values for a file, and it becomes like an additional ctime check available to libferris and used for example when indexing files.
The gain of having the UUID node use a bnode is that many files can share the same RDF metadata. This is useful if you can access the same file from multiple paths or if files get moved on file servers and you want to relink the old RDF metadata to the file with the new path. I mention file servers here because libferris will track the metadata for you if you use ferrismv/ferriscp, but if somebody moves a file on the file server you've got to have a way to tell libferris about that change.
Fileserver movements are a common enough thing that libferris can automatically relink RDF nodes for you. There are also smushing tools available to help with the task. But that's a story for another post.
In libferris the EA interface is virtualized along with the filesystem. So if you are using XFS (or ext3/4 with the right options) then libferris will let you read and write EA to the kernel filesystem. For other filesystems, libferris stores the EA behind the scenes in RDF for you. The difference is no seen by applications, its all just EA... magically every filesystem supports read/write EA.
Version 1.4.0 and above of libferris use Soprano and optionally Nepomuk for RDF support. To take RDF for a spin from the filesystem lets use FerrisFUSE and the normal console tools...
$ mkdir -p /tmp/RDFTESTING/backing /tmp/RDFTESTING/fs
$ date >| /tmp/RDFTESTING/backing/df1.txt
$ ferrisfs -u /tmp/RDFTESTING/backing /tmp/RDFTESTING/fs
As you can see, backing is where the filesystem is and fs is where you can access backing through libferris. You could just as easily use a HTTP server or emacs as your backing filesystem, anything that libferris can see is up for grabs.
The below uses the attr command to set and get and Extended Attribute. Assuming that the /tmp kernel filesystem does not allow EA to be set by users. If in doubt, use an NFS directory and you'll almost certainly not be able to attr -s directly on the backing filesystem.
$ cd /tmp/RDFTESTING/fs
$ cat df1.txt
Wed Oct 14 22:33:04 EST 2009
$ attr -s foo -V bar df1.txt
Attribute "foo" set to a 3 byte value for df1.txt:
bar
$ attr -g foo df1.txt
Attribute "foo" had a 3 byte value for df1.txt:
bar
So, you might ask where does all this metadata go and come from. And what does the RDF schema look like... The best solution would be to use SPARQL to query the data, but the default store is still a redland one with libferris 1.4.0 and its sparql is very, very slow. 1+ minutes for a simple query on this data vs <2 seconds using the sesame2 soprano backend. So the fastest way to explore the redland RDF store is to export the whole RDF store and grep it for now. Hopefully virtuoso and/or my own soprano backend will save the day in the future :/
$ cd ~/.ferris/rdfdb
$ time sopranocmd --backend redland \
--settings name=myrdf export t
I have changed the URIs to use prefixes in the grep output... as you would expect, there is data attached to the df1.txt URL.
$ grep RDFTE t
That UUID node has an mtime and a out-of-band-ea bnode.
$ grep 93f22bd8-b8be-11de-8e06-001bfc4f043c t
ferris:93f22bd8-b8be-11de-8e06-001bfc4f043c ferris:mtime "1255523953"^^
ferris:93f22bd8-b8be-11de-8e06-001bfc4f043c ferris:out-of-band-ea _:r1255523601r5448r1 .
And the bnode has the EA foo=bar set on it.
$ grep r1255523601r5448r1 t
_:r1255523601r5448r1 ferris:user.foo "bar"^^
As you see, the UUID node has a mtime assoicated with it, this way libferris can tell if you have updated any RDF values for a file, and it becomes like an additional ctime check available to libferris and used for example when indexing files.
The gain of having the UUID node use a bnode is that many files can share the same RDF metadata. This is useful if you can access the same file from multiple paths or if files get moved on file servers and you want to relink the old RDF metadata to the file with the new path. I mention file servers here because libferris will track the metadata for you if you use ferrismv/ferriscp, but if somebody moves a file on the file server you've got to have a way to tell libferris about that change.
Fileserver movements are a common enough thing that libferris can automatically relink RDF nodes for you. There are also smushing tools available to help with the task. But that's a story for another post.
Tuesday, October 13, 2009
libferris 1.4.x - Nepomuk!
So finally RDF handling is done using Soprano (The core of KDE4's nepomuk RDF library). Initially, ferris defaults to using redland in-process for RDF, so the code paths are very similar to what 1.3.x of libferris was using.
But the huge gain is that you can use the nepomukserver or the sesame2 soprano backends if you like. I'd also dearly love to write my own soprano backend, and am bashing out a design for it, but that's a story for another post.
I've run some initial benchmarks, and because much of the speed dependant code paths hit triples directly, giving S+P and looking up Object I suspect I'm not hitting the main benchmarked paths. For example, I've found that some SPARQL queries that perform set based S+P->O lookups run much faster on the sesame2 soprano backend.
Now if only OBS would hurry up, I'd have Fedora 11 packages of ferris 1.4.x available... but in the meantime, the steaming source tarball is on sf.
But the huge gain is that you can use the nepomukserver or the sesame2 soprano backends if you like. I'd also dearly love to write my own soprano backend, and am bashing out a design for it, but that's a story for another post.
I've run some initial benchmarks, and because much of the speed dependant code paths hit triples directly, giving S+P and looking up Object I suspect I'm not hitting the main benchmarked paths. For example, I've found that some SPARQL queries that perform set based S+P->O lookups run much faster on the sesame2 soprano backend.
Now if only OBS would hurry up, I'd have Fedora 11 packages of ferris 1.4.x available... but in the meantime, the steaming source tarball is on sf.
Thursday, October 8, 2009
User overlay virtual softlinks
This was introduced in libferris 1.3.8, The feature lets you have libferris create virtual softlinks for you when a directory is read. Overlay links are setup in the config file ~/.ferris/user-overlay-links.xml. The file sets up regex matches, when the regex is matched to a directory, a virtual link is created to a target location. The target location must actually exist at the moment, no dangling links are permitted. There is an example XML file in the dot-files subdirectory of the tarball.
For the gphoto:// filesystem I have this which creates the link G7 -> Canon PowerShot G7 (PTP mode) in the root of the gphoto:// filesystem.
If the camera as a stable subdirectory that new images are written too, then a link right into that can be made in the root, "latest" takes me right to the images subsubdirectory.
It seemed like a good idea to abstract this out of any specific filesystem like upnp and gphoto so virtual links can be made anywhere. In particular, for upnp a/v devices sometimes the server likes to use an insanely long name for itself, with virtual soft links, the client machine can give the server a nice short name of it's choosing.
For the gphoto:// filesystem I have this which creates the link G7 -> Canon PowerShot G7 (PTP mode) in the root of the gphoto:// filesystem.
<user-overlay-links>
<link-by-regex>
<vlink>
<match>^gphoto:[/]+$</match>
<target>Canon PowerShot G7 (PTP mode)</target>
<link-name>G7</link-name>
</vlink>
...
...
<vlink>
<match>^gphoto:[/]+$</match>
<target>gphoto://G7/store_00010001/DCIM/102CANON</target>
<link-name>latest</link-name>
</vlink>
...
It seemed like a good idea to abstract this out of any specific filesystem like upnp and gphoto so virtual links can be made anywhere. In particular, for upnp a/v devices sometimes the server likes to use an insanely long name for itself, with virtual soft links, the client machine can give the server a nice short name of it's choosing.
Thursday, September 17, 2009
Race to do nothing!
In the early days there was prelink as part of maemo. It disappeared in the n810 distribution ranges. I noticed that there is no armel deb in debian for prelink, but there were murmurs of folks hacking it to work. Push finally came to shove, and as I like C++ many of my apps on maemo have quite a few symbols that need to be resolved before the app can run.
The long story short, prelink_0.0.20090311 is up at my repo. Use at your own risk, if your device breaks you get to keep both pieces etc etc.
Initial benchmarking: for an app that uses a few libraries, is C++, and has quite a few symbols that the dynamic linker has to attend to. For a complete run before it was 5.2 seconds on a warm start, with prelink of the binary it is 2.5 seconds. Considering that in that 2.5 the app itself has to completely run, there was a huge amount of time spent, err wasted, in the dynamic linker.
Of course, if I can get hidden symbols to work too then that 2.5 might drop back to 1.5-2 seconds. But for hidden symbols you have to either wrap things in pragmas or declare each function and class as exposed or not, so for a large library its quite intrusive. But the prelink is a huge gain for no code changes, so far at least. YMMV.
It's a touch ironic that Nokia recommends using prelink for embedded devices. But that is for qtopia... maybe for the next distro prelink will reappear in the default installation.
The long story short, prelink_0.0.20090311 is up at my repo. Use at your own risk, if your device breaks you get to keep both pieces etc etc.
Initial benchmarking: for an app that uses a few libraries, is C++, and has quite a few symbols that the dynamic linker has to attend to. For a complete run before it was 5.2 seconds on a warm start, with prelink of the binary it is 2.5 seconds. Considering that in that 2.5 the app itself has to completely run, there was a huge amount of time spent, err wasted, in the dynamic linker.
Of course, if I can get hidden symbols to work too then that 2.5 might drop back to 1.5-2 seconds. But for hidden symbols you have to either wrap things in pragmas or declare each function and class as exposed or not, so for a large library its quite intrusive. But the prelink is a huge gain for no code changes, so far at least. YMMV.
It's a touch ironic that Nokia recommends using prelink for embedded devices. But that is for qtopia... maybe for the next distro prelink will reappear in the default installation.
Saturday, September 5, 2009
geotagged image files and libferris
Tip of the day, if you have a geotagged image file such as those produced using Location Tracker on Symbian phones, then you can use libferris to start Google Earth showing you that image on a map. Just run
ferris-open-google-earth-for-context.pl
with the image(s) you want to see. Its a fairly quick way to show the movements of the day on a map. I should also add timeline support there I guess, so you can retrace a trip in the order it occurred using just waypoint image files.
This also works with any type of file that is location tagged with libferris. If you have a digital long/lat then you can view it on a map (TM).
ferris-open-google-earth-for-context.pl
with the image(s) you want to see. Its a fairly quick way to show the movements of the day on a map. I should also add timeline support there I guess, so you can retrace a trip in the order it occurred using just waypoint image files.
This also works with any type of file that is location tagged with libferris. If you have a digital long/lat then you can view it on a map (TM).
Tuesday, September 1, 2009
n810 and gphoto
After noticing that gphoto is not packaged for maemo I did a bit of digging, seeing sad news from those who tried getting it going. But I thought I'd try with 2.4.6 to see if things are better now.
For those who want to tinker, see my packages. The library came across OK but gphoto itself uses a newer debhelper, so I just plucked the binary itself into the repo. If you want to use this with libferris, then expand the additional libraries into /usr/local/lib/ferris/plugins/context on the device.
I found that once USB mode was set to host, I could see the camera with gphoto but as reported in the past, there were frequent crashes. This is really unfortunate as all the blocks where in place for the command I was chasing:
Most of the time using gphoto locks everything and about 5-10 seconds later the device restarts. I might have to hunt for a newer kernel for diablo... or wait until I have gphoto support on an n9 (nn?), a P+S wifi camera, and a decent data plan...
For those who want to tinker, see my packages. The library came across OK but gphoto itself uses a newer debhelper, so I just plucked the binary itself into the repo. If you want to use this with libferris, then expand the additional libraries into /usr/local/lib/ferris/plugins/context on the device.
I found that once USB mode was set to host, I could see the camera with gphoto but as reported in the past, there were frequent crashes. This is really unfortunate as all the blocks where in place for the command I was chasing:
# ferriscp -av \
gphoto://Canon.../DCIM/102CANON/IMG_2442.JPG \
flickr://me/upload
Most of the time using gphoto locks everything and about 5-10 seconds later the device restarts. I might have to hunt for a newer kernel for diablo... or wait until I have gphoto support on an n9 (nn?), a P+S wifi camera, and a decent data plan...
Sunday, August 30, 2009
Images from n810 to the Web
New debs allowing you to copy to flickr, 23hq, vimeo, youtube etc from the n810 are available. The next step is closer integration, tapping into the "send to" menu to let you send any image file to these places. Hey, its just a filesystem now ;)
One interesting possibility is using the n810 to shuttle images from a digital camera to the net. This gets around the vendor lock-in that might be present on your camera because the n810 just grabs the images from the cam and then shuttles them onward to wherever.
I also updated libferris to allow you to authenticate from the console, just the ticket to let you set things up over ssh with your desktop browser.
OK, for maemo, check the repository and you'll want ferris, libferris, and libferrisui at least. Version 1.3.6.
If you get into dependency trouble, see the build order to get an idea of the tree. Sorry its a bit hard to get on the device, hopefully stuff will move into extras in time.
Anyone who has used desktop flickr tools, you'll know about the whole authenticate before you buy thing. The below commands setup 23hq and copy an image over to it.
Hopefully things will become easier in the future: installation, setup etc. Of course, streaming from the webcam to youtube will rock, but baby steps as they say.
One interesting possibility is using the n810 to shuttle images from a digital camera to the net. This gets around the vendor lock-in that might be present on your camera because the n810 just grabs the images from the cam and then shuttles them onward to wherever.
I also updated libferris to allow you to authenticate from the console, just the ticket to let you set things up over ssh with your desktop browser.
OK, for maemo, check the repository and you'll want ferris, libferris, and libferrisui at least. Version 1.3.6.
If you get into dependency trouble, see the build order to get an idea of the tree. Sorry its a bit hard to get on the device, hopefully stuff will move into extras in time.
Anyone who has used desktop flickr tools, you'll know about the whole authenticate before you buy thing. The below commands setup 23hq and copy an image over to it.
$ ferris-capplet-auth --list-auth-sites
...
flickr
23hq
...
$ ferris-capplet-auth --auth-with-site 23hq
...
Grant Auth following URL...
http://www.23hq.com/services/auth/?api_key=...
Then press return to continue...
<Open above link in browser, grant auth, hit return>
Done...
$ ferriscp -av 26082009071.jpg 23hq://me/upload
Hopefully things will become easier in the future: installation, setup etc. Of course, streaming from the webcam to youtube will rock, but baby steps as they say.
Friday, August 7, 2009
Them pesky web serviceses...
It always stuck me as a little odd that you have specific applications to upload to flickr, another little app with a cute interface for throwing video files at webserviceY etc. I should also mention that all of this is coming to the maemo build shortly... what better place to unlock web services than a tablet or phone, neh?
What ever happened to the unix philosophy, "everything is a file". Then you could just "cp" to flickr, write your facebook status in bash or C-x C-s your mood to facebook://status.
I added some support for the flickr API to ferris last year, and the recent 1.3.5 release expands that and adds more web service goodness. But, which of these nasty, closed source services should be supported? The answer: it doesn't matter, there just filesystems! Of course, I'd prefer to mount a FOSS, GNU, Free, Privacy respecting server, and who wouldn't.. but for some things you want to just cp to youtube and forget about it.
For facebook I have three objects at facebook:// level. A status file which is read/write, a recent directory which shows the feed of stuff your chums are up to, and a contacts directory with virtual vcard files for your pals. I even added in the profile picture of each friend into their v-vcard. Unfortunately facebook doesn't allow you to get phone number info through the web API, which is strange because you can just simulate a login and swipe it by screen scraping, but that's nasty T&C violating thinking neh?
The flickr API mounting should work, to some extent with any service offering that API. For example, 23hq offers free hosting which might appeal. The extensions in libferris 1.3.5 let you see your contacts in a directory, find images by photo ID, browse your photosets and images not yet in a photoset. If you look in your contacts directory, you'll see their photostream and favs as directories.
Note that the thumbnail image is available from flickr through libferris as 32bit RGBA decoded data, so a libferris client can graphically browse a flickr API photoset just as it can a digital camera mounted with libferris. The thumbail EA is the same. Of course, if you copy an image from flickr:// or 23hq:// libferris grabs the largest (or original) image from the site and supplies it to you. Feel the unlocking of captive data. Oh, and upload works as it did last year...
Finally, the mounting of Google docs and spreadsheets. This also has support for appending to plain text files, so you can redirect stdout to append to a google docs text file just as you can >> /tmp/foo.txt.
Spreadsheet mounting goes right down to the cell. So you can use google as a calculator from the command line if it tickles you...
And there are vimeo, youtube and others still needing mounting... but thats what 1.3.6 exists for right? Feel free to send in patches for your fav service! See plugins/context/facebook et al in the source.
What ever happened to the unix philosophy, "everything is a file". Then you could just "cp" to flickr, write your facebook status in bash or C-x C-s your mood to facebook://status.
I added some support for the flickr API to ferris last year, and the recent 1.3.5 release expands that and adds more web service goodness. But, which of these nasty, closed source services should be supported? The answer: it doesn't matter, there just filesystems! Of course, I'd prefer to mount a FOSS, GNU, Free, Privacy respecting server, and who wouldn't.. but for some things you want to just cp to youtube and forget about it.
For facebook I have three objects at facebook:// level. A status file which is read/write, a recent directory which shows the feed of stuff your chums are up to, and a contacts directory with virtual vcard files for your pals. I even added in the profile picture of each friend into their v-vcard. Unfortunately facebook doesn't allow you to get phone number info through the web API, which is strange because you can just simulate a login and swipe it by screen scraping, but that's nasty T&C violating thinking neh?
The flickr API mounting should work, to some extent with any service offering that API. For example, 23hq offers free hosting which might appeal. The extensions in libferris 1.3.5 let you see your contacts in a directory, find images by photo ID, browse your photosets and images not yet in a photoset. If you look in your contacts directory, you'll see their photostream and favs as directories.
Note that the thumbnail image is available from flickr through libferris as 32bit RGBA decoded data, so a libferris client can graphically browse a flickr API photoset just as it can a digital camera mounted with libferris. The thumbail EA is the same. Of course, if you copy an image from flickr:// or 23hq:// libferris grabs the largest (or original) image from the site and supplies it to you. Feel the unlocking of captive data. Oh, and upload works as it did last year...
Finally, the mounting of Google docs and spreadsheets. This also has support for appending to plain text files, so you can redirect stdout to append to a google docs text file just as you can >> /tmp/foo.txt.
Spreadsheet mounting goes right down to the cell. So you can use google as a calculator from the command line if it tickles you...
$ export SPREAD=google://spreadsheets/smalltest1/Sheet1
$ echo 5 | ferris-redirect --ea b $SPREAD/9
$ echo 23 | ferris-redirect --ea c $SPREAD/9
$ echo -n "=sum(B9,C9)" | ferris-redirect --ea d $SPREAD/9
$ fcat -a d $SPREAD/9
28
And there are vimeo, youtube and others still needing mounting... but thats what 1.3.6 exists for right? Feel free to send in patches for your fav service! See plugins/context/facebook et al in the source.
Wednesday, July 29, 2009
select...group by artist
So I finally updated my Amarok mounting to work with 2.x. I'm still having a few issues with the 2.x of 'rok but that's another story.
One of the cool things about being able to mount things with libferris is that everything above ferris can also see them, XQuery, SQLite etc. For example, the amarok://playlist URL shows, as you would imagine, the playlist as it currently appears in the running amarok application.
The following little SQLite is fairly self explanatory:
.load libferrissqlitevtable.so
create virtual table fs using libferris(
'amarok://playlist',
'',
name text,
title text,
size int,
length int,
playtime text,
link-target text,
ferris-delegate-url text
);
select name, playtime, title from fs
order by name desc;
select avg(length) from fs;
And for a particular album, the playlist and average track time.
9|352|Absolutely Curtains
8|246|Stay
7|255|Free Four
6|274|Childhood's End
5|260|Mudmen
4|309|Wots ... Uh The Deal
3|187|The Gold It's In The ...
2|209|Burning Bridges
1|147|When You're In
0|185|Obscured By Clouds
242.4
Of course, a litttle group by artist, album... would make the SQL more generic:
select avg(length),max(artist),max(album)
from fs
group by artist, album;
285.777777777778|Pink Floyd|Dark Side of the Moon
242.4|Pink Floyd|Obscured By Clouds
One of the cool things about being able to mount things with libferris is that everything above ferris can also see them, XQuery, SQLite etc. For example, the amarok://playlist URL shows, as you would imagine, the playlist as it currently appears in the running amarok application.
The following little SQLite is fairly self explanatory:
.load libferrissqlitevtable.so
create virtual table fs using libferris(
'amarok://playlist',
'',
name text,
title text,
size int,
length int,
playtime text,
link-target text,
ferris-delegate-url text
);
select name, playtime, title from fs
order by name desc;
select avg(length) from fs;
And for a particular album, the playlist and average track time.
9|352|Absolutely Curtains
8|246|Stay
7|255|Free Four
6|274|Childhood's End
5|260|Mudmen
4|309|Wots ... Uh The Deal
3|187|The Gold It's In The ...
2|209|Burning Bridges
1|147|When You're In
0|185|Obscured By Clouds
242.4
Of course, a litttle group by artist, album... would make the SQL more generic:
select avg(length),max(artist),max(album)
from fs
group by artist, album;
285.777777777778|Pink Floyd|Dark Side of the Moon
242.4|Pink Floyd|Obscured By Clouds
Wednesday, July 22, 2009
Done bought me a new calculator...
So, the next release of libferris will include support for mounting google spreadsheets. This works right through to the file manager now. You can mount a spreadsheet as a filesystem and edit it with ego. Changes to cell a written to google, any formulas that are updated appear in the file manager right away, and of course if you are viewing the worksheet in a web browser its updated too.
I'm looking forward to compiling the latest ferris for maemo so that you can edit stuff on the tablet and have it change on your desktop automatically ;)
export SPREAD=google://docs/spreadsheets/smalltest1/Sheet1
$ echo 5 | ferris-redirect --ea b $SPREAD/9
$ echo 23 | ferris-redirect --ea c $SPREAD/9
$ echo -n "=sum(B9,C9)" | ferris-redirect --ea d $SPREAD/9
$ fcat -a d $SPREAD/9
28
And of course, if you use the first row for headings...
$ echo -n result | ferris-redirect --ea d $SPREAD/1
$ fcat -a result $SPREAD/9
28
I'm looking forward to compiling the latest ferris for maemo so that you can edit stuff on the tablet and have it change on your desktop automatically ;)
export SPREAD=google://docs/spreadsheets/smalltest1/Sheet1
$ echo 5 | ferris-redirect --ea b $SPREAD/9
$ echo 23 | ferris-redirect --ea c $SPREAD/9
$ echo -n "=sum(B9,C9)" | ferris-redirect --ea d $SPREAD/9
$ fcat -a d $SPREAD/9
28
And of course, if you use the first row for headings...
$ echo -n result | ferris-redirect --ea d $SPREAD/1
$ fcat -a result $SPREAD/9
28
Thursday, June 25, 2009
preupgrade -- caveat emptor
A good rant every now and then?
I have been upgrading my Fedora machines by hand using yum. This seemed to work OK, for a machine with minimal tweaking there usually were not so many dependency issues that had to be solved by hand. Though it still is a time consuming process for a handful of machines.
So, I recently found out that I'm bad boy (oooh yeah!), and should have been using preupgrade. Looking at TFM I thought, "fair enough, upgrading a non live system seems like a wise thing". So I gave it a whirl. Which ended up in a strange exception in anaconda and after reboot /var/lib/rpm was missing. yay. So I trip to the reinstall Fedora 11 from DVD shops later and I was back in the mix. But there were/are still some other machines that needed updating to 11.
Once bitten, twice masocistic I thought. So I decided to try on another machine. It might have been something I tweaked on the laptop over the many Fedora updates etc that fluffed up preupgrade and anaconda...
Luckily the second time I didn't loose any data. For one thing, my /var/lib/rpm shouldn't be deleted assumption from before the first install was missing and everything was backed up (twice, I had running incremental baculas too). But preupgrade won't touch a system where /boot is on a RAID. So the second attempt stopped nice and fast.
I might try preupgrade again on the laptop next time. I left things fairly stock during the Fedora 11 install there just so preupgrade could possibly work its magic in the future. But I'm wondering if the pain of yum updating across versions makes sticking with Fedora worthwhile. I have better things to do than manually update and tweak things and if other distros don't chomp days of my life every 6-12 months then maybe I should check them out.
It's a bit sad that /boot on a software RAID-1 is such a rare setup that its not supported.
And that concludes this rant... was it as good for ya?
I have been upgrading my Fedora machines by hand using yum. This seemed to work OK, for a machine with minimal tweaking there usually were not so many dependency issues that had to be solved by hand. Though it still is a time consuming process for a handful of machines.
So, I recently found out that I'm bad boy (oooh yeah!), and should have been using preupgrade. Looking at TFM I thought, "fair enough, upgrading a non live system seems like a wise thing". So I gave it a whirl. Which ended up in a strange exception in anaconda and after reboot /var/lib/rpm was missing. yay. So I trip to the reinstall Fedora 11 from DVD shops later and I was back in the mix. But there were/are still some other machines that needed updating to 11.
Once bitten, twice masocistic I thought. So I decided to try on another machine. It might have been something I tweaked on the laptop over the many Fedora updates etc that fluffed up preupgrade and anaconda...
Luckily the second time I didn't loose any data. For one thing, my /var/lib/rpm shouldn't be deleted assumption from before the first install was missing and everything was backed up (twice, I had running incremental baculas too). But preupgrade won't touch a system where /boot is on a RAID. So the second attempt stopped nice and fast.
I might try preupgrade again on the laptop next time. I left things fairly stock during the Fedora 11 install there just so preupgrade could possibly work its magic in the future. But I'm wondering if the pain of yum updating across versions makes sticking with Fedora worthwhile. I have better things to do than manually update and tweak things and if other distros don't chomp days of my life every 6-12 months then maybe I should check them out.
It's a bit sad that /boot on a software RAID-1 is such a rare setup that its not supported.
And that concludes this rant... was it as good for ya?
Thursday, April 30, 2009
7th of May: A libferris talk in Ede, Netherlands.
If you are in Europe and want to hear about libferris and other cool storage stuff (register &) drop by the
NLUUG Spring Conference on Storage on the 7th of May.
I will of course have a maemo unit running libferris there as well as a laptop. Carrying a server with ferris is a bit too much though :) I'm brewing up the slides now with XQuery, KML, and SQLite goodness. Probably not the first three terms you think of when the words Virtual Filesystem are muttered.
NLUUG Spring Conference on Storage on the 7th of May.
I will of course have a maemo unit running libferris there as well as a laptop. Carrying a server with ferris is a bit too much though :) I'm brewing up the slides now with XQuery, KML, and SQLite goodness. Probably not the first three terms you think of when the words Virtual Filesystem are muttered.
Sunday, April 12, 2009
Keeping it sorted with libferris
Click through for the 720 HD version...
Sorting directories with libferris and ego on multiple metadata values at once from Ben Martin on Vimeo.
This is the libferris file manager, it is called ego.
Although the interface looks quite simple, there
are context menus associated with most things.
There are also a vast number of sidepanels for searching,
filtering, bookmarks and editing metadata and tags.
Today I'm talking about sorting.
Of course, you can click a column to sort by it, and
click again to reverse the sort.
You can also set that sort order to be the default for
this directory, or the entire filesystem tree rooted
at this directory. Subdirectories will inherit settings
for a tree but can explicitly override them too.
You can also override what ordering you want for each
column, like using the version sort of ls for any column.
If you sort on something with many files having the same
value, you can append another column to the sort order.
For example, sorting by image width, then by file name
for files that have the same width.
You can sort by any number of columns. This is built
into libferris, so you can sort on many metadata fields
with ferrisls or any libferris client.
Sorting directories with libferris and ego on multiple metadata values at once from Ben Martin on Vimeo.
This is the libferris file manager, it is called ego.
Although the interface looks quite simple, there
are context menus associated with most things.
There are also a vast number of sidepanels for searching,
filtering, bookmarks and editing metadata and tags.
Today I'm talking about sorting.
Of course, you can click a column to sort by it, and
click again to reverse the sort.
You can also set that sort order to be the default for
this directory, or the entire filesystem tree rooted
at this directory. Subdirectories will inherit settings
for a tree but can explicitly override them too.
You can also override what ordering you want for each
column, like using the version sort of ls for any column.
If you sort on something with many files having the same
value, you can append another column to the sort order.
For example, sorting by image width, then by file name
for files that have the same width.
You can sort by any number of columns. This is built
into libferris, so you can sort on many metadata fields
with ferrisls or any libferris client.
Sunday, March 22, 2009
Packages for maemo and Fedora 10!
So, long story short, the place I was hosting my maemo repository was on a host at the university I did my PhD at. The server has been suffering unpredictable hardware issues so I have now moved the repository. There is a new subdomain fuuko.libferris.com where I'll be putting up some libferris related stuff, including packages.
In particular, the new libferris maemo repository is now up.
Also note that I have binary rpm files for Fedora 10 in both 32 and 64 bit at OBS.
In particular, the new libferris maemo repository is now up.
Also note that I have binary rpm files for Fedora 10 in both 32 and 64 bit at OBS.
Monday, March 16, 2009
10: It's the way of the future; goto 10;
So, boost 1.34.1 was released back in July 2007. It is however still the current boost shipped with Fedora 10. This is extremely unfortunate as there are many cool advancements in boost over the nearly 2 years since 1.34x. At least rawhide has 1.37, even though 1.38 has been released for over a month.
Looks like I'll have to add another switch to libferris configure.ac to drop some features to be able to compile against old boosts. めんどうくさい!
Looks like I'll have to add another switch to libferris configure.ac to drop some features to be able to compile against old boosts. めんどうくさい!
Sunday, March 1, 2009
Like talking to a stranger
Now that I'm on planet Linux Australia too I thought I'd write up a brief howdee post. My main open source project is libferris which is a virtual filesystem with metadata extraction and index+search functionality. It also does a bunch of other interesting stuff, but as I've been writing it for what is getting close to 10 years now, one would imagine it has a few facets.
I mainly package libferris for Fedora and maemo because these are the platforms I tend to use it with myself. A while ago I did some packages for openSUSE and Ubuntu, but its hard to keep motivation for those when I don't use the distro personally. Plus on modern desktop or servers, stuffing ferris on a virtual Fedora machine isn't much of a hassle, given the price of even DDR3 RAM.
I'm also trying to stuff libferris itself into the main Fedora repository. But we'll see how that goes.
If you want to hear about libferris and other filesystem crack then you should come along to this NLUUG event in May. I'll be there presenting libferris and what I think the intranet metadata and search stuff should be doing (and probably will be doing in the libferris world) over the next few years.
As my blog description states, my posts are mainly about libferris but also include maemo stuff and generic C++, XML, RDF, and relational database topics. I seem to waffle on unmerciful about indexing, or so I've been told. Which is hardly surprising given the chunk of libferris devoted to implementing filesystem search through various techniques.
Also, if anybody is looking for user space virtual filesystem hacking or index and search work, then drop me a line :)
I mainly package libferris for Fedora and maemo because these are the platforms I tend to use it with myself. A while ago I did some packages for openSUSE and Ubuntu, but its hard to keep motivation for those when I don't use the distro personally. Plus on modern desktop or servers, stuffing ferris on a virtual Fedora machine isn't much of a hassle, given the price of even DDR3 RAM.
I'm also trying to stuff libferris itself into the main Fedora repository. But we'll see how that goes.
If you want to hear about libferris and other filesystem crack then you should come along to this NLUUG event in May. I'll be there presenting libferris and what I think the intranet metadata and search stuff should be doing (and probably will be doing in the libferris world) over the next few years.
As my blog description states, my posts are mainly about libferris but also include maemo stuff and generic C++, XML, RDF, and relational database topics. I seem to waffle on unmerciful about indexing, or so I've been told. Which is hardly surprising given the chunk of libferris devoted to implementing filesystem search through various techniques.
Also, if anybody is looking for user space virtual filesystem hacking or index and search work, then drop me a line :)
Saturday, February 28, 2009
Tracker: Reading it both ways :(
Normally I refrain from criticising other projects that are "competing" with my little hobby -- libferris. I mention competing in quotes because hobbies don't really compete in a commercial sense. But having read this post recently on planet maemo about Tracker progress I was a little overwhelmed, wondering, was the Tracker code really so bad a year ago?
"In this last year, we refactor (well, almost rewrote) the daemon"
and talks about replacing the crawler code wholesale. Sure progress sometimes includes churn of functionality: reimplementing stuff with the benefit of 20/20 hindsight. But this seemed a little dramatic...
"In this last year, we refactor (well, almost rewrote) the daemon"
and talks about replacing the crawler code wholesale. Sure progress sometimes includes churn of functionality: reimplementing stuff with the benefit of 20/20 hindsight. But this seemed a little dramatic...
Slow as a tree
Hmm, nepomuk integration as the core of amarok's collection manager... now if only flacs without seek tables were not so inconvenient in amarok it would be damn cool.
http://lists.kde.org/?l=amarok&m=120671838014374&q=p3
Finally it seems RDF is making its merry way into the core position of (meta) data sharing on the desktop. A good time to be a semantic hacker.
http://lists.kde.org/?l=amarok&m=120671838014374&q=p3
Finally it seems RDF is making its merry way into the core position of (meta) data sharing on the desktop. A good time to be a semantic hacker.
Wednesday, February 11, 2009
Keeping an index up to date... quickly?
Most of the index engine implementations in libferris will detect when you try to add the same file again, and when it hasn't changed in a meaningful way will just skip reindexing it. This makes it really easy to just use the below command to update the index for a specific filesystem.
$ find /Data | feaindexadd --filelist-stdin
The trick comes in when /Data is an NFS share with 400,000 files on it that you are accessing from a Nokia n810. Or when /Data is a file server that you are indexing from your laptop over wifi or another sluggish, higher latency network.
feaindexadd can be told to directly traverse one or more directories and so you don't have to use find in the above command. But separating out the find from the indexing has a really big advantage: you can update indexes of extremely large, but infrequently changing NFS shares very quickly -- Even over slow networks.
The trick is to do the filesystem traversal on the server side, and just pump the URLs that are interesting to the client machine:
ssh lowaccess@server 'find /Data -mtime -10' \
| feaindexadd --filelist-stdin
Of course, this relies on /Data being the same filesystem on both the server and the client. Otherwise you're in for some fun with sed or awk to mangle the paths to be what the client expects.
And the 10 in the above means that you'll have to run the command from cron within 10 days to maintain a complete index. You might find that doing a "time find /Data" on the client and server has significant performance differences, particularly if the filesystem has many files.
You can always store and search the index from the file server, but for disconnected searching, you really need to have the index itself stored on the maemo device.
$ find /Data | feaindexadd --filelist-stdin
The trick comes in when /Data is an NFS share with 400,000 files on it that you are accessing from a Nokia n810. Or when /Data is a file server that you are indexing from your laptop over wifi or another sluggish, higher latency network.
feaindexadd can be told to directly traverse one or more directories and so you don't have to use find in the above command. But separating out the find from the indexing has a really big advantage: you can update indexes of extremely large, but infrequently changing NFS shares very quickly -- Even over slow networks.
The trick is to do the filesystem traversal on the server side, and just pump the URLs that are interesting to the client machine:
ssh lowaccess@server 'find /Data -mtime -10' \
| feaindexadd --filelist-stdin
Of course, this relies on /Data being the same filesystem on both the server and the client. Otherwise you're in for some fun with sed or awk to mangle the paths to be what the client expects.
And the 10 in the above means that you'll have to run the command from cron within 10 days to maintain a complete index. You might find that doing a "time find /Data" on the client and server has significant performance differences, particularly if the filesystem has many files.
You can always store and search the index from the file server, but for disconnected searching, you really need to have the index itself stored on the maemo device.
Saturday, January 31, 2009
Searching indexes on a maemo device: a customized solution
In a previous post I had a video showing one of the libferris inverted file metadata index plugins on maemo. Unfortunately that implementation was designed for a desktop machine, meaning for many many more files, slow disk head seeks, blazingly fast CPU(s).
While these videos are not as snazzy as some others, I make them so folks can get an idea of the real performance of the software. Saying its "acceptably fast" or takes less than a second is all well and good, but seeing the snail or rabbit move gives an immediate gut feeling if its "fast enough".
The new maemo custom designed metadata architecture uses the mmap() design. I always have my reservations about mmap() because it encourages data structure and algo choices that are directed at an in-memory world. That is all great if you can cache the entire memory mapped file in main memory, but often leads to kittens crying when there isn't enough RAM for that. For maemo with 128mb total RAM this is also an issue, but with the much faster disk seeks of flash storage the relatively poor data clustering of mmap()ed designs is a bit less important for performance. But of course, data clustering and physical layout is always interesting when playing the indexing game.
libferris maemo audio search by regex on URL from Ben Martin on Vimeo.
Shown in the video above is a search using the new "custom mobile device" libferris metadata index. As I continue to type a regular expression to match against the URL, libferris gives some search results on each character entered. Note that this is against an index with over 10,000 URLs in it. This index format will be available in libferris 1.2.7+. The index format sports a probabilistic regular expression prefilter for URL matching. This means that the results of the prefilter might include some false positives, but anything the filter drops out is definitely not a match. You have to run the regex itself on all results of the prefilter to drop out the false positives. The prefilter can drop the number of times a regular expression is evaluated down to less than 100 in 10,000 URLs given only a 3-4 character regex. This helps out oodles when you want to run raw regex searches on big (for a portable device) indexes and you want your results asap.
An unfortunate current side effect of the mmap() design is that the index is architecture dependant. Mainly because some secondary offset-pointer information is 64 or 32 bit width dependant, so its a matter of tracking those peskies down and clamping them to a fixed width. To get around that for now, I indexed the 10,000 odd audio files over NFS from an n810.
Some information for those looking to play, if you are using boost to memory map, you may well discover that atomic_cas32() calls __sync_val_compare_and_swap() which is not actually available on the maemo device (for me). This is with a modified scratchbox using gcc version 4.2.1.
I'm not sure if there is an accepted cas et al library for maemo/armel. I leave the hacks for __sync_val_compare_and_swap() to the reader and their accepted level of concurrency control risk.
While these videos are not as snazzy as some others, I make them so folks can get an idea of the real performance of the software. Saying its "acceptably fast" or takes less than a second is all well and good, but seeing the snail or rabbit move gives an immediate gut feeling if its "fast enough".
The new maemo custom designed metadata architecture uses the mmap() design. I always have my reservations about mmap() because it encourages data structure and algo choices that are directed at an in-memory world. That is all great if you can cache the entire memory mapped file in main memory, but often leads to kittens crying when there isn't enough RAM for that. For maemo with 128mb total RAM this is also an issue, but with the much faster disk seeks of flash storage the relatively poor data clustering of mmap()ed designs is a bit less important for performance. But of course, data clustering and physical layout is always interesting when playing the indexing game.
libferris maemo audio search by regex on URL from Ben Martin on Vimeo.
Shown in the video above is a search using the new "custom mobile device" libferris metadata index. As I continue to type a regular expression to match against the URL, libferris gives some search results on each character entered. Note that this is against an index with over 10,000 URLs in it. This index format will be available in libferris 1.2.7+. The index format sports a probabilistic regular expression prefilter for URL matching. This means that the results of the prefilter might include some false positives, but anything the filter drops out is definitely not a match. You have to run the regex itself on all results of the prefilter to drop out the false positives. The prefilter can drop the number of times a regular expression is evaluated down to less than 100 in 10,000 URLs given only a 3-4 character regex. This helps out oodles when you want to run raw regex searches on big (for a portable device) indexes and you want your results asap.
An unfortunate current side effect of the mmap() design is that the index is architecture dependant. Mainly because some secondary offset-pointer information is 64 or 32 bit width dependant, so its a matter of tracking those peskies down and clamping them to a fixed width. To get around that for now, I indexed the 10,000 odd audio files over NFS from an n810.
Some information for those looking to play, if you are using boost to memory map, you may well discover that atomic_cas32() calls __sync_val_compare_and_swap() which is not actually available on the maemo device (for me). This is with a modified scratchbox using gcc version 4.2.1.
I'm not sure if there is an accepted cas et al library for maemo/armel. I leave the hacks for __sync_val_compare_and_swap() to the reader and their accepted level of concurrency control risk.
Thursday, January 29, 2009
Searching for audio files on an NFS share from maemo
Searching for audio files on an NFS share from maemo from Ben Martin on Vimeo.
The n810 only has one memory slot. With an 8gb card in there you might fit 1,000 ogg files onto your storage. That was quite boaring, so I instead indexed an NFS share that is over 10x larger ;) I'm using one of the libferris inverted file backends for the index, which is more targeted at the desktop machine assumptions of having a faster CPU and expensive disk head seeks. Needless to say, I'm hacking on a custom index implementation for maemo which will be more oriented at a slower CPU with much much less expensive disk seeks for flash based storage.
Note that the time after I run ferris-music-search to when the first message appears on the console "Using index..." is fairly much all wasted in dynamic linking. A major slowdown that I've yet to sweep away for running apps on maemo.
The artist and title info is taken from the ID3 tags in the audio files. Indexing time is roughly 3 to 10 milliseconds per file when performed on the desktop. The inverted file index format is portable from desktop to maemo device. I plan to make the new explicit maemo format portable too, so you can make indexes on powerful machines and rsync them over to the maemo device. Assuming you are indexing stuff that is stored on your file server, not the maemo device.
During the typing for the first search on title, possible completions are shown by taking your input as a substring of the title you seek. This is more effective if you keep it in mind because you can choose just a few keys in a substring of the title. All searches are performed using regex matching on strings, which is much slower than direct equality because of the huge complications it introduces for indexing. But it is interesting, even with modestly 10x the number of files you can cram onto an n810, using nasty slow regex searching, the performance is acceptable for much of the time. There are a few cases that I'll improve, particularly regex searching on whole URLs.
Notice that the name and URL are shown as columns, so you can easily "group by" when you click on the appropriate header. I need to also include the artist, title etc ID3 fields into the results,. Oh, and have the ability to click on a few files and see the whole ID3 and metadata of those files "side-by-side" so you do not have to try to read it from the results list.
So now when I see a CD in the shops, I wont have to wonder how many of those Mozart tracks I have already, I can know for sure :-p
Monday, January 19, 2009
Accelerometers hit Canola2 and PDF viewing
Of course, if a unit comes out with an embedded accelerometer in it then using a "slightly clunky" wiimote will no longer be needed as the source of motion data. I recently added the ability for libsixdof to control closed source applications as well as properly patched open source ones. This was mainly to let Google Earth finally take some advantage of proper sixdof devices, but it works equally well for Canola2 on maemo.
Controlling Canola2 using libsixdof and a wiimote from Ben Martin on Vimeo.
Two big features that libsixdof brings is the ability to configure what happens when the controller is moved on an axis, and how frequently that happens. So, if you don't like your images flicking past at 20 images / second in Canola, you can limit it to whatever / second. In the video, having maximum axis movement will only ever scroll 3 images / second. So when I hold the unit causing a continual next image movement I can still see what is happening and Canola2 does not get overloaded with events.
Flipping pages in a PDF by moving the n810 from Ben Martin on Vimeo.
Again the wiimote is actaully held behind the device. I'm sort of getting the hang of the controlling code etc so that you can use more subtle rotations of the device to issue commands to the running program. This is using the same stuff that the Canola2 video did, I could patch a PDF viewer, but for discrete events like next-page it doesn't really make a huge difference. Panning is the big thing that screams out for patching.
Controlling Canola2 using libsixdof and a wiimote from Ben Martin on Vimeo.
Two big features that libsixdof brings is the ability to configure what happens when the controller is moved on an axis, and how frequently that happens. So, if you don't like your images flicking past at 20 images / second in Canola, you can limit it to whatever / second. In the video, having maximum axis movement will only ever scroll 3 images / second. So when I hold the unit causing a continual next image movement I can still see what is happening and Canola2 does not get overloaded with events.
Flipping pages in a PDF by moving the n810 from Ben Martin on Vimeo.
Again the wiimote is actaully held behind the device. I'm sort of getting the hang of the controlling code etc so that you can use more subtle rotations of the device to issue commands to the running program. This is using the same stuff that the Canola2 video did, I could patch a PDF viewer, but for discrete events like next-page it doesn't really make a huge difference. Panning is the big thing that screams out for patching.
Sunday, January 18, 2009
Like a walk in the park..
A walk through the botanical gardens the other day with a digital camera and I had a nice little harvest of background images. Unfortunately the initial scale and crop of the scaled creature to 800x480 means his head is hidden somewhat under the status icons. But many of the other more generic ones work well for backgrounds. Plus I get the added bonus of having a collection of prescaled 800x480 images for demoing Canola hacks.
Wednesday, January 14, 2009
Didn't your n810 come with an accelerometer?
Shown below is a video of the forthcoming 0.1.0 release of libsixdof with a wiimote. I have hacked up maemo-mapper to have initial support for libsixdof, I'll have to clean up the patch before letting other human eyes see it's nastyness :| I decided to use the C++ sixdof API which means there is a single C++ source file in my build of maemo mapper, it was just faster to get it coded that way. Perhaps I'll convert to the libsixdofc ANSI C API before sending the patch.
Didn't your n810 come with an accelerometer? from Ben Martin on Vimeo.
Although libsixdof supports other 6-degrees of freedom controllers too, getting those controllers to work on maemo is a little more of a challenge. For example, I would love to use the portable spacenavigator usb device with maemo. Even though most folks have the USB cabling to hook up the unit, and it gets detected by the n810 when you have USB in the right mode, getting XInput events from evdev on maemo might be another battle. It is not like that part is trivial on a desktop distro either right now unfortunately. I might end up hacking support for direct /dev/input/foo kernel devices into libsixdof to get around that issue and have "real" 6dof devices working under maemo.
Didn't your n810 come with an accelerometer? from Ben Martin on Vimeo.
Although libsixdof supports other 6-degrees of freedom controllers too, getting those controllers to work on maemo is a little more of a challenge. For example, I would love to use the portable spacenavigator usb device with maemo. Even though most folks have the USB cabling to hook up the unit, and it gets detected by the n810 when you have USB in the right mode, getting XInput events from evdev on maemo might be another battle. It is not like that part is trivial on a desktop distro either right now unfortunately. I might end up hacking support for direct /dev/input/foo kernel devices into libsixdof to get around that issue and have "real" 6dof devices working under maemo.
Hello, planet maemo
OK, so now I have relevant posts aggregated onto planet maemo, hi everybody </dr-nick>.
For those who haven't discovered it, I run a personal repository of packages for maemo. If you break your maemo device by using my repository, you get to keep both pieces. I have a few n810 units and of course my primary open source software suite, libferris, is ported to maemo and available in my repository. Libferris is a virtual filesystem which includes rich index and search capabilities. There are many nice choices for index+search for maemo, with Tracker coming out soon in the standard maemo distribution, strigi available for the n810 etc. Of course, I use libferris on maemo for my indexing ;)
My repo also contains a few handy misc packages like unison, sshfs, fuse, and tinc. Anyone notice the hints that I like filesystems? The unison package could be made to have a few less dependencies, but it works well already. I have compiled afuse, obexftp and obexfs for maemo but those are not in my repository yet because I'm having trouble getting the 810 bluetooth to work from the normal obex packages instead of through the osso-gwobex layer. It would be nice to be able to mount a mobile phone's sd card through FUSE on the n810 and share storage a bit more, but I digress.
You'll also notice cwiid and libsixdof added recently to the repository. I have been playing with a wiimote on the desktop and on the n810. The libsixdof on the 810 can read the state of the wiimote OK. The wmgui tool locks the CPU of the n810 at 100% when using the accelerometer mode, so clearly you have to be somewhat smarter about what you update for each event on an embedded device. I have hopes that I can hack maemo mapper to use libsixdof soon and thus be able to control it on an n810 using a wiimote ;) Getting other six degree of freedom controllers to work will be more of a problem on maemo because you need to use a very recent evdev with a patch for some devices.
One of the issues that still plagues libferris on maemo is the lack of prelinking. Starting an application that uses libferris causes the CPU to spend a bunch of time resolving symbols before it can start executing anything. I've enabled hidden symbols, compiled with a later gcc than the scratchbox normally uses and played games to speed things up, but getting close how quick a desktop machine can resolve symbols at application startup has still escaped me. I might end up rsyncing / to nfs:/foo and running prelink on an n810 just to see if/how much that helps things out.
For those who haven't discovered it, I run a personal repository of packages for maemo. If you break your maemo device by using my repository, you get to keep both pieces. I have a few n810 units and of course my primary open source software suite, libferris, is ported to maemo and available in my repository. Libferris is a virtual filesystem which includes rich index and search capabilities. There are many nice choices for index+search for maemo, with Tracker coming out soon in the standard maemo distribution, strigi available for the n810 etc. Of course, I use libferris on maemo for my indexing ;)
My repo also contains a few handy misc packages like unison, sshfs, fuse, and tinc. Anyone notice the hints that I like filesystems? The unison package could be made to have a few less dependencies, but it works well already. I have compiled afuse, obexftp and obexfs for maemo but those are not in my repository yet because I'm having trouble getting the 810 bluetooth to work from the normal obex packages instead of through the osso-gwobex layer. It would be nice to be able to mount a mobile phone's sd card through FUSE on the n810 and share storage a bit more, but I digress.
You'll also notice cwiid and libsixdof added recently to the repository. I have been playing with a wiimote on the desktop and on the n810. The libsixdof on the 810 can read the state of the wiimote OK. The wmgui tool locks the CPU of the n810 at 100% when using the accelerometer mode, so clearly you have to be somewhat smarter about what you update for each event on an embedded device. I have hopes that I can hack maemo mapper to use libsixdof soon and thus be able to control it on an n810 using a wiimote ;) Getting other six degree of freedom controllers to work will be more of a problem on maemo because you need to use a very recent evdev with a patch for some devices.
One of the issues that still plagues libferris on maemo is the lack of prelinking. Starting an application that uses libferris causes the CPU to spend a bunch of time resolving symbols before it can start executing anything. I've enabled hidden symbols, compiled with a later gcc than the scratchbox normally uses and played games to speed things up, but getting close how quick a desktop machine can resolve symbols at application startup has still escaped me. I might end up rsyncing / to nfs:/foo and running prelink on an n810 just to see if/how much that helps things out.
Tuesday, January 13, 2009
Combine: wiimote, Google Earth, libsixdof and Linux
Oh, and a nice seat, and you get another HID to use for moving around. This is using the upcoming 0.1.0 release of libsixdof which has wiimote support. Note that you can also use the spacenavigator true 6dof device. If you are using the later then you'll need evdev 2.1.0 or later, and if you're using 2.1.0 see this bug for a patch to get more than X and Y axis working.
More information to follow, including a video and details on the space navigator and GE.
libsixdof and a wiimote to control Google Earth under Linux from Ben Martin on Vimeo.
More information to follow, including a video and details on the space navigator and GE.
libsixdof and a wiimote to control Google Earth under Linux from Ben Martin on Vimeo.
Saturday, January 10, 2009
Chinook vs Diablo
While most folks have upgraded to Diablo long, long ago, I fell into the trap of actually using the n810 early on. So I had a nice chinook setup and didn't really want to do the Wintel mantra of reinstalling just to get diablo. But now I've been playing with both, with two n810 devices, because its always more fun with two ;)
So one is running the Chinook from about Feb 2008 and the other RX-44_DIABLO_5.2008.43-7. I thought that wifi felt faster on Diablo, so I did a little measuring. Its quite slap happy but does show some considerable difference, using the same DLink DIR-855 AP for both devices.
On the server:
dd if=/dev/zero of=10mb bs=1024 count=10240
On the tablet:
time wget -Y off http://x/10mb
The results:
chinook:
real 0m29.301s
user 0m1.047s
sys 0m1.828s
350k/s
diablo:
real 0m24.815s
user 0m0.656s
sys 0m1.188s
546k/s
The chinook k/s is quite stable at around 350. On diablo things move around quite a bit in terms of speed. Perhaps this bursty nature works well for browsing making things 'feel' faster too.
So one is running the Chinook from about Feb 2008 and the other RX-44_DIABLO_5.2008.43-7. I thought that wifi felt faster on Diablo, so I did a little measuring. Its quite slap happy but does show some considerable difference, using the same DLink DIR-855 AP for both devices.
On the server:
dd if=/dev/zero of=10mb bs=1024 count=10240
On the tablet:
time wget -Y off http://x/10mb
The results:
chinook:
real 0m29.301s
user 0m1.047s
sys 0m1.828s
350k/s
diablo:
real 0m24.815s
user 0m0.656s
sys 0m1.188s
546k/s
The chinook k/s is quite stable at around 350. On diablo things move around quite a bit in terms of speed. Perhaps this bursty nature works well for browsing making things 'feel' faster too.
Subscribe to:
Posts (Atom)