QML BOM: Weather on the go...

As part of my foray into n9 apps, a new one that displays information from the Australian Bureau of Meteorology on the n9. Being pure QML + Javascript it also runs on the desktop. The forecasts include min and max temps, and for the Europeans reading this, the middle line "10..." is the UV prediction and roasting window for the day... all your sunscreen are belong to us.



Apart from saving a little on downloads for the topology and background images when on the move, you get to customize those as they are just png files living in the images subdirectory. So I have the local rocket launch site indicated. I'd like to make a LOTR style topology overlay, perhaps mt Glorious becomes Mordor?

Code available at https://github.com/monkeyiq/qmlbom

N9, QML, Javascript and differential update.

I had a server app that had both a desktop and web interface. Though I discovered that the web interface was less than monkey finger friendly on the n9 and required some authentication each time. I thought this was a job for QML and javascript on the device. The good news is that I now have a client using just those tools, so no cross compilation needed at all ;)

The core of the UI is updated via a timer which runs an ajax call, parses the JSON response and gives that to updateModel(). The first implementation of the later method was to clear the listModel and repopulate from scratch each time. Rather wasteful, even though it was only being called once every 2-3 seconds, it was eating up 8-9% CPU of the n9 in top. A better solution is to iterate the existing items, performing update, remove, or add as you go. This needs a "key" field for each tuple and a stable total order on the model JSON data, either from the server or through an explicit client side sort.

I'll probably put this up on github soon, but in the meantime, this is the updateModel() which has the smarts I mentioned above. It takes 3-4% of CPU instead, and should maintain view state better on the device. I use objid as the key, and the "state" is cloned to objstate because I noticed some issues there with QML interaction using state as the property. Interesting what one can do with QML+JS in very short time for the n9.

    function updateModel( model, data ) 
    {
 var existingIndex = 0;
 var existingIndexMax = model.count;
 var id;

 for( id in data ) 
 {
     var found = false;
     var name  = data[ id ];
     var col   = data[ id ];
     var k;
     var row = {};
     row[ "objid" ] = id;
     row[ "name" ] = "v";
     for( k in col ) 
     {
  if( k == "state" ) {
      row[ "objstate" ] = col[k];
  }
  else {
      row[ k ] = col[ k ];
  }
     }

      // find it if it exists...
     for( ; existingIndex < existingIndexMax; ++existingIndex )
     {
  if( model.get( existingIndex ).objid == id )
  {
      found = true;
       model.set( existingIndex, row );
      ++existingIndex;
      break;
  }
  else
  {
      // This element from the local model no longer exists on the server,
      // remove it. Do not advance the index as we have removed an element instead.
      model.remove( existingIndex );
      --existingIndex;
  }
     }

     // it didn't exist, add it as new.
     if( !found )
     {
  model.append( row );
     }
 }
    }

Libferris on both arms

I now have libferris on a 512mb ARM5t embedded device and a 1gb arm7 one (the Nokia n9). As part of the fun and games I updated the clucene and boostmmap index modules for libferris and created a QtSQL module which uses SQLite by default. Hey, when you can't choose which cake to have why not have a slice of them all... or a slice to fit the moment or device at hand. eg, desktop or server works nicely with the soprano or postgresql indexing modules, embedded is nicer on mmap or clucene.

I find it ironic that I never really thought much of "embedded" devices when hacking libferris. But devices with 512/1024mb of RAM are really not so much embedded I guess.

A few things I've tried to do in the design of libferris index+search is to design for many machines and also federations of them. It is possible to search a router, phone, and desktop's individual indexes as a federation from the laptop. Another thing that helps is that indexing is routed through the "findexadd" commands. So you can use find and split to break up indexing activity, and have it done from cron when you want.
The new --total-files-to-index-per-run option works in combination, causing findexadd to exit when it has indexed a given number of files. Note that if a file has not changed since it was last indexed it is not indexed again (no need), so that file does not count toward the
total-files-to-index-per-run tally.

The below is a little script to incrementally index just selected metadata from /usr and your home directory using clucene. The WHITELIST environment variable stops libferris from trying to sniff up metadata for files and has it only look for and add what metadata you want. If you have md5 in there then libferris will store the checksum for each file, at a commensurate cost in IO. Splitting into batches of 5000 prevents the process running too long and wanting too much RAM.

$ mkdir -p ~/.ferris/ea-index
$ cd ~/.ferris/ea-index
$ fcreate --create-type eaindexclucene .
$ vi update.sh
#!/bin/bash

TMPDIR=~/tmp
EAIDXPATH=~/.ferris/ea-index

cd $TMPDIR
find /usr | split -l 5000 - usr.split.
find ~ | split -l 5000 - home.split.

export LIBFERRIS_EAINDEX_EXPLICIT_WHITELIST=name,size,mtime,mtime-display,atime,ctime,user-owner-name,group-owner-name,user-owner-number,group-owner-number,inode
echo "whitelist: $LIBFERRIS_EAINDEX_EXPLICIT_WHITELIST"
cd $EAIDXPATH
rm -f write.lock
for if in $TMPDIR/*split.*
do
echo "Processing $if"
cat $if | feaindexadd -P `pwd` -1
done



Ironically the arm5 has given me much less trouble overall. One issue seems to be with gcc-4.4x on the n9. Charming little errors like my old friend the undefined __sync_val_compare_and_swap_4 which stops memory mapped boost data structures from working properly and also leaves the clucene-core-2.3.3.4 build laying on the side of the road bleeding. I've hacked the clucene code to get around the atomic errors, but then seem to have found that search results are not accurate. I guess my quick hack there was just bad^TM. Especially since the arm5 produces the right results using the virgin clucene codebase.

I've been trying to convince gcc 4.5 and 4.6 to build for me so I can use the updated compiler to generate a proper and working clucene for the device. I seem to run into little build issues after time consuming rebuilds. (uses VFP register arguments, yay). Once I stop compiling compilers then maybe I can get my favourite indexing code on the n9.

The Nokia N9: Going for a spin

In short, I now have a Nokia n9, and setup a scratchbox for it, and compiled & installed abiword, clawmotia, unison, and libferris for and on the device. If that didn't enlarge your page down key, the longer version now follows ;-)

FWIW my impression of the device is very positive. Hardware wise the biggest issue I have is the inability to add more storage to it. A high gb microsd slotted in would make things a whole bunch nicer, mainly for using the device as a music box on the road. The speed of interaction is very nice, and the swipe interface is very easy to get used to. Just flick an app away to minimize it or flick from another screen edge to close it.

gstreamer works quite nicely for viewing network streams, though there is a deadzone bar on the logical "bottom" of the screen (left of screen in portrait mode). I noticed this bar in abiword too, so I assume it is an issue with going fullscreen for non Qt native apps.

There isn't much to show for unison, as I tend to drive that from the dekstop/laptop end of the connection. Getting it on the device is always a matter for first compiling ocaml and installing that into the scratchbox.

One very pleasant surprise is how snappy clawmotia runs on the n9. Startup time is significantly less than on the n900 and the screen operations like swiping the main controls left to get to the jog shuttle page is now at an acceptably smooth speed. I'll have to work up a video of this in action to show the speed of things...



I got libferris compiled for the n9 too. Using boost::spirit for some of the parsers has dropped one compile time dependency away. I did make both ferris and ferriscreate able to compile in glib2 only mode. You loose one or two apps by doing this, but it also makes the footprint smaller so that I can use it with QML apps without needing to pull in gtk2 from a third party repo just for those apps in ferris itself. Mainly some capplets, gfcreate, and the gfcp and other graphical coreutils reimplementations are lost in building this way.

And what is not complete without a screenshot of things on the device? Sorry for the quality of the picture, I need to work out lighting for some device shots a bit better than this adhoc quick shot. Just to tap into the ferris power a tiny bit I decided to mount an XML file as a filesystem and fcat directly into it.. A few of the fun things I plan are a QML client for index/search using the customized maemo index backend in libferris and using the mounted pulseaudio and mounted gstreamer to implement yet another audio player for the device. The index+search in both these apps should be blisteringly fast, given that I could index an NFS mount with 10x the data I could hold on an n810 and search as you type in real time...



Abiword proved to be the harder thing to compile for the n9. It seems GTK2/3 has fallen off the device so I got gtk2 from a third party repository. The downside is that the app doesn't run entirely fullscreen and has pretty much no finger friendlyness from gtk2 itself. Oh yeah, the virtual keyboard doesn't come up on text entries in gtk2 either. I might be compiling and installing gtk2 incorrectly on the device, so this complaint about device interaction might be my own fault?

I then got libwv and libwmf, following the dependency rabbit down the white hole. Of course my abiword compile has full ODF support in it too! The below screenshot is of one of my sample ODF+RDF documents with the content that links to RDF shown in purple. For more info on RDF in abiword see the abiword talk video from lca 2012.



More fun an games to follow, and I'll post a link to a repo with the debs once I work out where I'm going to host the 100mb odd of files. Not all those are for the device, some are mainly for scratchbox such as the dev packages and full ocaml compiler debs.

Mounting statusnet/identica as a filesystem

With the 1.5.12 release of libferris hot off the presses, I thought I'd blog about how to use the new statusnet mounting. I tried to do some testing against identica at the start, and managed to get myself silenced there ;) I was resurrected within hours once I put forward my case as not-a-trollbot^tm. Anyway, that led me to setup my own local statusnet server for testing against, and now libferris can mount identi.ca and also one or more local servers if you so desire.... yay!

There is a little bit of setup required, web keys, secrets and oauth... the normal story.
The identica filesystem contains a directory for each server you have setup in ~/.ferris/identica. To update my public status I just write to the status file in my identica account:

$ echo "this is coming from the console via libferris, mounting the public statusnet ;-p" \
| ferris-redirect identica://identica/status

To see what has been going on I can list the timeline directory, using "-0" to get libferris to tell me what metadata is interesting for that directory instead of viewing the protection bits and other non interesting information.

$ fls -0 identica://identica/timeline
89868729 Welcome to Identi.ca, @monkeyiq! Welcome Bot 12 Feb 2 03:53
...

There is also support for direct messaging through the filesystem, seeing retweets, friends, and mentions.

Now for the details... Configuration happens through the virtual filesystem. Each statusnet server you want to mount has it's own directory inside ~/.ferris/identica. The configuration directory lists the base URL for performing REST requests on that server and the prefix of where the webkey and secret are to live. For the public identi.ca server I have the following:

$ pwd
~/.ferris/identica/identica
$ cat rest_base
https://identi.ca/api/
$ cat token_prefix
identica

And the webkeys go in a file in ~/.ferris prefixed with the token_prefix above:
$ cat ~/.ferris/identica-api-key.txt
5045332...

I do not distribute web keys, so you need to go to http://identi.ca/settings/oauthapps and create one yourself.

If you have a private statusnet server foobar then you create ~/.ferris/identica/foobar with the REST url for it, and foobar as the token_prefix. Then put the web API key into ~/.ferris/foobar-api-key.txt.

To setup the oauth part, we use our old friend capplet-auth. Again for our foobar custom server replace the auth-with-site with foobar instead. This will give you a URL to visit to authorize your webapp (libferris) on your account. You'll be given a PIN to copy back onto the console and then you should be setup.

$ ferris-capplet-auth --auth-service identica --auth-with-site identica

A time for Akad-au-me?

At LCA 2012 I was talking with one or two folks and the idea of a KDE miniconference at LCA 2013 came up. As I thought it was a worthwhile idea to bring Qt/KDE folks down under I sort of threw my hat in the ring to be the "local" aussie organizer for it. I say with quotes because the event in 2013 is 1000+km away from me anyway. I'll gladly defer the glory for this idea to those who suggested it save the issues and criticisms for myself.

Anyway, I thought I'd throw it out there on the k-planet, see if I should try to get onto the KDE-ev or where/if folks on the KDE side would like to further discuss the plan. My ideas involve using the official one day mini conf for KDE talks including some lightning intros to make sure everyone can put an hydrocarbon image to the @nicks and maybe using the Saturday &&|| Sunday after the event as a hackfest or two to get some stuff done. Normally the dorms for LCA are available through the sunday arvo (not that I'm an expert), so the hackfest weekend idea has some logistics help already.

Mounting KDE pastebin

Ferris has extended its claws to allow the pastebin at http://paste.kde.org to be mounted. Since bash doesn't know libferris at all, without fuse you can use ferris-redirect to pipe information into any file that libferris can use as shown below (-T truncates like ">" in bash).

mkdir /tmp/test
cd    /tmp/test
date > df1.txt
cat df1.txt | ferris-redirect -T pastebin://kde.org/new/foo
...
 http://paste.kde.org/199754/

If you just have one file, then the "new" directory itself will act as a target too:

echo anyone, anyone... | ferris-redirect -T pastebin://kde.org/new
...
 http://paste.kde.org/199760/

And because everybody knows that key protected pastebins are the new red, the following will need to have "goodkeyhere" on hand to read the data back...

echo this is secret, dudes | ferris-redirect -T pastebin://kde.org/private/goodkeyhere
...
 http://paste.kde.org/199772/35847813/

The final directory is the "list" which contains virtual files allowing you to get a paste through the virtual filesystem. Of course, you could just fcat the http:// URL for the paste, but a nice tree is a nice tree.

fls  pastebin://kde.org/list/
... 199700
fcat pastebin://kde.org/list/199700