N9: Two socks VPN solution

I always tend to lean more towards SSH for VPN stuff than other tools like openvpn. With this mindset I was working out how one might get access to servers on the Intranet while on the move. Having created a bunch of nice little apps that do things on my network is all well and good, but I wanted to also use them while on the move.

My current solution is to use two socks servers on the phone. It might seem strange to need two, but this allows routing depending on source address and destination address. The first SOCKs server is dante which routes all "normal" traffic directly to the chosen Internet connection on the phone. Traffic which wants to go to selected addresses on my Intranet is instead forwarded to a second socks proxy on the phone. This second socks proxy is implemented with ssh "DynamicForward" to send traffic over the SSH link to the Intranet.



The glory here is that Internet traffic goes directly, and Intranet traffic is sent over an encrypted pipe to the Intranet. If on the other hand I connect to any router on the Intranet, the phone gets a local IP address and dante is instructed to "route" traffic from that local address directly to the target instead of to the second local ssh socks server. Thus I don't have to concern myself with the details of which side of firewalls I am on, the phone automatically does that for me and chooses the most efficient route for packets.

Having a phone that runs so much like a proper Linux distro makes this sort of thing fairly easy to get going. No need for funky NDKs, NDAs, or TLAs.

If you want to play around with this stuff I have dante debs in my repo:
http://fuuko.libferris.com/maemo/n9/
Using non privlidged ports on the phone means you can run danted as a normal user. I have an icon on the n9 home screen that makes sure dante is running instead of using init.d at the moment. Also, an icon that runs a bash encapsulated with tsocks can be handy for tinkering:
Desktop file segment:

Exec=meego-terminal -n -e /bin/sh /home/user/bin/create-tsocks-env
...

$ cat /home/user/bin/create-tsocks-env
export http_proxy=http://1.2.3.4:1313/
tsocks bash

Using a similar style Exec in a desktop file one can start the SSH client in "-N" mode and use the passphrase to connect and setup an SSH connection with a socks server on the phone. It is quick and dirty without shiny QML interfaces, but it works^TM.

Zoneminder on the n9. Maybe n900 too?

My growing collection of Qt/QML applications has now hit 6 apps with the inclusion of a Zoneminder client for the n9. As I still have ideas for more apps I wonder how long until I fill half a screen with my own apps on the device ;)



Anyways, the Zoneminder app uses server side scaling and jpeg generation to keep bandwidth down and CPU/RAM very low on the device. I have modified some of the images for example purposes. The app is basically as shown but will display your real camera data instead of Koalas... Unless you are lucky enough to have some furry little friends. The first camera starts a live view right off the bat when the app starts, so there is as little finger use needed to get some useful action from running the app.

Only active cameras are listed. The play button changes the image on the right to be a 1 second rolling "live view" of that camera. The little watch moves to the recorded events for that camera which is shown below. The config is quite simple; the Zoneminder server URL, and the user/password to login to that server.



Alarm events are shown on the left side and the play button rolls through that event with the progress bar showing how far through the alarm you are. Naturally both the lists scroll to show more cameras or events if there are more than 5. By default only 25 of the most recent alarm events are shown in the list. A future expansion might allow paging back to see older ones.

Ferris on the n9: Search by URL, content, and mtime

The n9 libferris app now allows you to search by the URL of the file as before, but also now by the file text content and it's modification time. Query type is selected by a button in the top right corner which unfortunately isn't nearly as easy to read in the video file as it is in real life.

As libferris handles extraction, update, and storage of metadata from disparate locations I have also added a sprinkling of what that means into the video. Notice that the first search by URL shows a comment "REST interface to libferris" in bold. This is simply the "annotation" metadata of the file but is much more interesting to the searcher than it's URL. Likewise in the second query, which finds a Gutenburg text file by searhing on text content, the annotation offers the name of the book that the file contains. Again much more interesting content to the human who is at the helm.

The third query is on the modification time of the files. There are three ways offered to perform a time search, "more recently than this" or "than last" which can have month or day as options for example, and modified >= X months ago which obviously wants a number as the query text. When querying by time like this, libferris happily accepts some human readable terms like "begin this month" as the time you specify. This makes it just as convenient to use search in scripts as in the dedicated front ends like the n9 app.



The plan is as always, to push metadata from being an after thought to being a first class citizen: able to be created, read, written, indexed, and searched on. Any file with metadata should be able to expose that as simply as its mtime or size, which is all currently done via a key-value "Extended Attribute" interface at the lowest level.

If you like libferris or the recent updates (status.net mounting, these indexing tweaks etc) then please consider making a donation. If you want to use this technology at a corporate level, please feel free to contact me.

libferris in 512Mb RAM on arm5 at 1.2Ghz

I mentioned yesterday that I had started hacking my infix indexing optimization into the clucene index module of libferris. The short story is that on an aged ARM machine with fairly slow IO this optimization makes a huge difference to regex query on URLs. The numbers are below, this is on an index of ~/, /etc and /usr on the ARM5 machine running debian, about 100,000 odd files. Cold1 and hot1 are the same query executed against cold and hot caches (second run done just after the first). Naturally Cold2 and Hot2 are the same for a different query. Cold1 returns 22 results and Cold2 returns a single result.

       old     new
cold1  4.9     3.6
hot1   2.0     1.0
cold2  2.6     1.9
hot2   1.7     1.0

As can be seen, the optimization effects both hot and cold cache times which is quite handy as there are many times I use both, starting with a "-Z" evalutation to see how many matches there are and specializing or generalizing the query from there.

While the numbers may seem large, keep in mind that this is a slow arm running from an IO interface that needs some TLC. Using this indexing on an N9 or desktop machine will be much quicker even if there are 10 times the number of files indexed.

These arm deb builds will be up on fuuko.libferris.com sometime soon. I have to do a release of libferris with this optimization and the funky new JSON/REST support too.

Libferris on the N9: JSON, REST, QML, Index and Search, VFS all together

Continuing my n9 apps I now have an index search app that uses libferris. Currently I have only exposed URL regular expression search. This is searching an index maintained and stored in a PostgreSQL database. Solving regex query in a timely manner is a fairly complex problem with one useful solution outlined in one of my previous blog entries.

Note that data is downloaded over normal http(s) from the server using the n9 app, so no NFS or other network mounts are needed. This might be handy for grabbing a PDF, image or text file off an Intranet file server while on the move.



I have also been updating the clucene libferris module to allow more effective index use during regex/infix queries. This libferris engine works quite nicely on older ARM machines. Benchmarks on that to follow, once I reindex the arm and produce the comparative figures.

Freetronics MobSenDat: The rocketses tail.

I was fortunate enough to get my hands on a MobSenDat board which was on offer at the Arduino mini conf at LCA 2011. Being a software guy, soldering hardware is a traditional weak point for me, so I decided to construct a $30 Freetronics KitTen board first. Seemed like a good idea since these boards are cheaper and much more readily available in case of a dead starter construction.

Like many, I was overjoyed to see my flashing LED between the GRD and PIN 13 on the KitTen! So with the little courage boost I started on the MobSenDat board. After a few sessions slowly moving along there, and a little defluxing at the end, I got to a stage that I thought the board was "OK".

As you can see, I've not got the ZigBee love happening yet. But put the NTX2 antenna point on the board just for fun ;) It seems the go is to get a zigbee to usb for lappy or tablet usage and two zigbee pro chips to get a radio link up and running: http://australianrobotics.com.au/products/xbee-explorer-usb



Having tinkered in the arduino IDE for the kitten, I wanted to get the software up and ready in some state to upload to the mobsendat. The repo I grabbed from after a bit of sifting was this one: https://github.com/lukeweston/RocketInstrumentation
Another repo tucked away for later is this one: https://github.com/geekscape/aiko_mobsendat/

So with library setups I got the IDE to compile the code for the Pro8mhz/ATmega328P chip. After a little ginning around I uploaded that over the cable to the mobsendat. Slightly different to the kitten as there are no resets on the mobsen and it boots right off an external 9v battery.

Opening the serial console gave me the first worrying moment. Some stuff appeared on a regular basis but it was non ascii garbage. Changing the baud rate to 38k and some more expected text appeared! Times, temps, pressures, x,y,z readings...

A huge thank you to Luke Weston and the Freetronics guys for making these things available, and robust enough that even somebody with limited soldering skill can construct them.



In the process I also sniffed out the mobsendat2 board for those who are interested in these sorts of things:
https://github.com/lukeweston/MobSenDat2/blob/master/MobSenDat2-pcb.png

I might end up using the kitten as a parachute deployment board or for other flight instrumentation. My next build will be larger and probably based on a 54mm motor mount tube. Now that I have working electronics I want an electronics bay in the build...

QML BOM: Weather on the go...

As part of my foray into n9 apps, a new one that displays information from the Australian Bureau of Meteorology on the n9. Being pure QML + Javascript it also runs on the desktop. The forecasts include min and max temps, and for the Europeans reading this, the middle line "10..." is the UV prediction and roasting window for the day... all your sunscreen are belong to us.



Apart from saving a little on downloads for the topology and background images when on the move, you get to customize those as they are just png files living in the images subdirectory. So I have the local rocket launch site indicated. I'd like to make a LOTR style topology overlay, perhaps mt Glorious becomes Mordor?

Code available at https://github.com/monkeyiq/qmlbom

N9, QML, Javascript and differential update.

I had a server app that had both a desktop and web interface. Though I discovered that the web interface was less than monkey finger friendly on the n9 and required some authentication each time. I thought this was a job for QML and javascript on the device. The good news is that I now have a client using just those tools, so no cross compilation needed at all ;)

The core of the UI is updated via a timer which runs an ajax call, parses the JSON response and gives that to updateModel(). The first implementation of the later method was to clear the listModel and repopulate from scratch each time. Rather wasteful, even though it was only being called once every 2-3 seconds, it was eating up 8-9% CPU of the n9 in top. A better solution is to iterate the existing items, performing update, remove, or add as you go. This needs a "key" field for each tuple and a stable total order on the model JSON data, either from the server or through an explicit client side sort.

I'll probably put this up on github soon, but in the meantime, this is the updateModel() which has the smarts I mentioned above. It takes 3-4% of CPU instead, and should maintain view state better on the device. I use objid as the key, and the "state" is cloned to objstate because I noticed some issues there with QML interaction using state as the property. Interesting what one can do with QML+JS in very short time for the n9.

    function updateModel( model, data ) 
    {
 var existingIndex = 0;
 var existingIndexMax = model.count;
 var id;

 for( id in data ) 
 {
     var found = false;
     var name  = data[ id ];
     var col   = data[ id ];
     var k;
     var row = {};
     row[ "objid" ] = id;
     row[ "name" ] = "v";
     for( k in col ) 
     {
  if( k == "state" ) {
      row[ "objstate" ] = col[k];
  }
  else {
      row[ k ] = col[ k ];
  }
     }

      // find it if it exists...
     for( ; existingIndex < existingIndexMax; ++existingIndex )
     {
  if( model.get( existingIndex ).objid == id )
  {
      found = true;
       model.set( existingIndex, row );
      ++existingIndex;
      break;
  }
  else
  {
      // This element from the local model no longer exists on the server,
      // remove it. Do not advance the index as we have removed an element instead.
      model.remove( existingIndex );
      --existingIndex;
  }
     }

     // it didn't exist, add it as new.
     if( !found )
     {
  model.append( row );
     }
 }
    }

Libferris on both arms

I now have libferris on a 512mb ARM5t embedded device and a 1gb arm7 one (the Nokia n9). As part of the fun and games I updated the clucene and boostmmap index modules for libferris and created a QtSQL module which uses SQLite by default. Hey, when you can't choose which cake to have why not have a slice of them all... or a slice to fit the moment or device at hand. eg, desktop or server works nicely with the soprano or postgresql indexing modules, embedded is nicer on mmap or clucene.

I find it ironic that I never really thought much of "embedded" devices when hacking libferris. But devices with 512/1024mb of RAM are really not so much embedded I guess.

A few things I've tried to do in the design of libferris index+search is to design for many machines and also federations of them. It is possible to search a router, phone, and desktop's individual indexes as a federation from the laptop. Another thing that helps is that indexing is routed through the "findexadd" commands. So you can use find and split to break up indexing activity, and have it done from cron when you want.
The new --total-files-to-index-per-run option works in combination, causing findexadd to exit when it has indexed a given number of files. Note that if a file has not changed since it was last indexed it is not indexed again (no need), so that file does not count toward the
total-files-to-index-per-run tally.

The below is a little script to incrementally index just selected metadata from /usr and your home directory using clucene. The WHITELIST environment variable stops libferris from trying to sniff up metadata for files and has it only look for and add what metadata you want. If you have md5 in there then libferris will store the checksum for each file, at a commensurate cost in IO. Splitting into batches of 5000 prevents the process running too long and wanting too much RAM.

$ mkdir -p ~/.ferris/ea-index
$ cd ~/.ferris/ea-index
$ fcreate --create-type eaindexclucene .
$ vi update.sh
#!/bin/bash

TMPDIR=~/tmp
EAIDXPATH=~/.ferris/ea-index

cd $TMPDIR
find /usr | split -l 5000 - usr.split.
find ~ | split -l 5000 - home.split.

export LIBFERRIS_EAINDEX_EXPLICIT_WHITELIST=name,size,mtime,mtime-display,atime,ctime,user-owner-name,group-owner-name,user-owner-number,group-owner-number,inode
echo "whitelist: $LIBFERRIS_EAINDEX_EXPLICIT_WHITELIST"
cd $EAIDXPATH
rm -f write.lock
for if in $TMPDIR/*split.*
do
echo "Processing $if"
cat $if | feaindexadd -P `pwd` -1
done



Ironically the arm5 has given me much less trouble overall. One issue seems to be with gcc-4.4x on the n9. Charming little errors like my old friend the undefined __sync_val_compare_and_swap_4 which stops memory mapped boost data structures from working properly and also leaves the clucene-core-2.3.3.4 build laying on the side of the road bleeding. I've hacked the clucene code to get around the atomic errors, but then seem to have found that search results are not accurate. I guess my quick hack there was just bad^TM. Especially since the arm5 produces the right results using the virgin clucene codebase.

I've been trying to convince gcc 4.5 and 4.6 to build for me so I can use the updated compiler to generate a proper and working clucene for the device. I seem to run into little build issues after time consuming rebuilds. (uses VFP register arguments, yay). Once I stop compiling compilers then maybe I can get my favourite indexing code on the n9.