Ferris on the n9: Search by URL, content, and mtime

The n9 libferris app now allows you to search by the URL of the file as before, but also now by the file text content and it's modification time. Query type is selected by a button in the top right corner which unfortunately isn't nearly as easy to read in the video file as it is in real life.

As libferris handles extraction, update, and storage of metadata from disparate locations I have also added a sprinkling of what that means into the video. Notice that the first search by URL shows a comment "REST interface to libferris" in bold. This is simply the "annotation" metadata of the file but is much more interesting to the searcher than it's URL. Likewise in the second query, which finds a Gutenburg text file by searhing on text content, the annotation offers the name of the book that the file contains. Again much more interesting content to the human who is at the helm.

The third query is on the modification time of the files. There are three ways offered to perform a time search, "more recently than this" or "than last" which can have month or day as options for example, and modified >= X months ago which obviously wants a number as the query text. When querying by time like this, libferris happily accepts some human readable terms like "begin this month" as the time you specify. This makes it just as convenient to use search in scripts as in the dedicated front ends like the n9 app.



The plan is as always, to push metadata from being an after thought to being a first class citizen: able to be created, read, written, indexed, and searched on. Any file with metadata should be able to expose that as simply as its mtime or size, which is all currently done via a key-value "Extended Attribute" interface at the lowest level.

If you like libferris or the recent updates (status.net mounting, these indexing tweaks etc) then please consider making a donation. If you want to use this technology at a corporate level, please feel free to contact me.

libferris in 512Mb RAM on arm5 at 1.2Ghz

I mentioned yesterday that I had started hacking my infix indexing optimization into the clucene index module of libferris. The short story is that on an aged ARM machine with fairly slow IO this optimization makes a huge difference to regex query on URLs. The numbers are below, this is on an index of ~/, /etc and /usr on the ARM5 machine running debian, about 100,000 odd files. Cold1 and hot1 are the same query executed against cold and hot caches (second run done just after the first). Naturally Cold2 and Hot2 are the same for a different query. Cold1 returns 22 results and Cold2 returns a single result.

       old     new
cold1  4.9     3.6
hot1   2.0     1.0
cold2  2.6     1.9
hot2   1.7     1.0

As can be seen, the optimization effects both hot and cold cache times which is quite handy as there are many times I use both, starting with a "-Z" evalutation to see how many matches there are and specializing or generalizing the query from there.

While the numbers may seem large, keep in mind that this is a slow arm running from an IO interface that needs some TLC. Using this indexing on an N9 or desktop machine will be much quicker even if there are 10 times the number of files indexed.

These arm deb builds will be up on fuuko.libferris.com sometime soon. I have to do a release of libferris with this optimization and the funky new JSON/REST support too.