The vs10x3 DSP chips allow you to decode or encode various audio formats. This includes codec to mp3, ogg vorbis, as well as decoding flac and support for many other formats. So naturally I took the well trodden path and decided to use an arduino to build a "small" audio player. The breadboard version is shown below. The show is driven by the Uno on the right. The vs1063 is on a Sparkfun breakout board in the top of image. The black thing you see to the right of the vs1063 is an audio plug. The bottom IC in the line on the right is an attiny84. But wait you say, don't you already have the Uno to be the arduino in the room? But yes I say, because the attiny84 is a SPI slave device which is purely a "display driver" for the 4 bit parallel OLED screen at the bottom. Without having to play tricks overriding the reset pin, the tiny84 has just enough for SPI (4), power (2), and the OLED (7) so it's perfect for this application.
The next chip up from the attiny84 is an MCP23017 which connects to the TWI bus and provides 16 digital pins to play with. There is a SPI version of the muxer, the 23S17. The muxer works well for buttons and chip selects which are not toggled frequently. It seems either my library for the MCP is slow or using TWI for CS can slow down SPI operations where selection/deselection is in a tight loop.
Above the MCP is a 1Mbit SPI RAM chip. I'm using that as a little cache and to store the playlist for ultra quick access. There is only so much you can do with the 2kb of sram on the arduino. Above the SPI ram is a 4050 and SFE bidirectional level shifter. The three buttons in bottom left allow you to control the player reasonably effectively. Though I'm waiting for my next red box day for more controller goodies to arrive.
I've pushed some of the code to control the OLED screen from the attiny up to github, for example at:
https://github.com/monkeyiq/attiny_oled_server_spi2
I'll probably do a more just write up about those source files and the whole display driver subsystem at some stage soon.
C++, Linux, libferris and embedded development. Yet another blog from yet another NARG.
Tuesday, December 17, 2013
Sunday, December 8, 2013
Asymmetric Multiprocessing at the MCU level
I recently got a 16x2 character OLED display from sparkfun. Unfortunately that display had a parallel interface so wanted to mop up at least 7 digital pins on the controlling MCU. Being a software engineer rather than an electrical engineer the problem looked like one that needed another small microcontoller to solve it. The attiny84 is a 14 pin IC which has USI SPI and, once you trim off 4 pins for SPI, 3 for power, gnd, and reset, leaves you 7 digital pins to do your bidding with. This just so happens to be the same 7 pins one needs to drive the OLED over its 4 bit parallel interface!
Driving an OLED screen over SPI using an attiny84 as a display driver from Ben Martin on Vimeo.
Shown in the video above is the attiny84 being used as a display driver by an Arduino Uno. Sorry about having the ceiling fan on during capture. On the Uno side, I have a C++ "shim" class that has the same interface as the class used to drive the OLED locally. The main difference is you have to tell the shim class which pin to use as chip select when it wants to talk to the attiny84. On the attiny commands come in over SPI and the real library that can drive the OLED screen is used to issue the commands to the screen.
The fuss about that green LED is that when it goes out, I've cut the power to the attiny84 and the OLED. Running both the later with a reasonable amount of text on screen uses about 20mA, with the screen completely dark and the tiny asleep that drops to 6mA. That can go down to less than 2mA if I turn off the internal power in the OLED. Unfortunately I haven't worked out how to turn the power back on again other than resetting the power to the OLED completely. But being able to drop power completely means that the display is an optional extra and there is no power drain if there is nothing that I want to see.
Another way to go with this is using something like an MCP23S17 chip as a pin muxer and directly control the screen from the Uno. The two downsides to that design are the need to modify the real OLED library to use the pin muxer over SPI, and that you don't gain the use of a dedicated MCU to drive the screen. An example of the later is adding a command to scroll through 10 lines of text. The Uno could issue that command and then forget about the display completely while the attiny handles the details of scrolling and updating the OLED.
Some issues of doing this were working out how to tell the tiny to go into SPI slave mode, then getting non garbage from the SPI bus when talking to the tiny, and then working out acceptable delays for key times. When you send a byte to the tiny the ISR that accepts that byte will take "some time" to complete. Even if you are using a preallocated circular array to dispense with the new byte as quickly as possible, the increment and modulo operations take time. Time that can be noticeable when the tiny is clocked at 8mhz and the Uno at 16mhz and you ramp up the SPI clock speed without mercy.
As part of this I also made a real trivial SPI calculator for the attiny84. By trivial I mean store, add, and print are the only operations and there is only a single register for the current value. But it does show that the code that interacts with the SPI bus on the client and server side gets what one would expect to get from a sane adding machine. I'll most likely be putting this code up on github once I clean it up a little bit.
Driving an OLED screen over SPI using an attiny84 as a display driver from Ben Martin on Vimeo.
Shown in the video above is the attiny84 being used as a display driver by an Arduino Uno. Sorry about having the ceiling fan on during capture. On the Uno side, I have a C++ "shim" class that has the same interface as the class used to drive the OLED locally. The main difference is you have to tell the shim class which pin to use as chip select when it wants to talk to the attiny84. On the attiny commands come in over SPI and the real library that can drive the OLED screen is used to issue the commands to the screen.
The fuss about that green LED is that when it goes out, I've cut the power to the attiny84 and the OLED. Running both the later with a reasonable amount of text on screen uses about 20mA, with the screen completely dark and the tiny asleep that drops to 6mA. That can go down to less than 2mA if I turn off the internal power in the OLED. Unfortunately I haven't worked out how to turn the power back on again other than resetting the power to the OLED completely. But being able to drop power completely means that the display is an optional extra and there is no power drain if there is nothing that I want to see.
Another way to go with this is using something like an MCP23S17 chip as a pin muxer and directly control the screen from the Uno. The two downsides to that design are the need to modify the real OLED library to use the pin muxer over SPI, and that you don't gain the use of a dedicated MCU to drive the screen. An example of the later is adding a command to scroll through 10 lines of text. The Uno could issue that command and then forget about the display completely while the attiny handles the details of scrolling and updating the OLED.
Some issues of doing this were working out how to tell the tiny to go into SPI slave mode, then getting non garbage from the SPI bus when talking to the tiny, and then working out acceptable delays for key times. When you send a byte to the tiny the ISR that accepts that byte will take "some time" to complete. Even if you are using a preallocated circular array to dispense with the new byte as quickly as possible, the increment and modulo operations take time. Time that can be noticeable when the tiny is clocked at 8mhz and the Uno at 16mhz and you ramp up the SPI clock speed without mercy.
As part of this I also made a real trivial SPI calculator for the attiny84. By trivial I mean store, add, and print are the only operations and there is only a single register for the current value. But it does show that the code that interacts with the SPI bus on the client and server side gets what one would expect to get from a sane adding machine. I'll most likely be putting this code up on github once I clean it up a little bit.
Friday, November 15, 2013
Sparkfun vs1063 DSP breakout
The Sparkfun vs1063 breakout gives you a vs1063 chip with a little bit of supporting circuit. You have to bring your own microcontroller, sdcard or data source, and level shifting.
One thing which to my limited knowledge seems unfortunate is that the VCC on the breakout has to be 5v. There are voltage regulators on the vs1063 breakout which give it 3.3v, 2.8v and 1.8v. Since all vregs are connected to VCC and it wants to make its own 3v3 then I think you have to give 5v as VCC on the breakout board.
With the microsd needing to run on 3.3v I downshifted the outbound SPI ports, the sdcard chip select, and the few input pins to the vs1063 board. Those are the two little red boards on the breadboard. The sdcard is simply on a breakout which does no level shifting itself. The MISO pin is good to go without shifting because a 3.3v will trip as high on a 5v line. Likewise the interrupt pin DREQ which goes to pin 2 on the Uno doesn't have any upshifting.
I had a few issues getting the XDCS, XCS, and DREQ to all play well from the microcontroller. A quick and nasty hack was to attach that green LED in the middle of the photo to the interrupt line so I could see when it was tripped. During playback it gives a PWM effect as 32byte blocks of data are shuffled to the vs1063 as it keeps playing. The DREQ is fired when the vs1063 can take at least another 32 bytes of data from the SPI bus to it's internal buffer. Handy to have the arduino doing other things and also servicing that interrupt as needed.
I'm hoping to use a head to tail 3v3 design for making a mobile version of the circuit. I would have loved to use 2xAA batteries, but might have to go another way for power. Unfortunately the OLED screen I have is 5v but there are 3v3 onces of those floating around so I can get a nice modern low power display on there.
The next step is likely to prototype UI control for this. Which I'll probably use the 5v OLED in the meantime to get a feel for how things will work. I get the feeling that an attiny might sit between the main arduino and the parallel OLED screen so it can be addressed on the SPI bus too. Hmm, attiny going into major power save mode until chip selected back into life.
One thing which to my limited knowledge seems unfortunate is that the VCC on the breakout has to be 5v. There are voltage regulators on the vs1063 breakout which give it 3.3v, 2.8v and 1.8v. Since all vregs are connected to VCC and it wants to make its own 3v3 then I think you have to give 5v as VCC on the breakout board.
With the microsd needing to run on 3.3v I downshifted the outbound SPI ports, the sdcard chip select, and the few input pins to the vs1063 board. Those are the two little red boards on the breadboard. The sdcard is simply on a breakout which does no level shifting itself. The MISO pin is good to go without shifting because a 3.3v will trip as high on a 5v line. Likewise the interrupt pin DREQ which goes to pin 2 on the Uno doesn't have any upshifting.
I had a few issues getting the XDCS, XCS, and DREQ to all play well from the microcontroller. A quick and nasty hack was to attach that green LED in the middle of the photo to the interrupt line so I could see when it was tripped. During playback it gives a PWM effect as 32byte blocks of data are shuffled to the vs1063 as it keeps playing. The DREQ is fired when the vs1063 can take at least another 32 bytes of data from the SPI bus to it's internal buffer. Handy to have the arduino doing other things and also servicing that interrupt as needed.
I'm hoping to use a head to tail 3v3 design for making a mobile version of the circuit. I would have loved to use 2xAA batteries, but might have to go another way for power. Unfortunately the OLED screen I have is 5v but there are 3v3 onces of those floating around so I can get a nice modern low power display on there.
The next step is likely to prototype UI control for this. Which I'll probably use the 5v OLED in the meantime to get a feel for how things will work. I get the feeling that an attiny might sit between the main arduino and the parallel OLED screen so it can be addressed on the SPI bus too. Hmm, attiny going into major power save mode until chip selected back into life.
Saturday, November 9, 2013
RePaper 2.7 inch epaper goodness from the BeagleBone
A little while back I bought a rePaper 2.7 inch eInk display. While the smaller, down to 1.4 inch screens have few enough pixels to be driven from an Arduino, the 264x176 screen should need around 5.5k for a single frame buffer, and you need two buffers to "wax on, wax off" the image on the display in order to update. The short story is that these displays work nicely from the BeagleBone Black. You have to have a fairly recent kernel in order to get the right sys files for the driver. Hint: if you have no "duty" file for your pwm then you have too old of a kernel.
So the first image I chose to display after the epd_test was a capture of fontforge editing Cantarell Regular. Luckily, I've made no changes to the splineset so my design skills are not part of the image. The rendering of splines in the charview of fontforge uses antialiasing, as it was switched over to cairo around a year ago. As the eInk display is monochrome the image displayed is dithered back to 1 bit.
With the real time collaboration support in fontforge this does raise the new chance to see a font being rendered on eInk as you design it (or hint it). I'm not sure how many fonts are being designed with eInk as the specific consumption ground. If you are interested in font design, checkout Crafting Type which uses fontforge to create new type, and you should also be able to see the collaboration and HTML preview modes in action.
Getting the actual eInk display to go from the BeagleBone had a few steps. Firstly, I managed to completely fill up the 2gb of eMMC where my Angstrom was installed. So now I'm running the whole show off a high speed 8gb sandisk card. I spent a little extra cash on a faster card, its one of the extreme super panda + extra adjective sandisk ones. The older kernel I had didn't have a duty file for the PWM pin that the driver wanted to use. Now I that I have a fully updated beaglebone black boot area I have that file. FWIW I'm on kernel version 3.8.13-r23a.49.
Trying out the epd_test initially showed me some broken lines and after a little bit what looked like a bit of the cat from the test image. After rechecking the wireup a few times I looked at the code and saw it was expecting a 2 inch screen. That happens in a few places in the code. So I changed those to reflect my hardware. Then the test loop ran as expected!
The next step was getting the FUSE driver installed (change for size needed too). Then the python demos could run. And thus the photo above was made. My next step is to create a function to render cairo to /dev/epd/display in order to drive the display directly from a cairo app.
A huge thank you to rePaper for making this so simple to get going. The drivers for Raspberry and Beagle are up on their github page. I had been looking at the Arduino driver and it's SPI code thinking about porting that over to Linux, but now that's not necessary! I might design some cape love for this, perhaps with a 14 pin IDC connector on it for eInk attaching. Shouldn't look much worse than last night's SPI only monster, though something etched would be nicer.
The 2.7 inch changes are below, the first one is just slightly more verbose error reporting. You'll also want to set EPD_SIZE=2.7 in /etc/init.d/epd-fuse.
diff --git a/PlatformWithOS/BeagleBone/gpio.c b/PlatformWithOS/BeagleBone/gpio.c
index b3ded6f..d1df3df 100644
--- a/PlatformWithOS/BeagleBone/gpio.c
+++ b/PlatformWithOS/BeagleBone/gpio.c
@@ -767,7 +767,7 @@ static bool PWM_enable(int channel, const char *pin_name) {
usleep(10000);
}
if (pwm[channel].fd < 0) {
- fprintf(stderr, "PWM failed to appear\n"); fflush(stderr);
+ fprintf(stderr, "PWM failed to appear pin:%s file:%s\n", pin_name, pwm[channel]
free(pwm[channel].name);
pwm[channel].name = NULL;
break; // failed
diff --git a/PlatformWithOS/demo/EPD.py b/PlatformWithOS/demo/EPD.py
index da1ef12..41cc6c1 100644
--- a/PlatformWithOS/demo/EPD.py
+++ b/PlatformWithOS/demo/EPD.py
@@ -48,8 +48,8 @@ to use:
def __init__(self, *args, **kwargs):
self._epd_path = '/dev/epd'
- self._width = 200
- self._height = 96
+ self._width = 264
+ self._height = 176
self._panel = 'EPD 2.0'
self._auto = False
diff --git a/PlatformWithOS/driver-common/epd_test.c b/PlatformWithOS/driver-common/epd_test.c
index e2f2b5a..afe3cb8 100644
--- a/PlatformWithOS/driver-common/epd_test.c
+++ b/PlatformWithOS/driver-common/epd_test.c
@@ -72,7 +72,7 @@ int main(int argc, char *argv[]) {
GPIO_mode(reset_pin, GPIO_OUTPUT);
GPIO_mode(busy_pin, GPIO_INPUT);
- EPD_type *epd = EPD_create(EPD_2_0,
+ EPD_type *epd = EPD_create(EPD_2_7,
panel_on_pin,
border_pin,
discharge_pin,
So the first image I chose to display after the epd_test was a capture of fontforge editing Cantarell Regular. Luckily, I've made no changes to the splineset so my design skills are not part of the image. The rendering of splines in the charview of fontforge uses antialiasing, as it was switched over to cairo around a year ago. As the eInk display is monochrome the image displayed is dithered back to 1 bit.
With the real time collaboration support in fontforge this does raise the new chance to see a font being rendered on eInk as you design it (or hint it). I'm not sure how many fonts are being designed with eInk as the specific consumption ground. If you are interested in font design, checkout Crafting Type which uses fontforge to create new type, and you should also be able to see the collaboration and HTML preview modes in action.
Getting the actual eInk display to go from the BeagleBone had a few steps. Firstly, I managed to completely fill up the 2gb of eMMC where my Angstrom was installed. So now I'm running the whole show off a high speed 8gb sandisk card. I spent a little extra cash on a faster card, its one of the extreme super panda + extra adjective sandisk ones. The older kernel I had didn't have a duty file for the PWM pin that the driver wanted to use. Now I that I have a fully updated beaglebone black boot area I have that file. FWIW I'm on kernel version 3.8.13-r23a.49.
Trying out the epd_test initially showed me some broken lines and after a little bit what looked like a bit of the cat from the test image. After rechecking the wireup a few times I looked at the code and saw it was expecting a 2 inch screen. That happens in a few places in the code. So I changed those to reflect my hardware. Then the test loop ran as expected!
The next step was getting the FUSE driver installed (change for size needed too). Then the python demos could run. And thus the photo above was made. My next step is to create a function to render cairo to /dev/epd/display in order to drive the display directly from a cairo app.
A huge thank you to rePaper for making this so simple to get going. The drivers for Raspberry and Beagle are up on their github page. I had been looking at the Arduino driver and it's SPI code thinking about porting that over to Linux, but now that's not necessary! I might design some cape love for this, perhaps with a 14 pin IDC connector on it for eInk attaching. Shouldn't look much worse than last night's SPI only monster, though something etched would be nicer.
The 2.7 inch changes are below, the first one is just slightly more verbose error reporting. You'll also want to set EPD_SIZE=2.7 in /etc/init.d/epd-fuse.
diff --git a/PlatformWithOS/BeagleBone/gpio.c b/PlatformWithOS/BeagleBone/gpio.c
index b3ded6f..d1df3df 100644
--- a/PlatformWithOS/BeagleBone/gpio.c
+++ b/PlatformWithOS/BeagleBone/gpio.c
@@ -767,7 +767,7 @@ static bool PWM_enable(int channel, const char *pin_name) {
usleep(10000);
}
if (pwm[channel].fd < 0) {
- fprintf(stderr, "PWM failed to appear\n"); fflush(stderr);
+ fprintf(stderr, "PWM failed to appear pin:%s file:%s\n", pin_name, pwm[channel]
free(pwm[channel].name);
pwm[channel].name = NULL;
break; // failed
diff --git a/PlatformWithOS/demo/EPD.py b/PlatformWithOS/demo/EPD.py
index da1ef12..41cc6c1 100644
--- a/PlatformWithOS/demo/EPD.py
+++ b/PlatformWithOS/demo/EPD.py
@@ -48,8 +48,8 @@ to use:
def __init__(self, *args, **kwargs):
self._epd_path = '/dev/epd'
- self._width = 200
- self._height = 96
+ self._width = 264
+ self._height = 176
self._panel = 'EPD 2.0'
self._auto = False
diff --git a/PlatformWithOS/driver-common/epd_test.c b/PlatformWithOS/driver-common/epd_test.c
index e2f2b5a..afe3cb8 100644
--- a/PlatformWithOS/driver-common/epd_test.c
+++ b/PlatformWithOS/driver-common/epd_test.c
@@ -72,7 +72,7 @@ int main(int argc, char *argv[]) {
GPIO_mode(reset_pin, GPIO_OUTPUT);
GPIO_mode(busy_pin, GPIO_INPUT);
- EPD_type *epd = EPD_create(EPD_2_0,
+ EPD_type *epd = EPD_create(EPD_2_7,
panel_on_pin,
border_pin,
discharge_pin,
Wednesday, October 23, 2013
Open Logic Sniffing
Since I've started doing a little I2C/SPI work I finally got a hold of the gdb of the wire, a logic sniffer. The poor little 8 pin chip in the below is a small EEPROM which I'm doing a read/write cycle to every second from the BeagleBone Black. The sniffer is an Open Workbench Logic Sniffer which is available for around $50. It was a hard choice between that and the more expensive sniffers because a longer capture buffer is usually a handier capture buffer. Though if there are no artificial delays on the bus then I think the OWLS will probably capture the interesting stuff that is happening.
The first trick was to work out how to use triggers in a simple way. Which for me was if I find a 'read' command (byte=3) on the MOSI then that's a trigger. I've told the bone to clock the SPI back to 5khz and the sniffer to run at 10khz which gives me about 2.46 seconds of capture time. So I will surely see two read/write iterations one second apart.
I haven't worked out how to tell the software to ignore lines 1,2,4,5 so they don't show up as noise. So for now those physical snouts are explicitly grounded. After running the SPI analyser in mode 0 I get the below. The only MISO use is on byte 4 where the old value of 0x1 is read from the EEPROM. The activity on the right is setting write enable (0x6) writing the new value (0x2) and then write disable (0x4). Handy to see that chip select is held and released between those three write related tasks as the enable/disable have to be single byte commands where CS is dropped before they are effective.
The conversation is available in byte format in the analyse window shown below. Here you can see the read value go from 3 to 0 to 1. This is because I triggered on the first read, and the first read got back 3 because that is where I stopped the test program last time (ie, it left 3 at the nominated EEPROM address).
Now to get sigrok to have a sniff too.
The first trick was to work out how to use triggers in a simple way. Which for me was if I find a 'read' command (byte=3) on the MOSI then that's a trigger. I've told the bone to clock the SPI back to 5khz and the sniffer to run at 10khz which gives me about 2.46 seconds of capture time. So I will surely see two read/write iterations one second apart.
I haven't worked out how to tell the software to ignore lines 1,2,4,5 so they don't show up as noise. So for now those physical snouts are explicitly grounded. After running the SPI analyser in mode 0 I get the below. The only MISO use is on byte 4 where the old value of 0x1 is read from the EEPROM. The activity on the right is setting write enable (0x6) writing the new value (0x2) and then write disable (0x4). Handy to see that chip select is held and released between those three write related tasks as the enable/disable have to be single byte commands where CS is dropped before they are effective.
The conversation is available in byte format in the analyse window shown below. Here you can see the read value go from 3 to 0 to 1. This is because I triggered on the first read, and the first read got back 3 because that is where I stopped the test program last time (ie, it left 3 at the nominated EEPROM address).
Now to get sigrok to have a sniff too.
Monday, July 29, 2013
GDrive mounting released!
So version libferris-1.5.18.tar.xz is hot off the make dist; including this much ado about mounting Google Drive support. The last additional feature I decided to add before rolling the tarball was support for viewing and adding to the sharing information of a file. It didn't really do much for me being able to "cp" a file to google://drive without being able to unlock it for given people I know to have access to it. So now you can do that from the filesystem as well.
So, since the previous posts have been about the GDrive API and various snags I ran into along the way, this post is about how you can actually use this stuff.
Firstly run up the ferris-capplet-auth app and select the GDrive tab. I know I should overhaul the UI for this auth tool, but since it's mostly only used once for a web service I haven't found the personal desire to beautify it. So inside the GDrive tab, clicking on the "Authenticate with GDrive" button opens a dialog (should become a wizard), the first thing to do as it tells you is visit the console page on google to enable the GDrive API. Then click or paste the auth link in the dialog to allow libferris to get its hands on your data. The auth link goes to google and tells you what libferris is wanting. When you OK that you are given a "code" that you have to copy and paste back into the lower part of the auth capplet this dialog window. Then OKing the dialog will have libferris get a proper auth token from google and you are all set.
So to get started the below command will list the contents of your GDrive:
$ ferrisls google://drive
To put a file up on there you can do something like;
$ date >/tmp/sample.txt
$ ferriscp /tmp/sample.txt google://drive
And you can get it back with cat if you like. Or ferriscp it somewhere else etc.
$ fcat google://drive/sample.txt
Mon Jul 29 17:21:28 EST 2013
If you want to see your shares for this new sample file use the "shares" extended attribute.
$ fcat -a shares google://drive/sample.txt
write,monkeyiq
The shares attribute is a BINEBO (Bytes In Not Equal Bytes Out). Yay for me coining new terms! This means that what you write to it is not exactly what you will get when you read back from it. The handy part of that is that if you write an email address into the extended attribute, you are adding that person to the list of folks who can write to the file. Because I'm using libferris without FUSE and bash doesn't understand libferris URLs, I have to use ferris-redirect in the below command. You can think of ferris-redirect like the shell redirection (>) but you can also supply the extended attribute to redirect data into with (-a). If I read back the shares extended attribute I'll see a new entry in there. Google will have sent a notification email to my friend with a link to the file for me also.
$ echo niceguy@example.com \
| ferris-redirect -a shares google://drive/sample.txt
$ fcat -a shares google://drive/sample.txt
write,monkeyiq
write,Really Nice Guy
I could also add some hookup to your "contacts" to this, so your evolution addressbook nick names or google contacts could be used to lookup a person. In this case, with names changed to protect the innocent etc, so hypothetically google thinks the name for that email address is Really Nice Guy because he is in my contacts on gmail.
All of this extends to other virtual filesystem that libferris supports. You can "cp" from your scanner or webcam or a tuple of a database directly to google drive if that floats your boat.
I've already had a bit of a sniff at the dropbox API and others, so you might be able to bounce data between clouds in a future release.
So, since the previous posts have been about the GDrive API and various snags I ran into along the way, this post is about how you can actually use this stuff.
Firstly run up the ferris-capplet-auth app and select the GDrive tab. I know I should overhaul the UI for this auth tool, but since it's mostly only used once for a web service I haven't found the personal desire to beautify it. So inside the GDrive tab, clicking on the "Authenticate with GDrive" button opens a dialog (should become a wizard), the first thing to do as it tells you is visit the console page on google to enable the GDrive API. Then click or paste the auth link in the dialog to allow libferris to get its hands on your data. The auth link goes to google and tells you what libferris is wanting. When you OK that you are given a "code" that you have to copy and paste back into the lower part of the auth capplet this dialog window. Then OKing the dialog will have libferris get a proper auth token from google and you are all set.
So to get started the below command will list the contents of your GDrive:
$ ferrisls google://drive
To put a file up on there you can do something like;
$ date >/tmp/sample.txt
$ ferriscp /tmp/sample.txt google://drive
And you can get it back with cat if you like. Or ferriscp it somewhere else etc.
$ fcat google://drive/sample.txt
Mon Jul 29 17:21:28 EST 2013
If you want to see your shares for this new sample file use the "shares" extended attribute.
$ fcat -a shares google://drive/sample.txt
write,monkeyiq
The shares attribute is a BINEBO (Bytes In Not Equal Bytes Out). Yay for me coining new terms! This means that what you write to it is not exactly what you will get when you read back from it. The handy part of that is that if you write an email address into the extended attribute, you are adding that person to the list of folks who can write to the file. Because I'm using libferris without FUSE and bash doesn't understand libferris URLs, I have to use ferris-redirect in the below command. You can think of ferris-redirect like the shell redirection (>) but you can also supply the extended attribute to redirect data into with (-a). If I read back the shares extended attribute I'll see a new entry in there. Google will have sent a notification email to my friend with a link to the file for me also.
$ echo niceguy@example.com \
| ferris-redirect -a shares google://drive/sample.txt
$ fcat -a shares google://drive/sample.txt
write,monkeyiq
write,Really Nice Guy
I could also add some hookup to your "contacts" to this, so your evolution addressbook nick names or google contacts could be used to lookup a person. In this case, with names changed to protect the innocent etc, so hypothetically google thinks the name for that email address is Really Nice Guy because he is in my contacts on gmail.
All of this extends to other virtual filesystem that libferris supports. You can "cp" from your scanner or webcam or a tuple of a database directly to google drive if that floats your boat.
I've already had a bit of a sniff at the dropbox API and others, so you might be able to bounce data between clouds in a future release.
Saturday, July 27, 2013
The new google://drive/ URL!
The very short story: libferris can now mount Google Drive as a filesystem. I've placed that in google://drive and will likely make an alias from gdrive:// to that same location so either will work.
The new OAuth 2.0 standard is so much easier to use than the old 1.0 version. In short, after being identified and given the nod once by the user, in 2.0 you have to supply a single secret, in 1.x you have to use per message nonce, create hashes, send the key and token, etc. The main drawback of 2.0 is that you have to use TLS/SSL for each request to protect that single auth token. A small price to pay, as you might well want to protect the entire conversation if you are doing things that require authentication anyway.
A few caveats of the current implementation: mime types on uploaded files are based on file name sniffing. That is because the upload you might be using cp foo.jpg google://drive and the filesystem copies the bytes over. But GDrive needs to know the mimetype for that new File at creation time. The GDrive PATCH method doesn't seem to let you change the mimetype of a file after it has been sent. A better solution will involve the cp code prenotifying the target location so that some metadata (mimetype) can be prefetched form the source file if desired. That would allow full byte sniffing to be used.
Speaking of PATCH, if you change metadata using it, you always get back a 200 response. No matter what. Luckily you also get back a JSON file string with all the metadata for the file you have (tried to) updated. So I've made my PATCH caller code to ignore the HTTP response code compare the returned file JSON to see if the changes actually stuck or not. If a value isn't set how it is expected my PATCH returns an exception. This is in contrast to the docs for the PATCH method which claims that the file JSON is only returned "if successful".
Oh yeah, one other tiny thing about PATCH. If you patch the description it didn't show up in Firefox for me until I refreshed the page. Changing the title does update the Firefox UI automatically. I guess the sidepanel for description hasn't got the funky web notification love yet.
There are two ways I found to read a directory, using files/list and children/list. Unfortunately the later, while returning only the direct children of a folder, also only returns a few pieces of information for those children the most interesting being the child's id. On the other hand the files/list gives you almost all the metadata for each returned File. So on a slower link, one doesn't need thinking music to work out if one round trip or two are the desired number. The files/list also returns metadata for files that have been deleted, and files which other's have shared with you. It is easy to set a query "hidden = false and trashed = false" for files/list to not return those dead files. Filtering on the server exclusively for files that you own is harder. There is a query alias sharedWithMe but no OwnedByMe to return the counter set. I guess perhaps "not sharedWithMe" would == OwnedByMe.
Currently I sort of ignore the directory hierarchy that files/list returns. So all your drive files are just in google://drive/ instead of subdirs as appropriate. I might leave that restriction in the first release. It's not hard to remove, but I've been focusing on upload, download, and metadata change.
Creating files, updating metadata, and downloading files from GDrive all work and will be available in the next libferris release. I have one other issue to cleanup (rate limiting directory read) before I do the first libferris release with gdrive mounting.
Oh and big trap #2 for the young players. To actually *use* libferris on gdrive after you have done the OAuth 2.0 "yep, libferris can have access" you have to go to code.google.com/apis/console and enable drive API for your account otherwise you get access denied errors for all. And once you goto the console and do that, you'll have to OAuth again to get a valid token.
A huge thank you for those two contributed to the ferris fund raising after my last post proposing mounting Google Drive!
The new OAuth 2.0 standard is so much easier to use than the old 1.0 version. In short, after being identified and given the nod once by the user, in 2.0 you have to supply a single secret, in 1.x you have to use per message nonce, create hashes, send the key and token, etc. The main drawback of 2.0 is that you have to use TLS/SSL for each request to protect that single auth token. A small price to pay, as you might well want to protect the entire conversation if you are doing things that require authentication anyway.
A few caveats of the current implementation: mime types on uploaded files are based on file name sniffing. That is because the upload you might be using cp foo.jpg google://drive and the filesystem copies the bytes over. But GDrive needs to know the mimetype for that new File at creation time. The GDrive PATCH method doesn't seem to let you change the mimetype of a file after it has been sent. A better solution will involve the cp code prenotifying the target location so that some metadata (mimetype) can be prefetched form the source file if desired. That would allow full byte sniffing to be used.
Speaking of PATCH, if you change metadata using it, you always get back a 200 response. No matter what. Luckily you also get back a JSON file string with all the metadata for the file you have (tried to) updated. So I've made my PATCH caller code to ignore the HTTP response code compare the returned file JSON to see if the changes actually stuck or not. If a value isn't set how it is expected my PATCH returns an exception. This is in contrast to the docs for the PATCH method which claims that the file JSON is only returned "if successful".
Oh yeah, one other tiny thing about PATCH. If you patch the description it didn't show up in Firefox for me until I refreshed the page. Changing the title does update the Firefox UI automatically. I guess the sidepanel for description hasn't got the funky web notification love yet.
There are two ways I found to read a directory, using files/list and children/list. Unfortunately the later, while returning only the direct children of a folder, also only returns a few pieces of information for those children the most interesting being the child's id. On the other hand the files/list gives you almost all the metadata for each returned File. So on a slower link, one doesn't need thinking music to work out if one round trip or two are the desired number. The files/list also returns metadata for files that have been deleted, and files which other's have shared with you. It is easy to set a query "hidden = false and trashed = false" for files/list to not return those dead files. Filtering on the server exclusively for files that you own is harder. There is a query alias sharedWithMe but no OwnedByMe to return the counter set. I guess perhaps "not sharedWithMe" would == OwnedByMe.
Currently I sort of ignore the directory hierarchy that files/list returns. So all your drive files are just in google://drive/ instead of subdirs as appropriate. I might leave that restriction in the first release. It's not hard to remove, but I've been focusing on upload, download, and metadata change.
Creating files, updating metadata, and downloading files from GDrive all work and will be available in the next libferris release. I have one other issue to cleanup (rate limiting directory read) before I do the first libferris release with gdrive mounting.
Oh and big trap #2 for the young players. To actually *use* libferris on gdrive after you have done the OAuth 2.0 "yep, libferris can have access" you have to go to code.google.com/apis/console and enable drive API for your account otherwise you get access denied errors for all. And once you goto the console and do that, you'll have to OAuth again to get a valid token.
A huge thank you for those two contributed to the ferris fund raising after my last post proposing mounting Google Drive!
Subscribe to:
Posts (Atom)






