Google Breakpad and the post crash experience

Google Breakpad has many components to it, but at the basic level it lets you capture information at the time a crash occurs and upload that to the net. A really cute part of Breakpad is that the binary doesn't need to have the debug symbols in it, you don't even need to have them on the client machine at any location. When you build version $githash then you use a breakpad tool to copy out the debug symbols into separate files. When the user discovers a crash they upload a minidump file to a server of your selecting. Then you can combine the extracted symbols from build time and the minidump file to generate a backtrace with line number information. So software users don't have to know about gdb or lldb or whatnot and how to make a backtrace and where to paste it.

I recently updated FontForge's use of breakpad to use a small server on localhost to report the bug. The application dmg file for fontforge will soon also include the extracted symbols for the build. By telling breakpad to use a local server, that server can lookup the symbols that are shipped and generate a human readable backtrace with line number information. Because its also a web interface and running locally, it can spawn a browser on itself. So instead of getting the Mac dialog supplied by the osx crash reporter app telling you that there was a crash, you get a web page telling you the same thing. But the web page can use jQuery/Bootstrap (or $ui tool of choice) and ask what the user was doing and offer many ways to proceed from there depending on how the user wants to report things. The https://gist.github.com/ site can be used to report without any login or user accounts. It's also rather handy as a place to checking larger backtraces that might be, maybe, 50-100kb.

But once you can upload to gist, you can get a http and other URL links to the new gist. So it makes sense from there to offer to make a new github issue for the user too. And in that new issue include the link to the gist page so that developers can get at the full backtrace. It turns out that you can do this last part, which requires user login to github, by redirecting to github/.../issues/new and passing title and body GET parameters. While there is a github API, to report a new issue using it you would need to do OAuth first. But in the libre world it's not so simple to have a location to store the OAuth secure token for next time around. So the GET redirect trick nicely gets around that situation.


For those interested in this, the gist upload and callback to subsequently make a github issue are both available. The Google Breakpad hands over the minidump to a POST method which then massages the minidump into the backtrace and spawns a browser on itself. The GET serves up all the html, css, js, and other assets to the browser and that served html/js is what I link to at the start of the paragraph which is where the actual upload/reporting of the backtrace takes place.

The only thing left to do is to respond to the backtraces that come in and everybody gets a more stable FontForge out of the deal. It might be interesting to send off reports to a Socorro server too so that statistics month on month can be easily available.