Notes from FOSDEM 2013

Given that FOSDEM has been going for thirteen years, that it’s all about open source development, which I’ve been involved with for about the same length of time, that it’s held in Brussels, where I used to work, that it’s enormous (about 5,000 attendees), and that it costs nothing to attend, it’s perhaps surprising that I’d never been before. In fact, I wasn’t even really aware of it until late last year.

I’ve been missing out up until now.

FOSDEM is huge: there were twenty-four tracks of talks this year, over two days. You can’t see all of it. You can’t even see 4% of it, what with the time taken to walk all over campus. You can know people that are there and not see them all weekend!

I know that I missed many interesting talks, but I still saw plenty that piqued my interest. So here’s a themed dump of links:

Graph databases

I spent about half of Saturday in the graph database room, mainly because I have the feeling that there’s a whole lot of interesting work going on in that field that’s largely ignored and replicated badly on top of more traditional data stores.

  • Signal/Collect is a system for fast and scalable processing of graphs. Seems conceptually related to map/collect, and has similar scaling properties. Looks easy to write for. JVM-based
  • fluxgraph is a ’temporal graph database on top of Datomic. It allows querying of the graph at arbitrary moments in the past, and computation of a difference graph between two points in time. Developed to meet regulatory requirements in the pharmaceutical business.
  • structr is a CMS built on Neo4j. I couldn’t help but think that it was too clever for its own good: just because you can model an entire HTML document as a graph doesn’t mean you should.

Kernel programming

  • Rathaxes is a DSL for describing and building device drivers. The promise is that it can reduce boilerplate and repetition, and make it easy to decouple hardware interfacing from OS-level code, and thus make it easy to write cross-platform drivers. It’s still very immature, and I think we’re a long way from seeing whether this approach will work in the real world, but it’s a useful avenue of enquiry.

UPnP/DLNA

DLNA is a set of specifications for interoperability between entertainment devices. It encompasses addressing and transcoding, and lets a (relatively dumb) smart TV play video from a home server, or a phone push a photo to a big screen. What’s more, it actually exists: it’s widely deployed.

I saw a couple of good demos: in one, the presenter took a photo on his phone, picked it up on a computer, and pushed it to a computer attached to a projector. In the other, the presenter had written a small application that would convert the slides of a presentation to JPEG images and tell a computer running a DLNA renderer (XBMC, in this instance) to show the images on screen. Sounds easier than finding the correct MacBook dongle ….

  • Rygel is a server and associated libraries to serve media to DLNA devices.
  • Dleyna is a counterpart to Rygel: a set of libraries to implement players and renderers.
  • Guacamayo is ‘an Open Source software platform for creating multimedia appliances, including multimedia UPnP/DLNA servers, audio players, and full-featured multimedia centres.’

Communications

Besides open-source mobile platforms, breaking out of the mutually-incompatible walled gardens of Skype, Viber, FaceTime etc. was a big theme. The observation that you can’t have secure communications without open source received a cheer from the auditorium.

  • Replicant is a project to take the released open-source parts of Android and write the missing libraries for hardware interfacing, so that you can really be sure that your phone is doing what you want, and not secretly recording you, etc. This is surprisingly difficult, as some phones mediate access to GPS and voice through a closed modem; even if not, shared memory may allow a modem to steal voice and position data. Whilst this may sound a little paranoid, it’s interesting even if you don’t yet live under a repressive régime.
  • Firefox OS looks very good. I played with it on a couple of devices. It seems that a multiple-core processor is needed for decent responsiveness, and even then it feels a bit laggy. It took a long time for Android to sort that out, but I worry that Firefox OS doesn’t have that time to spare. I was happy to see that it copies the idea of Android’s Intents to allow easy replacement and augmentation of basic functions.
  • Kamilio is an open-source SIP server that you can use to set up a Skype-like service in under an hour.
  • ReSIProcate is ‘dedicated to maintaining a complete, correct, and commercially usable implementation of SIP and a few related protocols.’
  • Lumicall aims to provide easy-to-use free calling on Android.
  • Jitsi does open-source voice, video, and chat. Kind of like Skype, only actually secure, using documented and inspected protocols.

Music

The talk on OpenPipe was probably my favourite from the entire conference. The presenter and developer, Xulio Coira, started off playing some Galician bagpipes, and followed it up with an entertaining talk broken up by various recordings and live demos of electronic bagpipe music.

  • OpenPipe is DIY MIDI electronic bagpipes.
  • FluidSynth is a software SoundFont 2 synthesiser.

If you’re anywhere near Belgium next February (the first weekend, I believe), then I highly recommend it. It’s easy and fairly cheap to get to Brussels from London, even though, due to the uniquely xenophobic way in which the UK is governed, you do have to go through three separate passport checks to get back in.

Casualties of war

We need to take information, wherever it is stored, make our copies and share them with the world. We need to take stuff that’s out of copyright and add it to the archive. We need to buy secret databases and put them on the Web. We need to download scientific journals and upload them to file sharing networks. We need to fight for Guerilla Open Access. —— Aaron Swartz, Guerrilla Open Access Manifesto, 2008.

At the time of his suicide, Aaron Swartz was facing federal charges that could lead to thirty or more years in prison, and which would have cost at least $1.5 million to defend, for sneaking a laptop into an MIT cupboard and using it to download journal papers.

The system crushed him for his impertinence. He fought to liberate information. It seems quite likely that he died for it.

What struck me was this: what he did was the kind of thing I would do. It’s the kind of thing I have done. Some of you reading will have, too.

Mulled wine: theory and practice

In this part of the northern hemisphere, it’s the time of short days, long nights, miserable weather, historically implausible religious stories, and mulled wine. Well, it’s a bit past— Saturnalia is probably peak mulling time—but it’s still cold enough out to keep on mulling, and I’ve been iterating my recipe.

mulled wine in cup

I’ve never been quite satisfied with my mulled wine, but this year, I think I worked out what I’ve been doing wrong. It’s the wine part. It’s not that I was using bad wine, just that heated wine can be a bit overpowering. The trick seems to be to make something hot, spiced, and tasty, and then add wine to it—but you don’t necessarily need all that much wine. In other words, (my idea of) good mulled wine is a hot drink that contains wine, and not just wine with spices in. Given that many mulled wine recipes call for the addition of hard liquor to neat wine, this recipe is a bit contrarian, but I think it works—in terms both of flavour and of long-term liver function.

Here’s my current recipe:

750 ml water
375 ml red wine (i.e. half a bottle)
1 orange, in 1 cm transverse slices
4 cloves
1 small piece of cinnamon stick
4 peppercorns
¼ tsp fennel seeds
¼ tsp coriander seeds
1 star anise
1 teabag
50 g sugar

Put everything except the wine and sugar in a saucepan, bring to the boil, and simmer for 10 minutes or so. Remove the orange slices and teabag and discard (the orange pith will otherwise quickly make it bitter). Add sugar and wine and reheat but don’t boil.

It’s not final. Recipes never are—I might cut the sugar a bit next time, and maybe add some sliced ginger—but I think it’s heading in the right direction. It’s all a matter of taste, though.

Stupid security questions

If you’re going to make your customers choose and answer security questions, they should be factual, answerable, and not public knowledge. NS&I fail badly.

I signed into my account the other day to update my address (as I’ve moved house), and was prompted to add some security questions and answers. I had to choose five questions from a range of about a dozen:

What was the name of your first pet?

Fewer than half of British households have a pet. I’ve never owned one.

What is your mother’s middle name?

Guess whose mother doesn’t have a middle name? Chinese people, Japanese people, Korean people—and, yes, many British people—don’t have middle names.

Which sports team do you support?

I can’t find any numbers on this, but, anecdotally, many men and probably most women I know don’t support a team. I don’t.

However, people who do support a team are usually fairly vocal about it. Just look at their scarf, or their car.

What was the make of your first car?

A quarter of British households don’t own a car. I’ve never owned one.

What was the TV series you liked most as a child?

This is not a fact. I was a child for somewhere between 13 and 18 years, depending on how you calculate it. My memories of the television I liked a quarter of a century ago are hazy—besides which, my preferences did not remain static over this time.

What is your eldest child’s middle name?

I don’t have any children. Many people don’t. Younger people are more likely to have never had children.

What is your grandfather’s profession?

Ah! At last! A (near-) universal question. Most people know who their grandparents were, and what they did. The exclusive use of the present tense probably isn’t quite right, though.

But wait … which grandfather? Everyone has (or had) two.

In which year did you get married? (YYYY)

Like two thirds of the British population, I’m not married.

When is your wedding anniversary (DDMM)

No, really. I’m not married.

What is the first name of your eldest child?

I still don’t have any children.

Which university did you attend?

Another one I can answer! It’s not a secret, though, is it? And only about half of young people even go to university; the proportion of the entire population who have ever attended is lower.

What is the first name of your eldest brother/sister?

I can’t say how many people have at least one sibling, but the most common number of children per family is two, so it’s not unreasonable. However, it’s publicly-available information, and not very secure at all.

So there you have it. A small set of questions, few of which are universally applicable, many of which are public information that no-one would think to keep secret, and a couple of which are ambiguous.

I’d be interested to know what level of analysis and thought went into requiring and setting these questions. I’m guessing not much. Yes, I’ve complained. I don’t expect it to have much effect.

Wanted: Adblock for paper tickets

I just bought some Eurostar tickets online. (For FOSDEM, incidentally; anyone else going?) Like most travel operators, they allow you to download PDF tickets to print at home.

Like most travel operators, they see the extra space on a sheet of A4 paper as an excellent marketing opportunity. Well, I don’t like advertising at the best of times, but wasting my own toner printing adverts on the tickets I’ve just spent £80 to buy? That’s simply intolerable!

Eurostar ticket with ads

So what do I do? Load it up in PDFedit and draw white rectangles over the ads (because it’s easier than working out how to delete them), re-save the PDFs, then print them out.

I wonder if I’m missing an easier way, though—and I don’t mean learning to accept the consumer world as it is, although that would be easiest of all.

At the very least, it seems that an automated solution should be possible. It would take disproportionately longer to implement it than any time I’d save, but it might be a fun challenge: Adblock for paper tickets.

Ignorance is bliss: better software through stupidity

I often say that I love deleting code, and it’s true, but it’s not often that I get a really good example like this, of finding a generic problem hiding behind some specific code, and ending up with something that’s simpler, better, and can do more.

Background

I’ve spent the last couple of months working with Sidekick Studios on improvements to the client side of a mobile market research application. It’s been fun, and, especially after my previous experience in a rather large organisation, reinforced my belief that I enjoy and find it easiest to contribute in small companies.

One feature of the admin web app is the ability to generate zip files of multiple photos and videos on demand. As the main application stack (Ruby on Rails) isn’t well suited to this kind of streaming operation, the downloads were implemented with a short and simple Node.js application that fetched files from S3 and streamed them into a zip file and out to the client.

This approach works very well: the download starts almost instantly, and Node.js is a good fit for this straightforward IO-bound process. I suspect that Go would also work well, incidentally.

At the point when I started working on the project, the downloader worked out which files to put in the zip file by querying the SQL database directly. This wasn’t a terrible thing, but it posed a few problems as we added features:

  • It was coupled to the database schema
  • It didn’t know about the more complex permissions we were adding
  • We wanted to add HTML pages to the zip file to put the files into context

All of these things could have been added to the downloader, but it would have been duplication: the web app already knew how to do this stuff.

The solution, obviously, was to make the downloader more stupid, so that it could just freeload on the web app’s functionality. Here’s how I did it:

Step 1: Remove the database connection

Instead of querying the database to find the files, just ask the web app for a list of files to include. This is the manifest. As well as the list of files, it also includes what they should be stored as, and the name of the zip file to be generated.

As the manifest is JSON, it’s trivial to parse into a JavaScript data structure. In fact, since I used restler, I didn’t even have to do the deserialisation myself.

Step 2: Add the ability to download files from HTTP(S) as well as S3

Need HTML? Just ask the web app. This goes for static assets as well. We could put the latter in the downloader, but the general principle is for the downloader to know as little as possible.

This means that each file entry now has three pieces of information:

  • Where to find it
  • How to get it (HTTP or S3 API)
  • Where to put it

Step 3: Make it secure

We want to make sure that people can only download files that they’re allowed. Since we know that the download process starts in the web app, when an authenticated person follows a link, and that the downloader needs to request the manifest from the web app, we just need some identifier to tie the two together — the download token:

  • Person clicks link, e.g. http://www.example.com/downloads/42
  • Web app checks permissions, generates token for requested object
  • Web app redirects to downloader with token, e.g. http://dl.example.com/?token=4d3n7
  • Downloader requests manifest with token, e.g. http://www.example.com/manifests/?token=4d3n7
  • Web app generates manifest for object identified by token
  • Web app sends manifest to downloader

The token is also used when requesting any HTML files generated by the web app for the download.

The download token is a long string composed of four parts:

[ object type | object ID | timestamp | HMAC ]

The timestamp lets the application check that the token isn’t too old, whilst the HMAC ensures that the token is authentic and hasn’t been tampered with.

The token does not need to be persisted in a database for authentication, nor does it need to be parsed anywhere outside the web app. The downloader can treat it as an opaque string that it just appends to every request.

Is it stupid enough yet?

By this stage, the downloader only needs to know two pieces of information, neither of which is specific to the project:

  • Where to find the manifests
  • The S3 credentials

The S3 credentials are not even strictly necessary—we could fetch the files via HTTP—but requests through the public interface are more expensive than within AWS.

I’ve lied a bit, because I didn’t just remove code: I added features to the web side (notably the download token and manifest generation). However, the downloader app is now about 20% shorter and does a lot more, despite knowing a lot less.

Burning poppies

Arresting someone who posted a picture of a burning poppy to Facebook may please the mob, but I find it more than a little disturbing and authoritarian.

From the Kent Police website (emphasis theirs):

A man is due to be interviewed by police this morning following reports that a picture of a burning poppy had been posted on a social media website.

Officers were contacted at around 4pm yesterday, Sunday, 11 November 2012 and alerted to the picture, which was reportedly accompanied by an offensive comment.

Following an investigation by Kent Police a 19-year-old, Canterbury man was arrested on suspicion of an offence under the malicious communications act. He is currently in custody.

It may be a tasteless and provocative thing to do, to burn a paper-and-plastic Remembrance poppy. It may even release a small amount of noxious fumes into the immediate vicinity. But arresting someone for it? That feels like the heavy-handed suppression of political dissent. Even if it is the poorly-articulated inchoate dissent of a teenager. Even if it’s stupid.

We’re better than this.

Tablet computing

I’m no Amish. I sit here surrounded by technology. However, I’m beginning to think about technology more in terms of whether it can improve my life—which is, as I understand it, the basis of the Amish engagement with technology. It’s difficult to strike a balance: technology can drive changes in lifestyle, and they’re not always benign. At the same time, technology can have a positive impact, and I’ve found the Nexus 7 fits that category for me.

It was the impact of computers on my leisure time that led me to buy a tablet. I sit at a desk much of the day, and I don’t want to do the same all night. I’d found that I was using my mobile phone a lot to avoid this, but that such a small device was frustrating. I’d soon end up at the computer, and sitting at a 24” glowing rectangle is really no way to get ready for sleep. I found myself distracted and ended up staying up too late night after night.

I would never have bought an iPad: too bulky, too heavy, and too Apple. (I really don’t like their walled ecosystem, nor their proprietary connectors and protocols and—more practically—the result that it’s hard to use their products with Linux.) I didn’t see anything compelling from other makers, either: the HP Touchpad died before it was born, the BlackBerry tablet seems like a cruel joke, and most of the Android tablets suffered with value-subtracting manufacturer customisations, lacklustre hardware, and poor access to updates.

The Nexus 7 was the first tablet that really seemed viable to me. 7” is a good form factor. It has a beautiful screen. It’s well put together. Granted, it’s not the watchmaker assembly standard of Apple’s products, but it feels reasonably solid, and the composites used have appealing tactile qualities. And it’s cheap! The basic model is £159.

In fact, the £159 8 GB model isn’t readily available except online, for an extra delivery charge of £15, so I went for the 16 GB version, which I could buy round the corner for £199 with no shipping costs.

So far, I’ve had it for three or four weeks, and its doing exactly what I wanted: letting me communicate and seek information away from a computer. I frequently spend entire evenings without even turning my computer on. If that sounds like a banal achievement, it’s nonetheless an improvement in my quality of life, and it’s made it a lot easier to get to bed on time.

There are limitations: some (though relatively few) websites don’t work well, and I miss the extensibility of desktop browsers—especially when it comes to advertising. At the moment, I’m using an ad-blocking proxy on my home server to make the web tolerable. As a little computer, though, the Nexus is really competent. It feels responsive and smooth, and most of the oddities in earlier versions of Android have been rationalised.

Just in case you think it’s only good for passive activities, I’m sitting on the sofa writing this on the Nexus 7. It’s something of a challenge to type with only two thumbs, but predictive completion makes it easier than you’d think. It’s also lot nicer to type on a 7” tablet than a 3” phone.

I’m also beginning to realise that the age of the laptop is probably coming to an end. You can buy a device that’s more portable, with better battery life and a nicer screen, for half the price. With the right software and peripherals (like a keyboard), I could do almost all my work on this hardware. It certainly seems fast enough.

A feudal life

It’s nearly time to move house. Again. Just like last time, I’m quite happy living where I do, but my landlord has decided to—oh, I don’t know, realise the capital sequestered in their asset, or some such bullshit—so they’re selling it, and I’m shortly to become homeless unless I find a new dwelling.

When I was a child, my father was in the RAF. We had to move every two or three years as he was posted to somewhere else (invariably a rather godforsaken part of the UK, cut off from civilisation). I wouldn’t recommend it as a way of life. As a result, I was quite determined that once I became an adult I’d take control of my own life. No one would tell me where to go. And yet, here I am, forced to move every few years, only this time it’s capitalism rather than the military that wields the power. Freedom of choice is so illusory.

The common truth of buying houses is that location trumps all other considerations. This makes sense: once bought, you can’t move a house, but you can steadily replace the previous inhabitant’s avocado bathroom, flock wallpapers, or whatever other peculiar solecisms they may have left behind.

For renting, it’s different. In a sense, the same holds true as for buying: price is related largely to location and number of rooms, and is indifferent to the quality of the fittings. However, as a tenant, you’re stuck with the fittings and décor. For the fastidious tenant (such as your author), this is potentially a bonus: one simply has to decide on one’s requirements, then find somewhere that suits.

But this is harder than it sounds. Letting agents categorise things in terms of features—like the watch section of the Argos catalogue, enumerating each timepiece by the preposterously abyssal depths to which it is purported to function—but pay no heed to whether the kitchen is a fetid galley redolent of Trainspotting or a spacious hall in which a TV chef could happily swing a cat. But they all have ovens, for some definition of oven.

You’d have thought that letting agents were wasting their time gaining access to and showing people around properties that don’t meet their needs, and yet the information on their websites is always incomplete, and the pictures are always shot with techniques that disguise the actual size of rooms. Floor plans are very rare.

Furthermore, properties don’t sit around waiting for tenants for long. If you like somewhere, you’d better take it: it won’t be there tomorrow.

And then, you have to balance viewings with work.

All this adds up to an awful exercise in game theory—and I’m really not a gambler. Whatever the gene that codes for the vice of gambling is, I don’t have it. Usually, this saves me money, but when it comes to renting, I’m at a disadvantage: an ingénu thrust into a murky casino.

After seeing a whole load of inappropriate flats, I finally saw one that was actually nice. Overcome by relief, I decided to take it. I put down a week’s rent as a holding deposit.

And then I spent the next two nights racked with anxiety: debilitating stomach pains, and a heart rate more appropriate to a terrified rodent. It wasn’t quite a panic attack, but it was pretty awful nonetheless.

My subconscious was correct: the flat wasn’t right. Nicely fitted out, but not right. Too close to a busy road. Too little storage space. Too little space in general. The worst thing to do would be to end up stuck there for six months to avoid wasting my deposit, so I had to write it off. A week’s rent. Not a vast amount, but still, not insignificant. A waste. Maybe an education. Just another expense on top of all the other incidental expenses of moving house.

And I still don’t know where I’m going to live. I’ve got six weeks left to work that out. In a sense, I’m in a good position: my end date in my current flat is flexible. All I have to do is hold out until I find somewhere I want to live. But it’s a stressful way to exist.

It doesn’t have to be like this. We could have decent conditions for renters in England, as they do in other countries. We could have better-regulated agents, better rights for tenants, control of rent increases. But all that would get in the way of the rights of the property-owning class to make profit off us lowly plebs. It’ll never happen.

Destructuring assignment in Ruby

My post on underscores in Ruby attracted quite a lot of interest, particularly on the topic of destructuring assignment, so I thought I’d go into a bit more detail.

As mentioned previously, Ruby supports destructuring assignment. You can assign an array to multiple variables both in direct assignment:

a, b, c = [1, 2, 3]

and in block parameters:

[[1, 2, 3]].map { |a, b, c| "..." }

I’ll explain everything with the first form from here on, as it’s a little simpler. Unless I’ve missed something, all these forms should work equally well in both forms; let me know if you spot something that doesn’t. I’ll only be talking about Ruby 1.9, too, as Ruby 1.8 doesn’t support all of these syntactic constructs.

The outermost square brackets are optional when assigning variables, but I’ll use them throughout as I think it’s a little clearer in this context.

Multi-level destructuring

You can parenthesise any set of variables on the left hand side to mirror the structure on the right:

a, (b, c) = [1, [2, [3, 4]]]

a # => 1
b # => 2
c # => [3, 4]

but you don’t have to stop there. It’s turtles all the way down:

a, (b, (c, d)) = [1, [2, [3, 4]]]

a # => 1
b # => 2
c # => 3
d # => 4

‘Splatting’: assigning multiple elements

Extra values on the right hand side are usually discarded:

a, b, c = [1, 2, 3, 4, 5]

a # => 1
b # => 2
c # => 3

Prepending an asterisk—the ‘splat’ operator—to a variable tells it to gather up all the unassigned elements:

a, b, *c = [1, 2, 3, 4, 5]

a # => 1
b # => 2
c # => [3, 4, 5]

This works at the beginning of the list, too:

*a, b, c = [1, 2, 3, 4, 5]

a # => [1, 2, 3]
b # => 4
c # => 5

or in the middle:

a, *b, c = [1, 2, 3, 4, 5]

a # => 1
b # => [2, 3, 4]
c # => 5

But you can only use it once:

a, *b, *c = [1, 2, 3, 4, 5]

# stdin:81: syntax error, unexpected tSTAR
# a, *b, *c = [1, 2, 3, 4, 5]
#         ^

That is, you can only use it once at a given level. Each level of parentheses can have a splat:

*a, (b, *c) = [1, 2, [3, 4, 5]]

a # => [1, 2]
b # => 3
c # => [4, 5]

Ignoring elements

You can reuse an underscore to represent any element you don’t care about:

a, _, b, _, c = [1, 2, 3, 4, 5]

a # => 1
b # => 3
c # => 5

To ignore multiple elements, use a single asterisk—I’m going to call it a ‘naked splat’ for no better reason than that it sounds a bit amusing:

a, *, b = [1, 2, 3, 4, 5]

a # => 1
b # => 5

The same rules apply to naked splats as to splatted variables: you can only use one in any given level of parentheses, but you can reuse it at each level:

a, *, (*, b, c) = [1, 2, 3, [4, 5, 6, 7]]

a # => 1
b # => 6
c # => 7

I wouldn’t necessarily encourage you to leap straight out and start using destructuring assignment everywhere, but it has its place. Use it where it makes your code clearer, and remember: if you need to be compatible with Ruby 1.8, much of this won’t work.