It’s fairly simple to extract the data from a JSONP callback in a reliable, repeatable way once you know how. Here’s how.

I wanted to extract the playlist from an internet radio station. The server in question exposes currently playing information via a JSONP API, the intended purpose of which is to update an HTML element on a page with the details of the track currently playing.

JSONP, in case you’re not familiar with the concept, is a means of inserting data from a third-party service into a web page. The third-party server responds to a request with a fragment of JavaScript that calls a function in the originating page.

The service in question returns data like this:

var arr14b178d7 = { 'song': 'blah', 'listeners': 7 };

cc_streaminfo_get_callback(arr14b178d7)

How do we get this into a Ruby application? We could attempt to parse the JavaScript object itself, but that’s a bit ugly and error-prone. We can’t use a JSON parser, because it doesn’t conform to the (restrictive) rules of JSON. Much better to use something like Johnson to evaluate it.

Unfortunately, that doesn’t work all that well:

Johnson.evaluate(s)

# => Johnson::Error: cc_streaminfo_get_callback is not defined at (none):3

Well, let’s split on the blank line, then, and just evaluate the object:

Johnson.evaluate(s.split(/\n\n/).first)

# => nil

Still no good: variable assignment doesn’t return a value, so all we get is nil. And besides, that was a rather fragile attempt at parsing.

In fact, we can take advantage of the fact that the function call is the last statement in the JSONP. If we define the missing function as a stub that returns its argument, then that will be the final result of the JavaScript fragment:

callback = <<-END
  function cc_streaminfo_get_callback(details){
    return details;
  }
END
object = Johnson.evaluate(callback + s)

# => [object Object]

And that object is the thing we want:

object["song"]
# => "blah"

And there you go: a simple and reasonably reliable way of parsing the response from a JSONP API.

(Strictly speaking, I’m not sure if this API actually counts as JSONP as it doesn’t allow us to specify the function name. That doesn’t make any difference to the implementation.)

I ended up cycling on one of the new Cycle Superhighways last weekend. I’m not really convinced of how useful they actually are.

What is a Cycle Superhighway? It’s a metre and a half of blue tarmac along the edge of the road, with a route designation (e.g. CS7) marked at intervals.

That’s it, so far.

Are they any good? Well, no. Not really. There are two main problems:

1. If you take a four-lane road and cover half of one lane in each direction with blue paint to designate it as a ‘Cycle Superhighway’ (as TfL have done), cars still use the entire lane. So the cyclist doesn’t gain any space. In fact, because the gutter is now nominated as the zone for cyclists, there’s now pressure on cyclists not to take any more of the lane than the blue paint allows.
2. The blue paint is invisible at night under the yellow sodium lights. This is basic physics, but didn’t seem to occur to Boris and crew.

The only useful thing that the Cycle Superhighways provide, as far as I can tell, is that they allow a cyclist to decide to follow a nominated route for a long distance without getting lost. In that respect, however, it’s really no better than painting the route designation alone on the road, along with some clarifying markings at junctions. And, in fact, we already have such things: you might as well follow the A3 and A24 as CS7.

They do feel more like a rushed gimmick and less like a genuinely useful enhancement to cycling in the capital. Superficial Cycleways indeed.

These are the images that currently decorate the home page of the UK Border Agency website:

Someone being handcuffed:

And someone being fingerprinted:

This is not acceptable. Border control does indeed entail arrests and enforcement and all that nasty stuff. Sometimes. But it is not and should not be the core business of the UK Border Agency.

This kind of imagery is horrible, authoritarian, and dehumanising, and I’m appalled at the kind of mentality that it implies. It’s not the way that we should be presenting ourselves to visitors.

You know those tick boxes on websites? The ones that you just have to click that say something like ‘I have read and understood the terms and conditions’? It turns out that they’re a load of old rubbish.

That’s what I’d always thought, anyway. Legal arse-covering is like a memetic virus: people see some legalese on a website, and assume that it’s essential, useful, necessary, and—ho ho ho—that it’s actually going to indemnify them in some way. So they copy it, like a cargo cult building a runway in the jungle.

As I’ve been forced to implement the stupid tick boxes on one application after another, with none of us really asking why, I’ve often wanted to just say no. But, of course, my objections had as little justification as the custom and practice of putting those boxes everywhere.

Well, now I have something to back me up. All those tick boxes are useless anyway. That’s what the FSA says [PDF]:

Consumers should check the details of the contracts they enter into. But a contract term requiring consumers to declare that they have read and understood the terms of the contract is likely to be unfair because it binds customers to terms which, in practice, they may not have any real awareness of.

Of course, not everything is governed by the FSA—and, indeed, the FSA won’t be around for too much longer—but the principle is sound:

Ticking a box that says someone has read something doesn’t mean they’ve read it.

So how about not putting that pointless box there next time?

It seems that I’ve created the Muhammad cartoon of the programming world. Oops. I never intended it that way.

Six years ago, I was programming in Ruby. There weren’t so many of us back then. I spent a lot of time in the #ruby-lang IRC channel learning, teaching, and chatting.

One day, someone found a humorous Python propaganda poster that someone had made by adding captions to an old Chinese propaganda poster; captions like:

Don’t use Perl; it’s counter revolutionary

We all found this fairly entertaining. Around this time, some North Korean propaganda posters had been going around the media, and one of them struck me as a perfect base from which to work.

And this is what I came up with. It’s not subtle:

USE RUBY! CRUSH THE PYTHON!

But in the small IRC community, and in the context of the friendly rivalry with the Python community, it was well received. People asked me for different size versions so that they could use it as a desktop image. It was just a bit of fun, really.

However, …

Yesterday, someone rediscovered it and it started making its way around Twitter. Peter Cooper posted it (with my permission!) on Ruby Inside.

Oh dear. Some people really aren’t happy with it. I’ve had comments and emails complaining about it. I’ve been told that I should ‘remove it’. Well, it’s not going anywhere.

It’s old. It’s not trying to stir up a language war. There never was a language war. It was just an in-joke, really. From 2004.

I’d say I’m sorry for offending people, but, really, I’m not sorry at all. I just wish that people weren’t so easily offended. And that they could read dates.

The protocol used to enable secure shopping and banking on the internet can also be used to restrict your freedom to choose the hardware and software you use to connect. In fact, it’s already happening.

TLS allows a client (i.e. a web browser) to identify a server and to protect data in transit between the two ends of the connection. Most people are familiar with this in the form of HTTPS web sites; if the server isn’t what it purports to be—or even if it’s just poorly configured—you’ll see an error message.

What fewer people know is that TLS also works the other way round: it can identify the client to the server by means of a certificate installed on the client. This is mainly used by large organisations to reduce the risk of unauthorised computers connecting to their back-end systems.

And that’s all fine. Everyone is safe and secure, right? Unfortunately, there’s a dark side to TLS as well, but it’s one that I hadn’t realised until now.

I won’t go into great details about how public key certificates work. All you need to know is that it’s possible to identify the issuer of a certificate, and it’s effectively impossible to fake a certificate.

So here’s the problem: if a manufacturer supplies a device with their own client certificate already installed, it’s possible for a server to practice selective discrimination based on the manufacturer of the device. It might, for example, decide only to serve devices manufactured by Sony or Apple.

Far fetched?

Alas not. This is, in fact, used by the BBC today to implement the version of the iPlayer used by the Sony PS3 and the Apple iPad. The BBC server at securegate.iplayer.bbc.co.uk accepts only the following certificate authorities:

• Oregan BBC CA
• BBC Greenhouse Development Staff CA
• BBC Greenhouse Production Servers and Services CA
• Apple iPhone Device CA
• ADB Root CA – DTT
• BBC Greenhouse Production Operations CA
• Sony LFX Project BIVL – Root CA

Don’t believe me? Try it in your browser. (You’ll need a protocol analyser to find the list above.)

This won’t necessarily bother you unless you’re trying to reverse-engineer the iPlayer. However, as a proof of concept of something much more sinister, it’s alarming. We’ve already seen the obsequious level of favouritism given to Apple by the BBC, and they’re not even a commercial organisation.

I imagined that it would be governments that would take away internet freedoms. Between IP geolocation restricted services (taking the world out of world-wide web) and device-locked services, however, I wonder if corporations will get there first.

I don’t know what we can do about it. It’s particularly galling that the BBC is working to destroy the notion of a device-agnostic, protocol-based internet, and using free software to do so.

Anyway, there’s a prize for the first person to extract a working client certificate from an iPad.

Up until now, I could have believed that the BBC’s iPlayer platform strategy was plausibly based on customer reach. Thus, they targeted Windows first, then OS X and Linux and any platform that Adobe Flash supported, then the iPhone and a random smattering of other mobile devices.

But what about Android?

crickets

No Android, despite the fact that quite a lot of Android handsets have now been sold, and they’re all capable of playing video. The BBC did manage to make an iPad version before a single iPad had been sold in the country. But still no Android.

— Did you just tell me to go fuck myself?
— I believe I did, Bob.

(From here)

What a load of bollocks. The strategy isn’t based on anything measurable. It’s based on nothing more than the narcissistic gadget purchasing habits of the BBC executives.

Well, they’re a bunch of idiots. As I’ve said before, there’s nothing stopping the iPhone version of the iPlayer working on Android phones except for the fact that the BBC have specifically rigged it only to respond to devices manufactured by Apple, Inc.

It’s just regular HTTP with H.264 content. Except for the referrer/behaviour checks. If they’re operating under some misapprehension that the closed Apple platform prevents people from repurposing the content, then they must have been asleep for a couple of years.

The BBC say:

We’re currently waiting on Flash support on Android to deliver a service for that platform.

$DEITY wept. There’s no need to wait. Flash won’t save them (cough RTMPDump cough). Flash won’t run on most of the Android phones currently out there. H.264 will, and it’s practically there today, except for the boneheaded platform strategy.

Ah well. Their intransigence and disrespect gives me all the motivation I need to keep fighting.

Earlier this year, on 16th March 2010, Greenland Pier was precipitately and unexpectedly closed to passengers, inconveniencing Rotherhithe residents who rely on the Thames Clipper service to get to work.

The decision to close the pier was taken by Southwark Council, who are responsible for the upkeep of the pier. It was apparently taken without any advance warning to Thames Clippers. Southwark asserted that the closure was necessitated by collision damage to the pier. However, the documents I’ve received in response to a Freedom of Information request strongly suggest that might be a cover story to distract attention from poor maintenance of the pier.

The broad outline of events is as follows:

• 25th November 2009: Thames Clippers write to Southwark Council enquiring about the status of pin replacement work and recommending the replacement of roller horns on the pier. There is no reply.
• Various emails pass back and forth about gritting and lighting.
• 12th March 2010: Thames Clippers again write to Southwark about the maintenance of Greenland pier, noting the lack of response to the earlier letter, and stating that the situation has deteriorated.
• 16th March: Southwark send contractor to inspect the pier. The pier is determined to be too dangerous for public use, and is to be closed from 20:30 that day.
• Meanwhile, Southwark appear to be briefing that the closure is the result of collision damage.
• 17th March: An engineer visits the pier to inspect the situation and recommend remedies.
• 18th March: Repairs are completed, and the pier reopens.

Southwark’s public version of events doesn’t quite tally with the documents I’ve received. The letter from Thames Clippers to the council on 25th November 2009 makes it clear that the problem had been present for some time:

I show concern that the roller horns that secure the pier into the dolphin guides upstream and downstream are now non-existent after many years of service. To my knowledge these have never been replaced and should certainly be done so at least once every ten years, and given the age of the pier now the replacements are well overdue and this system is currently totally ineffective. [p. 10]

The follow-up message on 12th March shows that the wheels were already missing by this point:

I have attached some photographs taken on 11th March 2010 which clearly show significant deterioration and the need of immediate attention with regards to the 2 pier dolphin guides, especially the downstream end as seen in figure GR3 (it is important to note that should this solid stainless steel spindle that the wheels were once attached fail, the pier would float away from its position). [p. 24]

However, Southwark claim not to have received the attached images, and cast some aspersions on the method by which the letter was delivered:

11. 12.03.10 – Email of letter from —— to —— which was sent at 17:39 Friday evening. —— telephoned —— several hours earlier on Friday afternoon to inform him that he would be sending a letter. —— repeated that he hoped this letter would “assist me —— in getting the funding for these works from the council”. —— told —— that the best way for him to get the funding needed would be if T.C. paid its arrears and assumed responsibility for the R&M of the pier. At no time during the conversation did —— mention the damage to the pontoon locating system, not the fact that the pier was in a dangerous condition. As —— was on leave on the Monday, he did not receive the letter until Tuesday 16.03.100, and despite what was stated in the letter to —— and also in ——’s letter to the Mayor’s office, there were no attachments sent with the letter. On 18 March, —— asked —— in person why the photographs had not been sent. —— claimed they had been sent by special delivery and signed for by someone in C Magro’s office. This was false and in fact the letter with photos were not sent until 19 Mar. In addition they were sent to the Council’s PO box address and not the marina, and were therefore not received by —— until 25 Mar. [p. 2]

Nonetheless, it appears to be this letter that spurred Southwark into action and resulted in the closure of the pier for repairs a few days later. And yet, the Mayor’s Transport Advisor seems to have been given a contradictory view of the situation, judging by a letter of 17th March:

Southwark have explained to me that the damage to the dolphins at the pier was caused not through lack of maintenance (the pier is inspected monthly) but was almost certainly due to a collision – hence the lack of warning over the closure. [p. 30]

On the same day, Thames Clippers denied this version of events, reiterating the lack of maintenance argument:

To be clear Thames Clippers have absolutely not been involved in any recent collisions with Greenland Pier. We have been concerned with the general maintenance of Greenland Pier for some time and have voiced our concerns to Southwark Council on several occasions (see attached supporting documentation). [p. 31]

Southwark’s Strategic Director of Environment and Housing stuck to the council line:

The damage our inspectors have identified is without a doubt the result of severe impact/collision from the river side and nothing whatsoever to do with wear and tear. [p. 32]

Meanwhile, the engineer who visited the pier on 17th March mentioned nothing about a collision. This is confusing, though:

The 4No Rollers which should engage between two vertical 400×200 RHS Guides were missing. We understand that they fell off yesterday. [p. 40]

Yet we know that the wheels were already missing by 11th March.

There is a very different version of the story from each side. I find Thames Clippers’ take to be better supported by the correspondence, although there are still questions about the manner in which Thames Clippers raised their concerns with the council. You can read the whole saga in PDF form if you want to see all the details.

On a lighter note, there are some entertaining bits among the documents; the back-and-forth between the Managing Director of Thames Clippers and the Harbour Master is amusingly bad-tempered:

I would suggest that you keep your comments and your opinions to yourself … [p. 37]

I very much welcome your sudden interest in the safe operation of the pier. [p. 54]

Finally, there appears to be a desire by the council to divest itself of the responsibility for the pier:

Email letter from —— to —— re: R&M work needed on pier. —— had telephoned previously to inform —— that he would be sending the letter, hoping that it would “assist me —— in getting the funding for these works from the council”. —— repeated to —— what had been discussed at their meeting in February, that the responsibility for the R&M should be assumed by T.C. and that if the pier became unsafe in the future it would be closed. [p. 1]

And it was closed, although funding was obtained and the pier repaired and reopened. But what will happen next time? Is Greenland Pier on borrowed time?

I booked a flight with EasyJet a few months ago. I created an account, and, as is my habit, used an automatically generated password consisting of words and punctuation.

When I went back later to log into EasyJet’s website to supply the passport details for the flight, however, my credentials were rejected. No problem; there’s a ‘forgotten password’ link. I followed it, and my password was emailed to my account. This was a problem on two levels:

1. Storing passwords (rather than salted hashes) is bad practice.
2. It was the same password that I had typed in and which had been rejected.

I still couldn’t log in.

I went through the customer service wizard and found the closest matching (mandatory) heading under which to submit my problem: ‘I have forgotten my password.’ Here’s what I wrote:

Actually, I haven’t forgotten my password. It just doesn’t work.

I asked for my password to be emailed to me; it was correct. However, I still can’t log in, even using the password emailed to me: I just see “Sorry, but your login details are invalid.”

I also did a bit of searching and found that I wasn’t alone. The website is broken, but someone else had (rather cleverly) worked out how to get around the problem:

When you enter the password, replace the special character with %HEX_Key. It should work after that (at least for me). Change your password to one without special characters if you’d like after logging in.

Easy. I looked up the relevant ASCII codes and typed in my password, then changed it to something without any punctuation. Everything was fine.

A month passed

Today, at last, EasyJet deigned to reply to my original issue. (I suppose I’m lucky it wasn’t urgent!) Unfortunately, their muppetry in web development is not an outlier: it seems to be an organisation-wide issue. The reply is a bizarre non sequitur:

Dear – Battley,

Thank you for contacting us.

I am sorry that you have been inconvenienced by the cancellation of your flight and regret that I was unable to respond sooner to process your refund. However, I note from your booking that this has already been done by my colleague.

Please note that due to the high volume of disruptions we experience significant delays in responding. Although we try our best to get back to our customers as soon as possible, we would advise that any urgent queries should be made by telephone on 0871 244 2366 (calls cost 10p per minute; calls from mobiles and other networks may cost more). However, be aware that there can be longer waiting times on the phones.

I do hope I have been able to answer your question fully, if I have not, please click here and we will be more than happy to assist you further.

Yours sincerely

Peter Tate
Customer Experience Champion

I can only assume that they pasted a response they’d earlier sent to someone who couldn’t log in to deal with a cancelled flight.

What a bunch of muppets. I hope they do a better job of flying aeroplanes than they do of websites and customer service.

There’s an interesting site called Openbook that allows you to search public Facebook status updates. (The deliberate complexity of Facebook’s privacy settings means that many of the people posting them probably don’t realise that they’re public, of course, but that’s a digression.)

If you search for not racist on there, you get a lot of results that look like this:

POLICE ARE GOING AROUND PUBS AND CLUBS SAYING THAT WE CANT WEAR OUR ENGLAND TOPS 4 THE FOOTIE & GOTTA TAKE THE FLAGS DOWN AS IT IS UPSETTING THE PEOPLE THAT DONT COME FROM HERE !!NOW IM NOT RACIST , BUT THIS IS TAKING THE PISS! THIS IS OUR COUNTRY AND WE NEED 2 MAKE A STAND IF YOU/THEY DONT LIKE IT GO AWAY! WOULD YOU REMOVE UR TURBAN & BURKHA BECAUSE IT UPSETS ME, IF YOU AGREE POST THIS AS YOUR STATUS

1. No they aren’t.
2. ‘Im [sic] not racist’. Yes, you are.
3. You’re an idiot. (See previous point.)

It’s like the Protocols of the Elders of Zion for semi-literate credulous morons, and yet more proof that punctuation has a liberal bias.