BBC iPlayer countermeasures prove futile yet again

I left a conundrum behind last week, shortly before I left for a long weekend in Barcelona. (It was very nice, by the way, thank you.) I’m delighted to see that others have been hard at work in my absence, defeating the Beeb’s latest iPlayer countermeasure.

And what a weak countermeasure it is! XORing the stream with a two-byte repeating pattern. Really, it’s almost beneath them. This comment by No. 7 on the comment thread made me laugh:

...and to the BBC iplayer department, the secret is to bang the rocks together, guys.

I don’t yet understand how the iPhone picks up the two-byte key for this XOR scheme, or what kind of inside knowledge was needed to discover the capability, but I hear that the BBC have managed to break iPod Touch functionality in the process. If that’s true, it’s a spectacular own goal, and a complete waste of everyone’s time.

The power of random, self-organising groups, co-ordinating over the internet, is huge. As fast as the small team at the BBC comes up with ways to confound us, we’re working around them. They just can’t keep up.

I’ve applied the patches to the command-line downloader: it’s available via subversion as usual, or as a tarball. I’ll rebuild the Windows client in a day or two.

2008-06-10 22:12 UTC. Comments: 24.

Comments

  1. tonightwefly

    Wrote at 2008-06-10 23:03 UTC using Firefox 2.0.0.14 on Mac OS X:

    The real iPlayer site still works fine on my iPod Touch (non-jailbroken 1.1.4) – I just played a couple of movies tonight and they streamed no problem.

  2. Irregular Shed

    Wrote at 2008-06-11 09:37 UTC using Firefox 2.0.0.14 on Windows XP:

    I was wondering what the source of the bytes was myself. In ASCII it’s S, then a less-than symbol that your blog won’t let me encode in any way… but yes, it’s spectacularly similar to the fabled ROT13 encryption. I’d really like to know how the iPhone knew and expected the XORing as well – there has to be, somewhere, something that lets it know what’s going on. If we can locate that, we can find the next flowerpot that the key is left underneath.

    What we need here is an insider! Maybe a Freedom of Information Act request as to how much they’ve urinated up the wall achieving the Geeks 3 BBC 0 score as well.

  3. No. 7

    Wrote at 2008-06-11 10:46 UTC using Safari 525.18 on Mac OS X:

    Just to clarify…
    On a factory-fresh iPod touch (where program_a is the name of a video from iPlayer):

    program_a accessed on genuine iPlayer site: Works
    program_a.mov downloaded and dropped into iTunes: Fail
    program_a.mov downloaded and accessed by Safari on iPod via my local web-server: Fail
    program_a.mov downloaded, /cleaned/ and accessed by Safari on iPod via my local web-server: Works

    The genuine iPlayer site works on the iPod, but the iPod doesn’t like the /downloaded/ file. My suspicion is that they have found another way to distinguish between iPod/iPhone and the downloading scripts. But why, if that is the case, would they send only very weakly encrypted files rather than a large file full of random noise? Are they messing with our heads? Or are they just “going through the motions” to appease the content owners?

  4. Daniel Hardy

    Wrote at 2008-06-11 11:12 UTC using Firefox 3.0 on Mac OS X:

    This might sound stupid, but is there anyway it wasn’t intentional? An accident?

    Or perhaps a junior member of the team has injected some code in unsupervised or something, trying to be clever.

  5. kzap

    Wrote at 2008-06-11 11:25 UTC using Firefox 2.0.0.6 on Linux:

    well done I’m re-downloading Doctor Who now I hope it works.
    I had to watch it online last week, horrible quality on my linux box.
    BTW I find it best to play the videos in VLC with post processing on 5.

  6. Paul Battley

    Wrote at 2008-06-11 12:37 UTC using Firefox 3.0 on Linux:

    Based on the fact that the encrypted movie won’t play on the iPod when it’s embedded in a different page, my suspicion is that there’s a decryption key that gets passed along to the client.

    There is a mechanism for Access Keys in QuickTime, so that’s a possibility.

    I also found an intriguing forum thread, in which a coder in Ukraine stated earlier this year that they ‘had a task to port a DRM codec from Windows to Mac OS X. This codec isn’t a complicated one it just decrypts a file using a key and XOR function.’

  7. kzap

    Wrote at 2008-06-11 13:32 UTC using Firefox 2.0.0.6 on Linux:

    IT’S WORKS!!!!
    just thought I would shout it out.
    Now I have people looking at me strangely in the school library.

  8. kzap

    Wrote at 2008-06-11 13:42 UTC using Firefox 2.0.0.6 on Linux:

    Just an idea could you easly re-write it so you could download more than one program at a time so you would put iplayer-dl [b0074ftl, b00c3vb9, b00c191w]
    just a thought I’m having to catch up on my program downloading.
    See if you can find out what I have been watching.

  9. Strawp

    Wrote at 2008-06-11 13:53 UTC using Firefox 3.0 on Windows XP:

    I’m going to have a look through a packet dump of the entire thing in Wireshark later. I don’t think the key is in the header of the video itself so it must be getting it by some other method.

    It’s possible of course that these scripts won’t even see the key and a traffic dump from a real iPod would be needed.

  10. No. 7

    Wrote at 2008-06-11 13:58 UTC using Safari 525.18 on Mac OS X:

    Is it possible that this is some kind of XOR based Forward Error Correction system to improve the stability of the A/V stream?

  11. Math Campbell-Sturgess

    Wrote at 2008-06-11 14:50 UTC using Firefox 3.0 on Mac OS X:

    Now it’s working again, I suppose I should try and get a new version of my mac GUI out…

    Just to be dumb again, how can I use your script in my app without installing it??
    Or, if installing is needed, can the install itself be done by manually placing files, rather than by running the ruby installer?

    Also, what would be a good way of integrating better proxy support?
    I was thinking that since a lot of people might want to use a proxy for the downloader (but NOT for their regular web-browsing) then maybe an option to specify a proxy for the app would be good.
    I was going to add it in via an additional command-line argument on your script, which I can then factor in to my app (with preferences that remember last used proxy etc.) but what would be your take on it Paul?

    I must say, I know the refactoring you did was needed to get things much more slimmed down, but it does make reading the script (when you don’t know Ruby :) ) a bit harder.
    Maybe I should learn more Ruby and stuff. I know Cocoa fully supports Ruby to be used as the main language, rather than just running a script as a task, like I am (because I’m lazy and stupid!), so maybe that’d be a better avenue for me.
    Thanks anyway for getting it working again! Glad you enjoyed your trip!

  12. Mike Trim

    Wrote at 2008-06-11 18:04 UTC using Firefox 3.0 on Linux:

    Math: Paul’s script already includes proxy support, just pass the—http-proxy=HOST:PORT option on the command line.

    You can run the script (in the iplayer-dl/bin/ directory) without installing it, but you will need the iplayer-dl/lib/ directory to be in your Ruby include path. An easy way to do this is to add the following line before the require statements at the top of iplayer-dl:

    $:.unshift(File.join(File.dirname(FILE), ’..’, ‘lib’))
    (Note FILE above is surrounded by double underscores which get eaten by the blog (as did the left shift operator I was using instead of unshift))

    I think there is a way to specify that as an argument to ruby on the command line too.

  13. Math Campbell-Sturgess

    Wrote at 2008-06-11 18:51 UTC using Firefox 3.0 on Mac OS X:

    See, I thought the http_proxy line was taking in the environment’s proxy.
    My bad.

    Now I have to work out how to pass the arguments needed and the files and whatnot in an NSTask. I don’t run the ruby from a terminal, but using a programmatically defined NSTask, meaning I can assign buttons and sliders and whatnot to it.
    I am seriously considering learning enough Ruby and Ruby-Cocoa so as to integrate the ruby script into a proper gui, not just a gui-based task wrapper.

    Saying that, I don’t even know where to start. Programming is haaard!!!

  14. Richard

    Wrote at 2008-06-11 20:14 UTC using Firefox 2.0.0.6 on Linux:

    It’d be interesting to look at the file in terms of the actual atoms/records in the movie. If the key is in the file, it’s perhaps in a “reserved” field that should equal zero according to specification, and does equal zero once the XOR is removed?

  15. James Raybould

    Wrote at 2008-06-11 21:45 UTC using Safari 525.20 on Mac OS X:

    I don’t suppose it would be possible to have the XOR’ing be performed after the file has finished downloading? Surely it would enable the file to be downloaded as fast as possible then enable the XOR’ing to happen all at one and make that faster as well?

  16. Richard

    Wrote at 2008-06-11 22:24 UTC using Firefox 2.0.0.6 on Linux:

    Hi James—interesting question, but I think it’s transferring the large video file over the internet, or from disk into memory and back that takes the most time.

    Doing the XOR as data is arriving allows the program make only one write to disk, whereas if it made the download and the XOR as separate stages, the video data potentially makes three journeys between disk and memory.

  17. James Raybould

    Wrote at 2008-06-11 22:28 UTC using Safari 525.20 on Mac OS X:

    I’m not sure – I was getting download speeds of around 800kbs before the XOR’ing and now I’m getting speeds of around 250kbs and this is running from the terminal in OSX on a 2.2GHZ Macbook Pro so there is plenty of horsepower there… Has anyone else compared the downloading times to older versions? I mean its possible I just keep grabbing programs when the going is slow from the BBC servers

  18. Paul Battley

    Wrote at 2008-06-11 22:33 UTC using Firefox 3.0b5 on Linux:

    Ruby really suffers doing the XOR a byte at a time, and pegs the CPU at 100%. Thanks, BBC, for wasting our time and heating up the planet!

    I’m planning to look into the binary extension options to get it working faster—it really should be a quick operation.

  19. Nick

    Wrote at 2008-06-11 22:47 UTC using Firefox 3.0 on Linux:

    Looking great – thanks.

    On my ubuntu box I had to do a little fiddling to get the gui version to work. In case anyone else needs help:

    1. Install rubygems from synaptic
    2. Run “sudo gems update”
    3. Run “sudo gem install wxruby”
    4. Install the iplayer-dl package as per README
    5. Open /usr/local/lib/site_ruby/1.8/iplayer/gui/main_frame.rb in a text editor and comment out line 32 (the icon can’t be found – Paul: can you maybe put in a test as it wasn’t here for me)
    6. Run it using “ruby -rubygems /usr/bin/iplayer-dl-gui”

    Thanks again Paul.

  20. P Lewis

    Wrote at 2008-06-11 23:07 UTC using Firefox 3.0 on Linux:

    @Paul – Have you tried performing the XOR on several kbytes at a time in your ruby script? I do that on the iplayer_decode perl script and it makes one hell of a difference :-) In fact I just bumped the buffer up to 4MB and it now decodes in ‘superfast’ time.

  21. Paul Battley

    Wrote at 2008-06-11 23:29 UTC using Firefox 3.0b5 on Linux:

    That’s a good suggestion, but, unfortunately, there isn’t a stringwise XOR function in Ruby.

  22. Richard

    Wrote at 2008-06-12 02:09 UTC using Firefox 2.0.0.6 on Linux:

    Hi Paul,

    in downloader.rb line 74:
    buffer=’’
    on downloader.rb, line 82:
    buffer pipes-into d.char

    does this mean that buffer is growing one byte at a time, and internally making lots of copies of itself? eg. perhaps ruby only allocates a 256 byte area of memory to hold buffer, and then when you pipe into it, it has to copy and grow, and it next it picks a 512 byte region, and more data arrives so it then copies and grows again…

    if you preallocate

    buffer=(some array of space that’s contiguous and zeroed out)
    i=0

    and then set buffer[i]=d.char
    i++

    in place of the piping, does that perform any better? I’ve seen some MATLAB scripts that were totally crippled by the poor behaviour of the “append to string” function, but which worked a dream when the string was pre-allocated and then written into, perhaps it’s the same issue (perhaps not?)

  23. Paul Battley

    Wrote at 2008-06-12 07:43 UTC using Firefox 3.0b5 on Linux:

    Richard, I think you’re right: that probably wasn’t optimal.

    I’ve improved the XORing in Ruby. I think it’s now about as good as it can get without a binary extension.

  24. Paul Battley

    Wrote at 2008-06-12 13:40 UTC using Firefox 3.0 on Mac OS X:

    Comments are now closed: please visit the new project page for more information and future developments.