I’m an evil iPlayer hacker
Paul Battley
I’m not really evil
But I am a hacker
But I am a hacker
(BBC News said so)
Cat & mouse
A brief history
27 July 2007
- Open Beta
- Custom client
- Windows XP
- DRM
- Time-bombed (30 days)
- P2P (Kontiki)
13 December 2007
- Flash streaming version
- Windows, OS X, (x86) Linux (kinda)
- 7 day window
- Still proprietary
Interlude
7 March 2008
- iPhone (& iPod Touch) version
- MPEG4
- HTTP
7 March 2008
- iPhone (& iPod Touch) version
- MPEG4
- HTTP
- No DRM
- User agent detection
a few hours later …
one week later …
The Empire Strikes Back
13 March 2008
That lunchtime
- One iPod Touch
That lunchtime
- One iPod Touch
- One debugging proxy (Charles)
That lunchtime
- One iPod Touch
- One debugging proxy (Charles)
- 30 minutes over lunch
- CoreMedia user agent for video
- Extra ‘Range’ header
The same evening
The next morning
alles ruhig
10 June 2008
- Downloaded files stop working
10 June 2008
- Downloaded files stop working
- Compare before and after
10 June 2008
- Downloaded files stop working
- Compare before and after
- Partially XORed
10 June 2008
- Downloaded files stop working
- Compare before and after
- Partially XORed
- Same 2-byte sequence
XOR_KEYS = [0x3c, 0x53] XOR_START = 0x2800 XOR_END_OFFSET = 0x400
bytes = data.unpack('C*') if bytes_got >= XOR_START && (bytes_got + data.length) < xor_end bytes.each_with_index do |d, i| offset = bytes_got + i bytes[i] = d ^ XOR_KEYS[(offset-XOR_START) & 1] end else bytes.each_with_index do |d, i| offset = bytes_got + i if (offset >= XOR_START) && (offset < xor_end-2) d ^= XOR_KEYS[(offset-XOR_START) & 1] elsif (offset >= xor_end-2) && (offset < xor_end) d ^= XOR_KEYS[(xor_end-offset+1) & 1] end bytes[i] = d end end data = bytes.pack('C*')
In Perl:
$plain = $cipher ^ $key
- Different 2-byte sequences
- Different 2-byte sequences
- 4-byte sequences
18 June 2008
- XOR was a red herring
- Wireshark
18 June 2008
- XOR was a red herring
- Wireshark
- Bug image ‘blesses’ cookie
Clever
Clever
(but futile)
27 June 2008
- Version 2 of iPlayer
- New URLs
27 June 2008
- Version 2 of iPlayer
- New URLs
- No new countermeasures
23 September 2008
- Radio added
- MP3
- HTTP
23 September 2008
- Radio added
- MP3
- HTTP
- Almost no code changes
23 September 2008
- Radio added
- MP3
- HTTP
- Almost no code changes
- Just a different filename when saving
Pretending to be an iPhone
- Headers
- Cookies
- Two user agents
IPHONE_UA = 'Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1A543a Safari/419.3' QT_UA = 'Apple iPhone v1.1.4 CoreMedia v1.0.0.4A102' DEFAULT_HEADERS = { 'Accept' => '*/*', 'Accept-Language' => 'en', 'Accept-Encoding' => 'gzip, deflate', 'Connection' => 'keep-alive', 'Pragma' => 'no-cache' }
url = URI.parse(location) http = Net::HTTP.new(url.host, url.port) if url.query path << '?' << url.query end response = http.request_get(path, headers, &blk) headers['Cookie'] = response.cookies.join('; ')
Parsing JavaScript
- BBC programmes are identified by a PID
- E.g. ‘b00b3zjr’
- Each version also has a PID
- Set via JavaScript
# Poor man's JavaScript parser to evaluate the multiple versions begin versions = html[/ iplayer\.versions \s* = \s* \[ .*? \]; /mx].split(/filesize/) pid = html[/ iplayer\.pid \s* = \s* '([a-z0-9]{8})' /x, 1] rescue => e $stderr.puts "The BBC iPlayer appears to have changed.", "Please look for an updated version of this script." exit 1 end # Find out which versions are available right now available_versions = versions.inject({}){ |acc, e| begin type = e[/ type \s* : \s* ' ( [^']+ ) ' /x, 1] vpid = e[/ pid \s* : \s* ' ( [^']+ ) ' /x, 1] mp4 = e[/ iplayer_streaming_http_mp4 \s* : \s* \[ ( [^\]]+ ) /x, 1] mp4_starts = mp4.scan(/ start \s* : \s* new \s+ Date\( ( [^\)]+ ) /x) mp4_ends = mp4.scan(/ end \s* : \s* new \s+ Date\( ( [^\)]+ ) /x) now = DateTime.now mp4_starts.zip(mp4_ends).each do |a, b| if (now >= js_date(a[0]) && now <= js_date(b[0])) acc[type] = vpid end end rescue end acc }
ugh.
[{
type : 'Original',
pid : 'b00b09pv',
iplayer_broadband_streaming : [{
start : new Date(2008, 3, 22, 22, 58, 28),
end : new Date(2008, 3, 29, 22, 19, 00)
}]
}]
Tokenisation
TOKENS = [ [ %| '([^']+)' |, :string ], [ %| "([^"]+)" |, :string ], [ %| new \\s+ Date \\s* \\( |, :date ], [ %| ([_a-zA-Z][_a-zA-Z0-9]*) |, :identifier ], [ %| ([0-9]+) |, :integer ], [ %| \\[ |, :array ], [ %| \\] |, :close_bracket ], [ %| \\{ |, :hash ], [ %| \\} |, :close_brace ], [ %| : |, :colon ], [ %| , |, :comma ], [ %| \\. |, :period ], [ %| \\) |, :close_parens ], [ %| ; |, :semicolon ], ]
def tokenize array = [] tail = @source while tail token, data, tail = next_token(tail) if token array << [token, data].compact end end array end
def next_token(s) TOKENS.each do |pattern, name| if m = s.match( Regexp.new( "\\A \\s* #{pattern} (.*)", Regexp::EXTENDED | Regexp::MULTILINE ) ) if m[2] # if using a capturing group return [name, m[1], m[2]] else return [name, nil, m[1]] end end end return nil end
[ [:array], [:hash], [:identifier, "type"], [:colon], [:string, "Original"], [:comma], [:identifier, "pid"], [:colon], [:string, "b00b09pv"], [:comma], [:identifier, "iplayer_broadband_streaming"], [:colon], [:array], [:hash], [:identifier, "start"], [:colon], [:date], [:integer, "2008"], [:comma], [:integer, "3"], [:comma], [:integer, "22"], [:comma], [:integer, "22"], [:comma], [:integer, "58"], [:comma], [:integer, "28"], [:close_parens], [:comma], [:identifier, "end"], [:colon], [:date], [:integer, "2008"], [:comma], [:integer, "3"], [:comma], [:integer, "29"], [:comma], [:integer, "22"], [:comma], [:integer, "19"], [:comma], [:integer, "00"], [:close_parens], [:close_brace], [:close_bracket], [:close_brace], [:close_bracket] ]
Recursive parser
[{ :type => "Original", :pid => "b00b09pv", :iplayer_broadband_streaming => [{ :start => DateTime.civil(2008, 4, 22, 22, 58, 28), :end => DateTime.civil(2008, 4, 29, 22, 19, 00) }] }]
Evolving a hacky script
- Wrapper around curl: ~50 lines
- Wrapper around curl: ~50 lines
- Using Net::HTTP: ~100 lines
- Wrapper around curl: ~50 lines
- Using Net::HTTP: ~100 lines
- Added features: ~180 lines
not too bad
- JavaScript parser
- GUI consumers
- Maintainability
Time to refactor
- Human interface
- Download process
- iPhone behaviour level
- JavaScript parsing
- Errors
- Programme metadata (titles)
- Thin CLI
- Facilitated other interfaces
- Thin CLI
- Facilitated other interfaces
- (e.g. GUI)
- More complicated.
- Most GUIs wrap CLI
Reverse-engineering RTMP
- Flash player (& server)
- Streaming audio/video/data
- Proprietary
HTTP downloads are not possible: we don’t own most of the content. That’s why you’ve spotted RTMP being used – it’s a form of non-invasive Content Restriction And Protection. I’m sorry we have to use it. But we have to use it. —James Cridland
Partial open-source implementations
- Red5
- RubyIZUMI
- Gnash
- RTMPy
- Stream carries RTMP packets
- Stream carries RTMP packets
- Which carry AMF packets
- Stream carries RTMP packets
- Which carry AMF packets
- or raw audio/video data
- Tries to be efficient
- Tries to be efficient
- Re-using headers
- Tries to be efficient
- Re-using headers
- Compact headers
Implementing RTMP
- Bottom up
- RTMP first
- Tests based on publicly-available specs
- Captured real conversations
- Tried to decode packets
- Pencil & paper
- Captured real conversations
- Tried to decode packets
- Pencil & paper
- Rewrote tests
- Captured real conversations
- Tried to decode packets
- Pencil & paper
- Rewrote tests
- Round-trip decode & re-encode
def get_data header = Packet::Header.new header.parse(@socket) if packet = @packets[header.oid] packet.endow(header) else if previous_header = @headers[header.oid] header.inherit(previous_header) end packet = @packets[header.oid] = Packet.new(header) end @headers[header.oid] = header packet << @socket.read(packet.bytes_to_fetch) if packet.complete? @packets.delete(header.oid) yield packet end end
- Moved up to AMF
- Recursing parser (no tokenisation needed)
- Tests based on publicly-available specs
- Tried to decode real data
- Pencil & paper
- Rewrote tests
def recursive_parse(io) elements = [] until io.eof? || (e = next_element(io)) == EndOfPacket elements << e end elements end
def next_element(io) data_type = io.read(1).unpack('C')[0] case data_type when DT_NUMBER io.read(8).unpack('G')[0] when DT_BOOLEAN io.read(1).unpack('C')[0] != 0 when DT_OBJECT hash = {} until (key = read_length_prefixed_data(io)) == '' hash[key] = next_element(io) end hash when DT_STRING, DT_LONG_STRING read_length_prefixed_data(io) when DT_OBJECT_END EndOfPacket when DT_NULL_VALUE nil else read_length_prefixed_data(io) end end
["connect", 1.0, {"capabilities"=>15.0, "videoFunction"=>1.0, "audioCodecs"=>1639.0, "app"=> "ondemand?_fcs_vhost=cp48184.edgefcs.net&auth=daEcIaKaQdfaic ZcBa_aLa9dYbhdCaCc3d9-bizwQB-cCp-FnrDCqBnNDoGuwF&aifp=v001& slist=secure/6music/AMI_e6d01bf639fe37be3a42e423f9f38425_b0 0c73d2_6m_lamacq_thu", "videoCodecs"=>252.0, "swfUrl"=>"http://www.bbc.co.uk/emp/player.swf?revision=3704", "pageUrl"=>"http://www.bbc.co.uk/iplayerbeta/episode/b00c73fc", "tcUrl"=> "rtmp://84.53.177.140:1935/ondemand?_fcs_vhost=cp48184.edgef cs.net&auth=daEcIaKaQdfaicZcBa_aLa9dYbhdCaCc3d9-bizwQB-cCp- FnrDCqBnNDoGuwF&aifp=v001&slist=secure/6music/AMI_e6d01bf63 9fe37be3a42e423f9f38425_b00c73d2_6m_lamacq_thu", "fpad"=>false, "flashVer"=>"LNX 9,0,124,0"}]
Not finished
- Switch from AMF to media streaming
- Flash DRM
What I’ve learned
Releasing software is hard
Releasing Ruby software to users is hard
Releasing Ruby software to users is hard
- Gems?
- Tarballs?
- Exes?
GUIs are tedious
GUIs are tedious
except, apparently, on OS X
Reverse engineering can be fun
Reverse engineering can be fun
and enlightening
Obfuscation doesn’t work
Obfuscation doesn’t work
- Open source
Obfuscation doesn’t work
- Open source
- Public forum
Obfuscation doesn’t work
- Open source
- Public forum
- Big corps are slow
Feedback channel
Feedback channel
(Google forms are good)
Collaboration works