I’m an evil iPlayer hacker

Paul Battley

http://po-ru.com/

I’m not really evil

But I am a hacker

But I am a hacker

(BBC News said so)

Cat & mouse

A brief history

27 July 2007

  • Open Beta
  • Custom client
  • Windows XP
  • DRM
  • Time-bombed (30 days)
  • P2P (Kontiki)

13 December 2007

  • Flash streaming version
  • Windows, OS X, (x86) Linux (kinda)
  • 7 day window
  • Still proprietary

Interlude

7 March 2008

  • iPhone (& iPod Touch) version
  • MPEG4
  • HTTP

7 March 2008

  • iPhone (& iPod Touch) version
  • MPEG4
  • HTTP
  • No DRM
  • User agent detection

a few hours later …

one week later …

The Empire Strikes Back

13 March 2008

That lunchtime

  • One iPod Touch

That lunchtime

  • One iPod Touch
  • One debugging proxy (Charles)

That lunchtime

  • One iPod Touch
  • One debugging proxy (Charles)
  • 30 minutes over lunch
  • CoreMedia user agent for video
  • Extra ‘Range’ header

The same evening

The next morning

alles ruhig

10 June 2008

  • Downloaded files stop working

10 June 2008

  • Downloaded files stop working
  • Compare before and after

10 June 2008

  • Downloaded files stop working
  • Compare before and after
  • Partially XORed

10 June 2008

  • Downloaded files stop working
  • Compare before and after
  • Partially XORed
  • Same 2-byte sequence
XOR_KEYS       = [0x3c, 0x53]
XOR_START      = 0x2800
XOR_END_OFFSET = 0x400
bytes = data.unpack('C*')
if bytes_got >= XOR_START && (bytes_got + data.length) < xor_end
  bytes.each_with_index do |d, i|
    offset = bytes_got + i
    bytes[i] = d ^ XOR_KEYS[(offset-XOR_START) & 1]
  end
else
  bytes.each_with_index do |d, i|
    offset = bytes_got + i
    if (offset >= XOR_START) && (offset < xor_end-2)
      d ^= XOR_KEYS[(offset-XOR_START) & 1]
    elsif (offset >= xor_end-2) && (offset < xor_end)
      d ^= XOR_KEYS[(xor_end-offset+1) & 1]
    end
    bytes[i] = d
  end
end
data = bytes.pack('C*')

In Perl:

$plain = $cipher ^ $key
  • Different 2-byte sequences
  • Different 2-byte sequences
  • 4-byte sequences

18 June 2008

  • XOR was a red herring
  • Wireshark

18 June 2008

  • XOR was a red herring
  • Wireshark
  • Bug image ‘blesses’ cookie

Clever

Clever

(but futile)

27 June 2008

  • Version 2 of iPlayer
  • New URLs

27 June 2008

  • Version 2 of iPlayer
  • New URLs
  • No new countermeasures

23 September 2008

  • Radio added
  • MP3
  • HTTP

23 September 2008

  • Radio added
  • MP3
  • HTTP
  • Almost no code changes

23 September 2008

  • Radio added
  • MP3
  • HTTP
  • Almost no code changes
  • Just a different filename when saving

Pretending to be an iPhone

  • Headers
  • Cookies
  • Two user agents
IPHONE_UA = 'Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1A543a Safari/419.3'

QT_UA = 'Apple iPhone v1.1.4 CoreMedia v1.0.0.4A102'

DEFAULT_HEADERS = { 
  'Accept'          => '*/*',
  'Accept-Language' => 'en',
  'Accept-Encoding' => 'gzip, deflate',
  'Connection'      => 'keep-alive',
  'Pragma'          => 'no-cache'
}
url = URI.parse(location)
http = Net::HTTP.new(url.host, url.port)
if url.query
  path << '?' << url.query
end
response = http.request_get(path, headers, &blk)


headers['Cookie'] = response.cookies.join('; ')

Parsing JavaScript

  • BBC programmes are identified by a PID
  • E.g. ‘b00b3zjr’
  • Each version also has a PID
  • Set via JavaScript

# Poor man's JavaScript parser to evaluate the multiple versions
begin
  versions = html[/ iplayer\.versions \s* = \s* \[ .*? \]; /mx].split(/filesize/)
  pid      = html[/ iplayer\.pid \s* = \s* '([a-z0-9]{8})' /x, 1]
rescue => e
  $stderr.puts "The BBC iPlayer appears to have changed.",
               "Please look for an updated version of this script."
  exit 1
end
# Find out which versions are available right now
available_versions = versions.inject({}){ |acc, e|
  begin
    type = e[/ type \s* : \s* ' ( [^']+ ) ' /x, 1]
    vpid = e[/ pid  \s* : \s* ' ( [^']+ ) ' /x, 1]
    mp4  = e[/ iplayer_streaming_http_mp4 \s* : \s* \[ ( [^\]]+ ) /x, 1]
    mp4_starts = mp4.scan(/ start \s* : \s* new \s+ Date\( ( [^\)]+ ) /x)
    mp4_ends   = mp4.scan(/ end   \s* : \s* new \s+ Date\( ( [^\)]+ ) /x)
    now = DateTime.now
    mp4_starts.zip(mp4_ends).each do |a, b|
      if (now >= js_date(a[0]) && now <= js_date(b[0]))
        acc[type] = vpid
      end 
    end 
  rescue
  end 
  acc 
}

ugh.

[{
  type : 'Original', 
  pid  : 'b00b09pv',
  iplayer_broadband_streaming : [{
    start : new Date(2008, 3, 22, 22, 58, 28),
    end   : new Date(2008, 3, 29, 22, 19, 00)
  }]
}]

Tokenisation

TOKENS = [
  [ %| '([^']+)' |, :string ],
  [ %| "([^"]+)" |, :string ],
  [ %| new \\s+ Date \\s* \\( |, :date ],
  [ %| ([_a-zA-Z][_a-zA-Z0-9]*) |, :identifier ],
  [ %| ([0-9]+) |, :integer ],
  [ %| \\[ |, :array ],
  [ %| \\] |, :close_bracket ],
  [ %| \\{ |, :hash ],
  [ %| \\} |, :close_brace ],
  [ %| :   |, :colon ],
  [ %| ,   |, :comma ],
  [ %| \\. |, :period ],
  [ %| \\) |, :close_parens ],
  [ %| ;   |, :semicolon ],
]
def tokenize
  array = []
  tail = @source
  while tail
    token, data, tail = next_token(tail)
    if token
      array << [token, data].compact
    end
  end
  array
end
def next_token(s)
  TOKENS.each do |pattern, name|
    if m = s.match(
      Regexp.new( "\\A \\s* #{pattern} (.*)",
      Regexp::EXTENDED | Regexp::MULTILINE )
    )
      if m[2] # if using a capturing group
        return [name, m[1], m[2]]
      else
        return [name, nil,  m[1]]
      end
    end
  end
  return nil
end
[ [:array], [:hash], 
  [:identifier, "type"], [:colon], [:string, "Original"], [:comma], 
  [:identifier, "pid"], [:colon], [:string, "b00b09pv"], [:comma], 
  [:identifier, "iplayer_broadband_streaming"], [:colon], [:array], [:hash], 
    [:identifier, "start"], [:colon], 
      [:date], 
        [:integer, "2008"], [:comma], [:integer, "3"], [:comma], 
        [:integer, "22"], [:comma], [:integer, "22"], [:comma], 
        [:integer, "58"], [:comma], [:integer, "28"], 
      [:close_parens], [:comma], 
    [:identifier, "end"], [:colon], 
      [:date], 
        [:integer, "2008"], [:comma], [:integer, "3"], [:comma], 
        [:integer, "29"], [:comma], [:integer, "22"], [:comma], 
        [:integer, "19"], [:comma], [:integer, "00"], 
      [:close_parens], 
  [:close_brace], [:close_bracket], 
[:close_brace], [:close_bracket] ] 

Recursive parser

[{
  :type => "Original",
  :pid  => "b00b09pv",
  :iplayer_broadband_streaming => [{ 
    :start => DateTime.civil(2008, 4, 22, 22, 58, 28), 
    :end   => DateTime.civil(2008, 4, 29, 22, 19, 00) 
  }]
}]

Evolving a hacky script

  • Wrapper around curl: ~50 lines
  • Wrapper around curl: ~50 lines
  • Using Net::HTTP: ~100 lines
  • Wrapper around curl: ~50 lines
  • Using Net::HTTP: ~100 lines
  • Added features: ~180 lines

not too bad

  • JavaScript parser
  • GUI consumers
  • Maintainability

Time to refactor

  • Human interface
  • Download process
  • iPhone behaviour level
  • JavaScript parsing
  • Errors
  • Programme metadata (titles)
  • Thin CLI
  • Facilitated other interfaces
  • Thin CLI
  • Facilitated other interfaces
  • (e.g. GUI)
  • More complicated.
  • Most GUIs wrap CLI

Reverse-engineering RTMP

  • Flash player (& server)
  • Streaming audio/video/data
  • Proprietary

HTTP downloads are not possible: we don’t own most of the content. That’s why you’ve spotted RTMP being used – it’s a form of non-invasive Content Restriction And Protection. I’m sorry we have to use it. But we have to use it. —James Cridland

Partial open-source implementations

  • Red5
  • RubyIZUMI
  • Gnash
  • RTMPy
  • Stream carries RTMP packets
  • Stream carries RTMP packets
  • Which carry AMF packets
  • Stream carries RTMP packets
  • Which carry AMF packets
  • or raw audio/video data
  • Tries to be efficient
  • Tries to be efficient
  • Re-using headers
  • Tries to be efficient
  • Re-using headers
  • Compact headers

Implementing RTMP

  • Bottom up
  • RTMP first
  • Tests based on publicly-available specs
  • Captured real conversations
  • Tried to decode packets
  • Pencil & paper
  • Captured real conversations
  • Tried to decode packets
  • Pencil & paper
  • Rewrote tests
  • Captured real conversations
  • Tried to decode packets
  • Pencil & paper
  • Rewrote tests
  • Round-trip decode & re-encode
def get_data
  header = Packet::Header.new
  header.parse(@socket)
  if packet = @packets[header.oid]
    packet.endow(header)
  else
    if previous_header = @headers[header.oid]
      header.inherit(previous_header)
    end
    packet = @packets[header.oid] = Packet.new(header)
  end
  @headers[header.oid] = header
  packet << @socket.read(packet.bytes_to_fetch)
  if packet.complete?
    @packets.delete(header.oid)
    yield packet
  end
end
  • Moved up to AMF
  • Recursing parser (no tokenisation needed)
  • Tests based on publicly-available specs
  • Tried to decode real data
  • Pencil & paper
  • Rewrote tests
def recursive_parse(io)
  elements = []
  until io.eof? || (e = next_element(io)) == EndOfPacket
    elements << e
  end
  elements
end
def next_element(io)
  data_type = io.read(1).unpack('C')[0]
  case data_type
  when DT_NUMBER
    io.read(8).unpack('G')[0]
  when DT_BOOLEAN
    io.read(1).unpack('C')[0] != 0
  when DT_OBJECT
    hash = {}
    until (key = read_length_prefixed_data(io)) == ''
      hash[key] = next_element(io)
    end
    hash
  when DT_STRING, DT_LONG_STRING
    read_length_prefixed_data(io)
  when DT_OBJECT_END
    EndOfPacket
  when DT_NULL_VALUE
    nil
  else
    read_length_prefixed_data(io)
  end 
end
["connect",
 1.0,
 {"capabilities"=>15.0,
  "videoFunction"=>1.0,
  "audioCodecs"=>1639.0,
  "app"=>
   "ondemand?_fcs_vhost=cp48184.edgefcs.net&auth=daEcIaKaQdfaic
    ZcBa_aLa9dYbhdCaCc3d9-bizwQB-cCp-FnrDCqBnNDoGuwF&aifp=v001&
    slist=secure/6music/AMI_e6d01bf639fe37be3a42e423f9f38425_b0
    0c73d2_6m_lamacq_thu",
  "videoCodecs"=>252.0,
  "swfUrl"=>"http://www.bbc.co.uk/emp/player.swf?revision=3704",
  "pageUrl"=>"http://www.bbc.co.uk/iplayerbeta/episode/b00c73fc",
  "tcUrl"=>
   "rtmp://84.53.177.140:1935/ondemand?_fcs_vhost=cp48184.edgef
    cs.net&auth=daEcIaKaQdfaicZcBa_aLa9dYbhdCaCc3d9-bizwQB-cCp-
    FnrDCqBnNDoGuwF&aifp=v001&slist=secure/6music/AMI_e6d01bf63
    9fe37be3a42e423f9f38425_b00c73d2_6m_lamacq_thu",
  "fpad"=>false,
  "flashVer"=>"LNX 9,0,124,0"}]

Not finished

  • Switch from AMF to media streaming
  • Flash DRM

What I’ve learned

Releasing software is hard

Releasing Ruby software to users is hard

Releasing Ruby software to users is hard

  • Gems?
  • Tarballs?
  • Exes?

GUIs are tedious

GUIs are tedious

except, apparently, on OS X

Reverse engineering can be fun

Reverse engineering can be fun

and enlightening

Obfuscation doesn’t work

Obfuscation doesn’t work

  • Open source

Obfuscation doesn’t work

  • Open source
  • Public forum

Obfuscation doesn’t work

  • Open source
  • Public forum
  • Big corps are slow

Feedback channel

Feedback channel

(Google forms are good)

Collaboration works