Google’s new AJAX API cracked
Following on from my last post, I just had a quick look at the data being transferred by Google’s AJAX search API. It’s really simple:
http://www.google.com/uds/GwebSearch? callback=f&context=0&lstkp=0& rsz=small&hl=en&q=YOUR_QUERY_HERE& key=internal-documentation&v=1.0
That returns a JavaScript expression of the form:
f('0', {results:[...]}, 200, null, 200)
To parse the results, just throw the second parameter of the function through a JSON parser.
Unfortunately, the Ruby JSON parser doesn’t like the non-string hash keys (result and so on) but it wouldn’t be a huge effort to fix that.
Update: I’ve written a basic Ruby library. There are more details in the comments.
2006-12-19 17:38 UTC. Comments: 9.
Tieg
Wrote at 2006-12-19 22:05 UTC using Firefox 1.5.0.8 on Mac OS X:
Nice! This even makes the uds_compiled.js more understandable off the bat, although I’ve tried changing the “rsz” parameter from “large” to “small” and it still returns 1 result (assuming that the size of the result set is the number of results it returns).ActiveSupport::JSON.to_json will return JSON with non-string keys, but I don’t know how Rails actually reads a JSON hash—any idea? Also, I just found a post by _why pointing out that “all JSON is valid YAML” (http://redhanded.hobix.com/inspect/yamlIsJson.html), so I tried the YAML library to parse JSON and it worked!
Example:
require ‘yaml’
person = ”{country: { name:’sweden’, continent:’europe’, language:’swedish’ }, name: ‘foobar’}”
person_obj = YAML::load(person)
Paul Battley
Wrote at 2006-12-19 22:11 UTC using Firefox 2.0 on Mac OS X:
Ah, using a YAML parser! How clever!Tieg
Wrote at 2006-12-19 22:40 UTC using Firefox 1.5.0.8 on Mac OS X:
Gah, I spoke too soon– you’ll have to remember to put a whitespace between each colon and value to get the correct keys/values:person = YAML::load ”{country: {name: ‘sweden’, continent: ‘europe’, language: ‘swedish’ }, name: ‘foobar’}”
Paul Battley
Wrote at 2006-12-19 22:41 UTC using Firefox 2.0 on Mac OS X:
That’s interesting. There’s an additional ‘sig’ parameter that I took out. The request still works without it, but only one result is returned, which I hadn’t noticed.All requests from one widget seem to use the same ‘sig’. It doesn’t seem to be tied to a particular client, either: I tried the same request URI on another computer, on a completely different IP address, and the results were the same.
With a valid ‘sig’, you can use ‘rsz=large’ to get more results.
Obviously, generating a valid ‘sig’ value is the next challenge.
Paul Battley
Wrote at 2006-12-19 23:44 UTC using Firefox 2.0 on Mac OS X:
OK, got it! Here’s a first stab at a library: google-ajax.rb.It uses a really ugly hack to massage the JSON into something that the Ruby JSON parser can accept. It also needs a key; Google’s key of ‘internal-documentation’ seems to work fine for testing.
GoogleAJAX::Search.new(key).search(search_terms)
Paul Battley
Wrote at 2006-12-19 23:47 UTC using Firefox 2.0 on Mac OS X:
Incidentally, I found out how to get the ‘sig’ value: Google supply it in the initial JavaScript file!Tieg
Wrote at 2006-12-20 00:21 UTC using Firefox 1.5.0.8 on Mac OS X:
That’s very cool, you should publish it somewhere eventually! Does the version of ruby you’re running include the ‘json’ library? I had to install the gem myself (and omit ri and rdocs because of a dumb error) and then “require ‘rubygems’” before I included the code.You might have already seen this, but there have been some interesting comments over at
http://scripting.wordpress.com/2006/12/19/
scripting-news-for-12192006/#comment-25879
about the whole issue. I wonder how Google would know they were serving these to a non-ajax api (maybe through the user-agent header?), and if they’d ever try to shut it down if it was being used in a live app? Nice job tho!
Paul Battley
Wrote at 2006-12-20 00:31 UTC using Firefox 2.0 on Mac OS X:
I was using JSON via rubygems, but I feel that it’s poor form (read: non-portable) to require it in the library (I just set the environment variable RUBYOPT=rubygems to use gems in any program).Google could get involved in a blocking arms race, but I’m not sure it would be worthwhile to them. Instead, they’ll probably just do what they already do with the HTML pages, and restrict clients that make a large number of automated requests.
dirk luesebrink
Wrote at 2007-02-19 18:00 UTC using Firefox 2.0.0.1 on Mac OS X:
(!i have no idea how this post will render in a readable way, but) as a minor bug fix i suggest fixing the regex to include ’_’ as valid character for variable names before passing it to the JSON parser:([a-z0-9]+) becomes -> ([_a-z0-9]+)
just at this place in your lib:
# Shameful quick and dirty hack to make it work with Ruby’s JSON parser:
json.gsub!(/([{}[],]) ?([a-z0-9]+) ?:/i){ %{#{$1} ”#{$2}”:} }
cheers