How to fetch Android Market data when there is no API?

There is a Java project on Google code which attempts to build an open source api for the market: http://code.google.com/p/android-market-api/

There're also some JRuby bindings for it: http://github.com/jberkel/supermarket.

However they both require a Google/Market account, and the API is only partially implemented (searching, comments and images at the moment). Also, the reconstructed protocol buffer descriptor (market.proto) could be used to generate bindings for languages other than Java.

Another option to capture data is using the tcpdump tool that comes with some custom images (like Cyanogenmod and others). You can capture all the packets and then analyze them using Wireshark. This route is easier than hijacking wifi.

Checkout this splendid posts to clear out some things:

• Gathering packets with tcpdump from Android
• Downloading Market applications without the vending app

Update I have been working on this a bit more and @gsmd is right and they use Protocol Buffers to transfer the information and with the technique described above you can capture the packets. Then you'll need to dig into protobuffers to understand the messages.