How to Detect Physical object in Android Augmented Reality?

The bad news is, you can't use AndAR to detect physical objects. AndAR is based on a fiducial marker approach, where the marker is made of two components: a solid border and an interior pattern. The pattern encodes a value that can be used to address a particular model to render on the marker, and the border makes it easy to determine the relative orientation of the marker to the device. Clearly this is just planar image recognition.

To do object recognition on a 3D object is a more complicated problem, and I don't know of any Android libraries that provide a turn-key solution, but recognizing just one object is probably feasible on a mobile device.

One possibility might be to investigate the available Android AR toolkits (Layar, Junaio, Qualcomm AR SDK) which all now support some image recognition. It may be that by taking images of your teapot at various rotations and using those as the images you want your app to match against that you might get this solution working, but keep in mind they are only designed to do planar matching on images, not real 3D objects, so the performance might not be great. (Well, the Metaio Mobile SDK Pro does 3D recognition and tracking, but it's very expensive).

While object recognition is perhaps best done by comparing camera frames with images of the object you wish to recognize (or by comparing image features from the camera frames with pre-computed image features etc), tracking is a different matter. If you want to accurately track your 3D object in 3D space you'll really need to have or build a 3D model of it, and for each frame determine point correspondences between the camera image and the 3D object for tracking. True unassisted (i.e. no depth-camera) 3D tracking is hard.

I hope this gives you some background you can use to evaluate your next steps.

Update: Qualcomm's Vuforia SDK allows you to track "multi targets", which are objects with a set of planar tracking surfaces with a fixed spatial relationship. If you made a "cube" different photos of the 6 sides of your object (teapot) that might work somewhat. https://ar.qualcomm.at/qdevnet/developer_guide/Trackables

End of 2013 Update:

I have no experience with these, but:

Metaio now offer 3D tracking of CAD models: https://dev.metaio.com/sdk/tutorials/3d-tracking-based-on-cad-data/

ARLab.com's LinkAR promises object matching. http://www.arlab.com/objecttracking

I would note the use of the word "matching" - I think the use case here is you know the object you want to overlay (a toy-box, and engine etc). Differentiating between multiple 3D objects may be entirely out of scope.

I recently read about research being doing on hierarchical shape vocabularies used for object representation. Of course, there is no library available for download, but in case you are interested in the general approach here you can find some papers.

Also you might be interested in this paper. It describes an algorithm for detecting objects based on a set of contours.