When should I use amsrefs instead of regular bibtex ?

I don't think that amsrefs is new, exactly, although updates to it appear to be reasonably recent. I haven't heard of many people using it. The much more recent biblatex package is probably a better way to go — it has a lot of momentum and supports, well, everything you would ever want to do with bibliographies. Some more information can be found in the TeX FAQ:
http://www.tex.ac.uk/cgi-bin/texfaq2html?label=biblatex

With bib(la)tex, the following is not correct

\begin{theorem}[\cite[p.~100]{blabla}]

you must use an extra pair of {}

\begin{theorem}[{\cite[p.~100]{blabla}}]

See for instance Cite in theorem environment argument

I find this quite annoying. With amsrefs, you write:

\begin{theorem}[\cite{blabla}*{p.~100}]

PERSONALLY, I prefer the amsrefs way.

It is consistent with the citation label in the text: "[1, p. 100]".

On the other hand, bib(la)tex may be consistent with the way we say it: "page 100 of [1]".