Anchor tag inside <h1> or <h1> inside anchor tag: which is better?

If you are using HTML5, just pick one; they're equivalent.
HTML5 does allow block-level links, but in your case there's no particular reason to do it, since there's only one block-level element. Personally, I wouldn't do it here, because having the <h1> tag on the outside would make it easier to scan for in source code.

Anything else (XHTML, HTML4, etc) and the second one is just plain wrong. It wouldn't be valid code, and on some level that's bad for your search optimization [Insert standard disclaimer about how much any single offense really affects anything, etc.].

They're the same as far as SEO is concerned. (Usually block level elements contain inline elements and not vice-versa so you should use the first example but it won't affect SEO).

They're both correct in html5, html allows block elements in inline elements. This also has no effect on SEO, both cases the text is wrapped in the heading, so it remains to have the same value.

It's not a choice of validness, but a preference in User Interface:

• If you wrap the header around the anchor, you create a large anchor, only the text will be clickable.
• When you wrap the anchor around the header, the whole line get's clickable.

I've made you an example on jsFiddle.net