Undefined behaviour in (X)HTML?

It's a little more complex than just inspecting the DTD as given by Yi Jiang and mu is too short.

It's true that the XHTML 1.0 DTDs explicitly forbid <a> elements as children of <button> elements as given in your question. However it does not forbid <a> elements as descendants of <button> elements.

So

<button>
    normal text
    <b>bold text</b>
    <span><a href="http://www.example.com/">linked text</a></span>
</button>

is XHTML 1.0 Strict DTD conforming. But it has the same behavioural difference between Firefox and Chrome as the button fragment in the question.

Now, it is known that DTDs have problems describing limitations on descendant relationships, so it's maybe not surprising that the above sample is DTD conforming.

However. Appendix B of the XHTML 1.0 spec normatively describes descendant limitations in addition to the DTD. It says:

The following elements have prohibitions on which elements they can contain (see SGML Exclusions). This prohibition applies to all depths of nesting, i.e. it contains all the descendant elements.

button
must not contain the input, select, textarea, label, button, form, fieldset, iframe or isindex elements.

Note that it does not contain an exclusion for the <a> element. So it seems that XHTML 1.0 does not prohibit the <a> element from being non-child descendant of <button> and the behaviour in this case is indeed undefined.

This omission is almost certainly a mistake. The <a> element should have been in the list of elements prohibited as descendants of button in Appendix B.

HTML5 (including XHTML5) is much more thorough on the matter. It says:

4.10.8 The button element

Content model: Phrasing content, but there must be no interactive content descendant.

where interactive content is defined as

Interactive content is content that is specifically intended for user interaction.

• a
• audio (if the controls attribute is present)
• button
• details
• embed
• iframe
• img (if the usemap attribute is present)
• input (if the type attribute is not in the Hidden state)
• keygen
• label
• menu (if the type attribute is in the toolbar state)
• object (if the usemap attribute is present)
• select
• textarea
• video (if the controls attribute is present)

So in (X)HTML5 the <a> element is prohibited from being a descendant of the <button> element.

The HTML 4 specifications declares the <button> as such:

<!ELEMENT BUTTON - -
     (%flow;)* -(A|%formctrl;|FORM|FIELDSET)
     -- push button -->

Which, if my reading of the DTD is correct (and I'm not exactly familiar with this), <a> elements are explicitly forbidden from being nested in buttons, so what you're looking at there is invalid HTML, and therefore it is undefined behavior.

XHTML says this about <button>:

<!-- button uses %Flow; but excludes a, form and form controls -->
<!ENTITY % button.content
   "(#PCDATA | p | %heading; | div | %lists; | %blocktext; |
    table | %special; | %fontstyle; | %phrase; | %misc;)*">

So <a> is explicitly excluded from XHTML as well. The allowable elements inside <button> appear to be pretty much the same in XHTML-1.0 as in HTML-4.0.