Why are nested anchor tags illegal?

While the other good people of the community gave excellent answer, I would like to contribute by giving an example of how to mimic nested anchor tags without actually nesting them in the markup, by making them appear visually as if one is inside another's area:

.parent{
  position: relative;
  width: 300px;
  height: 150px;
  border: 1px solid salmon;
}

.parent .main-link{
  display: block;
  margin-bottom: 2em;
}

/* 
  expands the clickable area of the main link 
  to fill the parent container, because it's the nearest
  ancestor with "position:relative"
*/
.parent .main-link::before{
  content: '';
  position: absolute;
  top: 0;
  right: 0;
  bottom: 0;
  left: 0;
}

.parent a:hover{
  color: green;
}

/* bring other links "forward" to make them clickable */
.parent .secondary-link{
  position: relative;
  z-index: 1;
}

<div class='parent'>
  <a href='#111' class='main-link'>main link, full area</a>
  <a href='#222'class='secondary-link'>secondary link</a>
</div>

Keep in mind that an anchor isn't just a link, it's also something to which one can link. (Though the former use is far more common than the latter.) Quoting W3C (old, but relevant):

An anchor is a piece of text which marks the beginning and/or the end of a hypertext link.

To that end, don't think of an anchor as a link. Think of it as a point in the document which connects to (and/or from) another point (in the same document or another document, it makes no difference). Two points in some non-physical space which are connected by a non-physical thread.

Given that, anchors shouldn't contain a lot of content. If it contains a lot of content, it ceases to become a "point" and starts to become an "area." For example, imagine an anchor which when rendered takes up more space than the browser can display at once. If something links to that anchor, where should it go? The beginning? Middle? End? (Intuitively you might think the beginning, but you get the idea.)

Furthermore, anchors definitely shouldn't contain other anchors. The non-physical connections between the non-physical points can become ambiguous. Does the child anchor connect to the other point, or does the parent anchor connect to the other point? This probably doesn't result in being a big deal in most cases, since the vast majority of anchors today are one-way links from the anchor to another document. But the purpose of the anchor is more than just a one-way link to another document, so the definition continues to embody that purpose.

The spec for <a> has a content model of:

Transparent, but there must be no interactive content descendant.

So the spec is actually more complicated than you say. This also applies to <button> -- essentially you can't have working links inside of links or buttons inside of buttons.

Unfortunately I don't have a strong answer to your question why -- I can only speculate. One clear reason is the ambiguity that it causes (e.g. which anchor should be followed when the inner one is clicked?)

This creates not only functional ambiguity, but also semantic ambiguity. The content of the <a> is a label for its hyperlink. So does this mean that the inner hyperlink is part of the label for the content of the outer link?