Strange error when unvboxing in `\hbox`

On page 286 of the TeXbook we find

Some commands are incompatible with horizontal mode because they are intrinsically vertical. When the following commands appear in unrestricted horizontal mode, they cause TeX to conclude the current paragraph:

    ⟨vertical command⟩ → \unvbox | \unvcopy | \halign | \hrule |
        \vskip | \vfil | \vfill | \vss | \vfilneg | \end | \dump

The appearance of a ⟨vertical command⟩ in restricted horizontal mode is forbidden, but in regular horizontal mode it causes TEX to insert the token par into the input; after reading and expanding this par token, TeX will see the ⟨vertical command⟩ token again. (The current meaning of the control sequence \par will be used; par might no longer stand for TeX’s \par primitive.)

You get exactly the same error with \hbox{a\vfil}.

Module 1062 of tex.web might help

Many of the actions related to box-making are triggered by the appearance of braces in the input. For example, when the user says ‘\hbox to 100pt{⟨hlist⟩}’ in vertical mode, the information about the box size (100pt, exactly) is put onto save stack with a level boundary word just above it, and cur_group ← adjusted_hbox_group; TeX enters restricted horizontal mode to process the hlist. The right brace eventually causes save_stack to be restored to its former state, at which time the information about the box size (100pt, exactly) is available once again; a box is packaged and we leave restricted horizontal mode, appending the new box to the current list of the enclosing mode (in this case to the current list of vertical mode), followed by any vertical adjustments that were removed from the box by hpack.

^{Read par as the a boxed par.}

There might be an error such as “vertical command found in restricted horizontal mode” supplementing the “Missing } inserted” (in your case there's also an \endgroup to supply first). There is a class of error messages You can’t use x in y mode (module 1049), but it would not be very useful here, because inserting the proper tokens to end restricted horizontal mode would be needed anyway and ignoring the \unvbox command would need further processing and possibly more errors than what you find here.

Now, since \unvbox is only legal in unrestricted horizontal mode, you could fix your code as

\relax\ifhmode
  \ifinner
    % possibly issue a warning
    \box
  \else
    \unvbox % will trigger insertion of \par
  \fi
\else
  \unvbox
\fi

The warning should point out that the output might not be as intended. No need to \expandafter in order to remove \fi or \else, because after \box or \unvbox TeX will expand tokens until finding a ⟨number⟩.

A lot of it comes down to the structure of the TeX program. From §1064 of tex.web:

We have to deal with errors in which braces and such things are not properly nested. Sometimes the user makes an error of commission by inserting an extra symbol, but sometimes the user makes an error of omission. TeX can’t always tell one from the other, so it makes a guess and tries to avoid getting into a loop.

In this instance what's happening is that TeX's best guess is that there should have been a } before the \unvbox since the \unvbox doesn't belong there. It's worth remembering that TeX is written largely with 1970s technology and the limitations thereof.

Is there a rationale for this behaviour?

As others have said, you could argue that this is due to the age of TeX, but I think the behaviour of force ending the inner hmode an thus allowing the vertical list to be output is quite reasonable even now, even though it often causes more or less spurious errors when the "real" end of group is encountered.

After any error the PDF should just be seen as a potential debugging aim, sensible PDF output is a non-aim. The suggested alternative fallback of skipping the unvbox would hide most of the output from the PDF and (perhaps) make it harder to see the location of the problem.

This is consistent with the rest of the error recovery in TeX. If you use alpha in tex mode it force starts math mode putting a $ in front, even though this almost always trashes the rest of the paragraph as it has no good way to stop math mode. It does not drop the \alpha.