What is the formal definition of a stellar day?

According to the Wikipedia article you reference, "stellar day" is supposedly a new name for a planet's sidereal rotation period. However, I cannot find any documentation of this new name anywhere, and that includes my copy of Explanatory Supplement to the Astronomical Almanac, 3rd edition, edited by Urban and Seidelmann (University Science Books, 2012) which was just published within the month. This source is definitive and the term doesn't appear therein (okay at least not in the index). However, further digging found reference to it here

http://hpiers.obspm.fr/eop-pc/models/constants.html

but I've yet to find an actual statement of the change in terminology from "sidereal rotation period" to "stellar day" anywhere in the IERS conventions. Anywhere, the distinction is that the sidereal day is measured relative to the moving vernal equinox, which accounts for precession, whereas the sidereal rotation period (stellar day) is relative to the fixed inertial frame of background stars.

The sidereal day is the (mean) time between two transits of the R.A. origin, i.e. the vernal equinox. You are right, and this is strictly not equal to the time it takes the Earth to do a rotation in relation to the fixed stars, because the Vernal Equinox itself is precessing (and nutating).

The amount of time it takes the Vernal Equinox to do a full rotation is suposed to be well known (26000 yr) so you can simply make the correction: since the Vernal Equinox moves towards west, that period you call "stellar day" must be slightly larger, and you can do the correction very easily (I don think there is a more formal definition based upon any reference star, but please post it here if you find it).

Just for curiosity, in which context are you using that "stellar day"? I had never heard about it. Astronomers use the sidereal day as a synonym. The difference is surely much smaller than the "imprecisions" of the Earth movement itself, so it should have had no sense in the past (nor today with atomic clocks).

(Not to be confused with the Solar Day, i.e. the time between two transits of the Mean Sun, which is the "normal" 24-hour day, about 4 minutes larger than the Sidereal Day)