What is the average of no numbers?
From a statistical point-of-view, the average of no sample points should not exist. The reason is simple. The average is an indication of the centre of mass of the distribution. Clearly, for no observations there can be no way to prefer one location vs. another as their centre of mass since the the empty set is translation invariant.
More mathematically, taking the average is a linear operation, which means if you add a constant $c$ to each observation, then the average $a$ becomes $a+c$. Now if you add $c$ to each observation in the empty set, you get the empty set again, and thus the average will have to satisfy $a+c=a$ for all $c$, clearly nonsense.
The Fréchet mean generalizes the concept of mean to arbitrary metric spaces. It is the point which minimizes the sum-of-squared distances between elements of the dataset $X$: $$\text{arg}\min_\bar{x} \sum_{x\in X} d(\bar{x},x)^2$$
In the case that $X=\varnothing$, the summation is the empty sum and hence $0$, thus there is no minimizer and the mean is undefined.
However, in general there are multiple points that minimize this sum (consider the dataset consisting of a pair of antipodal points on the sphere), so we should not speak of the mean, rather we should consider the set of such points as mean. Then, the empty set has as mean the entire space (presumably $\mathbb{R}$ in this case).
To directly answer the question, NaN would be better, since being undefined is certainly not-a-number and, likewise, a set is not-a-number.
The correct answer is "Error: Cannot compute the average without any numbers. Please enter at least one number."
$0$ is incorrect, because division by $0$ is undefined, not $0$: $\frac00\neq 0$. If you have $0$ elements, you simply cannot compute their average.
NaN is slightly better, but still kind of wrong. It's a special value of the IEEE floating point standard and represents the result of a calculation that is undefined. It's an implementation detail of how numbers work under the hood, not a proper result to show to the user. You should at least translate it into something like "undefined" or "N/A". Or display a message to explain why there is no result (see above).