Higher $\infty$-categories
Using Street's "one type" definition of strict $\infty$-category one can see that the concept of "strict $P$-category" makes sense not justs for any ordinals $P$ but in fact for any posets $P$ (though I expect the definition below is not right when $P$ is not totally ordered, see the remark at the end)
Definition : a $P$-category is a set $X$ endowed with:
(source and target) For each $p \in P$, two functions $\pi_p^+$ and $\pi_p^{-}$ from $X$ to $X$.
(composition) For each $p \in P$ a partially defined composition operation $\#_p : X \times X \rightarrow X$.
Satisfying the following conditions:
(globularity relations) For any $q \leqslant p$ one has $ \pi_p^{\mu} \pi_q^{\epsilon} = \pi_q^{\epsilon}$ and if $q>p$ one has $ \pi_p^{\mu} \pi_q^{\epsilon} = \pi_p^{\mu}$.
(dimension axiom) For any $x \in X$, there exists $p \in P$ such that $\pi^+_p(x)=x$.
(domain of composition) $x \#_p y$ is defined if and only if $\pi^+_p x= \pi^-_p y$.
(boundaries of compositions) $\pi_p^+(x \#_p y)=\pi_p^+ y$ ; $\pi_p^-(x \#_p y)=\pi_p^- x$ and $\pi^{\epsilon}_q(x \#_p y) = \pi^{\epsilon}_q(x) \#_p \pi^{\epsilon}_q(y)$ if $q>p$.
(unit law) For any $x \in X$ and any $n$ one has $x \#_n \pi_n^+ x= x = \pi^-_n x \#_n x$.
(associativity of compositions) $(x \#_p y )\#_p z = x \#_p (y \#_p z)$ when either side is defined.
(exchange law) for $p< q$, $(w \#_q x) \#_p (y \#_q z) = (w \#_p y) \#_q (x \#_p z)$ when the left hand side is defined.
(where $\epsilon$ and $\mu$ denotes arbitrary signs)
Saying that a cell is ``invertible'' makes sense exactly as in strict $\infty$-category, so you can talk about $(\alpha,\beta)$ categories for any ordinals $\alpha$ and $\beta$.
Now:
So far we are lacking motivation (interesting examples) to develop such a theory.
weak version of this notion have not been defined. Moving from strict $\infty$-categories to weak $\infty$-categories is a difficult jump, so there might be a lot of work to came up with such a notion. And as I said we don't really have a motivation to do so.
Even in the strict case, the "homotopy theory" of such object has not been developed (I'm talking about an analogue of the Folk model structure). This is probably easier than the point above, but still would need to be done.
As I said we are lacking examples, so it is not clear this notion has any interest at all, but I don't think the notion is trivial in any sense, and actually we do have a few notable examples beyond $\infty$-categories:
For example a $2 \omega$-category with only one cell of dimension $<\omega$ is exactly the same as a strict $\infty$-category with a commutative monoide structure, and I expect that the weak version should be a $E_{\infty}$-monoidal $\infty$-category in the same way that a weak $\infty$-category with only one cell of dimension $<k$ is the same as a $E_{k}$-monoidal $\infty$-category
As mentioned by Mike above, $\mathbb{Z}$-groupoids are closely connected to spectra. I believe using a similar argument to the Dold-Kan equivalence for strict $\infty$-category one can show that a strict $\mathbb{Z}$-groupoids is the same as an unbounded chain complexes (not quite sure about this... maybe there are problems in $-\infty$ and only a special type of $\mathbb{Z}$-groupoids needs to be considered). If true this is definitely a good indicator that a weak $\mathbb{Z}$-groupoids should be a spectra.
(In fact I remember seeing an arxiv preprint using $\mathbb{Z}$-categories to model spectra or something in this spirit... but I did not really read it and I havn't been able to find it back.)
To me the observation about infinite dimensional sphere is not really relevant here: to me what it says is that spaces don't have information past $\omega$, but this is just the fact that spaces up to homotopy are $\omega$-groupoids and not $\omega^+$-groupoids or anything like this.
Also if I'm correct the $\omega^+$-category freely generated by an arrows $\theta$ of dimension $\omega$ has for arrows $\theta$ and all the $\pi^{\epsilon}_n \theta$ for $ n < \omega$ and $\epsilon=+/-$ and that they are all different, so the type of collapse Harry mentioned in his answer do not seem to happen, at least with this definition.
Finally, I believe the definition given here is only right when $P$ is totally ordered, but I suspect that there is a modification of this definition, which is equivalent for totally ordered set, and such that for example a $\mathcal{P}(\{1,\dots,n\})$-category is the same as an $n$-fold category.
Because what we today call $(\infty,n)$-categories are really homotopy-coherent strict $n$-categories. It is an open question if they are equivalent to some of the algebraic definitions of weak higher categories where you could try to formalize this.
The only case I've ever looked into was Street's strict $\omega^+$-categories, and the additional layer ends up forcing some things to be trivial, so it was concluded that this is not a natural or useful model.
The numbers that appear in terms such as $n$-categories, are the dimensions of cells. However, infinite-dimensional spheres are contractible. Therefore, at least from that perspective, it seems that infinite ordinals will contribute no homotopical information.
This of course assumes that we are interested in homotopy-theoretic information. It is plausible that one may alter the definitions, or be interested in something different altogether, and then infinite ordinals will give new information.
(this is a rewrite of my comment above)