$\kappa$-homogeneous topological spaces

This is a great question!

The disjoint union of two circles is $1$-homogeneous, but not $2$-homogeneous. It is $1$-homogenous, since you can swap any two points and extend this to a homeomorphism (basically, "all points look alike"). But it is not $2$-homogeneous, since you can let $A$ be two points from one circle, and let $B$ be two points from different circles; there is no way to extend a bijection of $A$ with $B$ to a homeomorphism of $X$ (not all pairs look alike).

The real line $\mathbb{R}$ is $2$-homogeneous, but not $3$-homogeneous. For $2$-homogeneity, given any two pairs of reals $a,b$ and $x,y$, then no matter how you map these bijectively, you can extend to a homeomorphism of the line by affine translation (all pairs look alike). But the line is not $3$-homogeneous, since we can biject the triples $0,1,2$ mapping to $0,2,1$, respectively; this does not extend to a homeomorphism, since it doesn't respect between-ness (not all triples look alike).

The unit circle is $3$-homogeneous, but not $4$-homogeneous (thanks to Andreas Blass in the comments). For $3$-homogeneity, given two triples of points, we can match up the first in each by rotating the circle, and then match up the other two by stretching or by flipping and stretching, depending on whether the orientation was preserved or not (all triples look alike). It is not $4$ homogeneous, since we can have four points in clockwise rotation, and then try to fix the first two and swap the other two; this cannot extend to a homeomorphism, since fixing the first two fixes the orientation, which is not respected by swapping the other two (not all quadruples look alike).

I don't know examples yet that are $4$-homogeneous, but not $5$-homogeneous, or $n$-homogeneous but not $n+1$-homogeneous, for $n\geq 4$.

The real plane $\mathbb{R}^2$ appears to be $n$-homogeneous for every finite $n$, but not $\omega$-homogeneous (thanks again to Andreas). It is $n$-homogeneous, because given any two sets of $n$ points, we can imagine the plane made of stretchable latex and simply pull the points each to their desired targets, with the rest of plane getting stretched as it will. One can see this inductively, handling one additional point at a time: having moved any finitely many points, nail them down through the latex; now any additional point can be stretched to any desired target, before also nailing it down, and so on (so all $n$-tuples look alike). It is not $\omega$-homogeneous, since a countable dense set can be bijected with a countable bounded set, and this will not extend to a homeomorphism.

Meanwhile, the infinite case is settled. For any infinite cardinal $\beta$, the discrete space of size $\beta$ is $\alpha$-homogeneous for every $\alpha<\beta$ but not $\beta$-homogeneous, in the OP's terminology. Any bijection of small subsets can be extended to a permutation, since there are $\beta$ many points left over, but if $X$ has size $\beta$, then we can take $A=X$ and $B=X-\{a\}$, which are bijective, but this bijection cannot be extended to a permutation of $X$.

In particular, the countable discrete space is $n$-homogeneous for every finite $n$, but not $\omega$-homogeneous, just like $\mathbb{R}^2$.

The sort of space you describe is usually called strongly $\kappa$-homogeneous. If you google that phrase you will find some interesting results about these kinds of spaces (mostly concerning how this property relates to other homogeneity properties).

The earliest reference I could find to strongly $n$-homogeneous spaces (only finite values of $n$ are considered) is in a 1953 paper by C. E. Burgess (available here).

Despite the fact that these kinds of spaces have been in the literature for well over half a century, it is unknown whether there is a topological space that is strongly $4$-homogeneous but not strongly $5$-homogeneous. (This is stated explicitly in this paper by Ancel and Bellamy from last year -- see the second paragraph of the second page.) As far as I can tell (although I don't have an authoritative reference), it is also unknown whether there is a strongly $n$-homogeneous space that is not also strongly $(n+1)$-homogeneous for any finite $n \geq 4$.

Therefore Joel's answer (along with some of Andreas's comments below it) constitutes the state-of-the-art knowledge on this question.