TicTacToe State Space Choose Calculation

Which states do you have that are illegal?
Note that the positions you have counted are either legal, or one move past legal as if the loser was allowed to make one more move after losing. So we need to count the ways that the loser has too many squares.

To count the way X's have a line, but O's and X's have the same number of squares (two cases: 3 of each or 4 of each), we pick the line (8 ways) and then fill the rest of the squares to get $8\times\left(\binom{6}{3} + 2\binom{6}{4}\right)=400$. The $\binom{6}{3}$ is choosing the locations of the O's in the 3 case. If we have 4 of each, then there are $\binom{6}{4}$ ways to place the O's and 2 places left to put the last X.
To count the ways that O's have a line, but there are more X's than O's, we get $8\times\left(\binom{6}{4} + \binom{6}{5}\right)=168$. $\binom{6}{4}$ is the number of ways to have 4 X's and 3 O's. $\binom{6}{5}$ is the case where we have 5 X's and 4 O's.

Subtracting these from your 6046 gets 5,478, which matches the computer.

There are very few, so an exhaustive computer search is quite easy. There are 5,478 total positions, of which 958 are terminal, either because the board is full or because one of the players has won.

This assumes that X always plays first. If O is allowed to start, the set of positions is larger, but still less than twice as large.

This counts board positions, not play sequences. For example, if X plays in space 1, then O plays in space 2, then X plays in space 3, that is considered the same as if X plays in space 3, then O plays in space 2, then X plays in space 1, and it is not counted separately; nor is any subsequent position counted again.

It does not take rotations or reflections of the board into account; rotated or reflected boards are considered different positions. This would be quite easy to fix, if desired.

Source code is here. The game_over subroutine detects whether one of the players has three in a row; the next_moves subroutine calculates the moves available from a position, first using game_over to detect if one of the players has won; if so it reports that there are no legal moves and therefore no following positions.