How many 7-note musical scales are possible within the 12-note system?

Since $7$ is prime and there is no $7$-note signature that sums to $12$ with all $7$ steps identical, we don't have to worry about periodicity; we can just divide by $7$ in the end. Thus, we just have to count the number of ways of distributing $12-7=5$ balls into $7$ bins with capacity $3-1=2$. There are $\binom75=21$ ways to have $5$ steps of $2$, $\binom7{1,3,3}=140$ ways to have $3$ steps of $2$ and $1$ step of $3$, and $\binom7{2,1,4}=105$ ways to have $1$ step of $2$ and $2$ steps of $3$, for a total of $21+140+105=266$ scales in $266/7=38$ cyclically inequivalent types.

In the present case, inclusion-exclusion would be a bit of an overkill, but since you said you'd like a method that generalises to any number of notes with any number of maximum steps, let's generalise: For $k$ notes with a maximum of $m$ steps that sum to $12$, we want to distribute $12-k$ balls into $k$ bins with capacity $m-1$. As explained at Balls In Bins With Limited Capacity, inclusion-exclusion yields a count of

$$ \sum_{t=0}^{12-k}(-1)^t\binom{12-k}t\binom{12-k+k-tm-1}{12-k-1}=\sum_{t=0}^{12-k}(-1)^t\binom{12-k}t\binom{11-tm}{11-k}\;, $$

where, contrary to convention, binomial coefficients with negative upper index are taken to be zero. For the present case of $k=7$, $m=3$, this again yields

$$ \sum_{t=0}^7(-1)^t\binom7t\binom{11-3t}6=\binom{11}6-\binom71\binom86=266 $$

signatures. If $k$ isn't prime, or if it divides $12$, then you have to do a bit more to deal with periodicity; otherwise, you can just divide the above result by $k$.

We begin by writing a list of partitions of $12$ (with the numbers in descending order). The restricted partitions of $12$ into specifically seven parts with maximum part size $3$ are the following:

$3,3,2,1,1,1,1$

$3,2,2,2,1,1,1$

$2,2,2,2,2,1,1$

The list was constructed with the number of threes used in mind. It is clear that no other partitions match the desired conditions.

As we are counting the number of arrangements where cyclic shifts are irrelevant, we notice that those that fall into the first two cases can be described with respect to the location of the number that occurs exactly once. Without loss of generality, let us count then the number of these where the unique number occurs at the beginning.

There are then, $\binom{6}{2}=15$ and $\binom{6}{3}=20$ of these respectively

The final case can be counted by considering the smallest distance between the ones. The smallest distance is always either $1,2$ or $3$, corresponding to $(1,1,2,2,2,2,2),(1,2,1,2,2,2,2),(1,2,2,1,2,2,2)$ respectively.

There are then a total of $38$ possible heptatonic scales.

Here is another method. A scale contains the tonic, by definition, and 6 of the other 11 pitch classes. There are $\binom{11}{6}=462$ ways to choose these. As joriki notes, none of these are modes of limited transposition, so the number of signatures is $1/7$ of this, namely $66$. But from these, we must remove:

• $(6,1,1,1,1,1,1)$
• the $6$ ways to substitute $2$ for a $1$ in $(5,1,1,1,1,1,1)$
• the $6$ ways to substitute $3$ for a $1$ in $(4,1,1,1,1,1,1)$
• the $\binom{6}{2}=15$ ways to substitute $2$ for two of the $1$s in $(4,1,1,1,1,1,1)$

leaving $66-1-6-6-15=38$ signatures with no interval greater than $3$ semitones.