Is the so called kTC noise, that is inherent in an RC filter, dependent on bandwidth?

The KTC noise is actually the total RMS noise power at the output of the low pass RC filter. The noise power is the product of the noise power spectral density (PSD) and the bandwidth (more accurately "effective bandwidth") of the system. So, actually the total noise power does depend on the bandwidth.
Interestingly, for the case of RC filter although the resistor is the only noise source present in the system, the total noise power is independent of the resistor value, as seen from the noise power expression, $$v_n^2 = \frac{kT}{C}$$ This is because although increasing the resistor value increases its noise PSD (= 4kTR) but the effective bandwidth (\$\frac{1}{4RC}\$) goes down proportionally. $$v_{n}^2 = 4kTR \cdot \frac{1}{4RC}$$ This can also be understood from the principles of thermodynamics. Since the origin of Johnson noise is due to random motions of electrons inside the resistor, the kinetic theory of gases is applicable for it. And the equipartition theorem says that in thermal equilibrium, energy is shared equally between each state of the system and the shared energy is given by \$E = \frac{1}{2}kT\$.
For the RC system, the only energy state is the energy stored in the capacitor which is given by: $$E = \frac{1}{2}CV^2$$ Equating the two terms gives: $$\frac{1}{2}CV^2 = \frac{1}{2}kT$$ $$\implies V^2 = \frac{kT}{C}$$ This is exactly the noise power derived from circuit equations but is derived much easily from laws of thermodynamics.

the ktC noise formula simply happens when you insert the noise bandwidth of an RC filter, B=1/(4RC), into the formula for voltage square over a resistor. The R cancels out.

So, no, there's correctly no bandwidth in that formula, because the bandwidth is "hidden" in the C.

The wikipedia article on Johnson-Nyquist Noise actually containst this explanation, and a lot more stuff that you might want to know about kTC noise, considering you seem to be coming at this from a non-intuitive, formularistic side. It's usually easier to understand what's happening than to have all the right formulas exactly right :)