Finite Square Well Inside an Infinite Square Well

The potential you have written is perfectly legitimate, let's do the derivation from scratch: $$ V(x) = \begin{cases} -V_0&0<x<b \\ 0&b<x<a \\ \infty&x>a \\ \end{cases}$$

and $V(-x) = V(x)$ the Hamiltonian will commute with the parity operator $[\hat{H},\hat{\mathscr{P}}] = 0$, therefore the eigenstates of the energy will be eigenstates of the parity operator too. Also, we can solve the problem in the interval $[0,a]$ with $b<a$ since the eigenfunctions of $\hat{H}$ are parity eigenstates too. Now and this is the most important thing: an infinite well can ONLY have bound states, so we have to look for bound states of the infinite well and adapt this fact to work with the finite well inside (which adimts "scattering" (positive energy) states). This basically means that $E>0$ (you can easily check that if you started from the condition to get the bound states inside the finite well you would get an impossible condition for the infinite one at the point $x=a$)

Region $[0,b]$:

The Shrodinger equation is $$-\frac{\hbar^2}{2m}\nabla^2\psi-V_0\psi= E\psi$$ with $V_0>0$ in one dimension the solution will be: $$\psi_+(x)=C_1 cos(\omega x)\\ \psi_-(x)=C_1 sin(\omega x)$$ we will work with parity $+$. for comodity. With $\omega=\sqrt{\frac{2m}{\hbar^2}(V_0+E)}$.

Region $[b,a]$:

Here the Shrodinger equation is: $$\frac{\hbar^2}{2m}\nabla^2\phi= E\phi \\ \phi(a)=0$$ notice that $\phi(a)=0$ is the foundamental condition that "tells" the particle that there is an infinite well, and this condition is also the reason that $E>0$ infact if $E<0$ you would have $\phi_+= A_1 \cosh(kx)$ which only has immaginary solutions for $\phi(a)=0$ or the useless $k=0$. The solution for $E>0$ here will be: $$\phi_+(x)=A_1 cos(k x)\\ \phi_-(x)=A_1 sin(k x)$$ with $k=\sqrt{\frac{2m}{\hbar^2}E}$, now since $\phi_{\pm}(a) = 0$ we have that $k$ must be: $$k_+=\frac{(2n+1)\pi}{2 a} \\ k_-=\frac{n\pi}{a} $$.

Now we have to match the solutions in $x=b$. (we have to match the derivatives too since the general solution must be $C^2$)

$$\psi_+(b)=\phi_+(b) \\ \psi'_+(b)=\phi'_+(b)$$ this leads to (for the parity even eigenstates):

$$C_1 cos(\omega b) = A_1 cos(k b)\\ C_1 \omega sin(\omega b) = A_1 k sin(k b)$$ The solution reads: $$\tan(\omega b) = \frac{k}{\omega} \tan(k b) $$ which can be written as : $$\tag 1 \omega \tan(\omega b) = \frac{(2n+1)\pi}{2a} \tan\left( \frac{2n+1}{2a}\pi b \right)$$

Now, if you do $b \rightarrow a$ you get that $\tan\left(\frac{2n+1}{2}\pi\right) \rightarrow \infty$ which means that $cos(\omega b)=0$ which means that $\omega = \frac{2n+1}{2 b}\pi $ (also $k$ vanishes) which is the usual energy for the infinite well.

The limit $b \rightarrow 0$ is consistent too. Consider equation $(1)$ written as $$\omega=k \frac{\tan\left(kb\right)}{\tan\left(\omega b\right)}$$ now take the limit $b \rightarrow 0$ on both sides $$\ \omega = k\cdot \frac{k}{\omega}\longrightarrow \omega = k$$ and you are left with $k = \frac{(2n+1)\pi}{2 a}$ which is the usual infinite well energy.

I do not see the problems you appear to be worried about--perhaps others! To start with, with the infinite walls, all states are bound, with positive or negative energies. You might as well take your 0 of energy at the very bottom of the potential, and a step up at $V_0$, instead, the physics being the same. It would also spare you needless grief to work in units s.t. ħ=1 and m=1/2. In that case, there are 3 types of solutions, for $E<V_0$, $E=V_0$, and $E>V_0$. You might consult a very similar problem in Gilbert et al, EurJPhys 26 (2005) 815–825.

Your limit b → a appears fine: The left-hand side blows up, and so therefore the tangent on the r.h.side must too, so the cosine in the denominator must vanish, and bK = n π/2 for some odd n, while K is the unique "inner" wavenumber, K=nπ/2b, familiar from the infinite square well. Might choose to flip the sign of your energy as Trimok points out.

Other limits you may have to check involve b → 0 when the r.h.side vanishes, and the zero of the energy loses its significance, so the "outer" wavenumber can take an i in front of it, so the left hand side vanishes by Euler's formula, too.

The limit $V_0$ → 0 is more interesting, and with care should yield a constraint equation only depending on a, but you may have to tweak it and use more convenient variables to see that. Remember, the momentum at the outer, infinite, wall is discontinuous. Try to find just the ground state.

There is an even more interesting limit you should consider, with the energy flush with the top of the step, so k, E → 0 in your language, and $E=V_0$ in mine. The outer wavefunction is a straight line, e.g., $A(x-a)$ on the right, and you get the condition collapse to a condition between $a,b$ and $z_0$.