What is the integral of 1/(z-i) over the unit circle?

Contour integrals are the same as integrals in this respect: you can't simply stick a pole in their path of integration. In this case, you can employ what is known as a Cauchy principal value of the contour integral, which excludes the pole. This involves deforming the path of integration around the pole. Whether you deform the path inside or outside the unit circle does not matter, as I will illustrate.

Normally, I deform inside the circle so as to exclude the pole; this is the contour $C$. The integral over $C$ is zero by Cauchy's theorem. In this case, the integral looks like

$$\oint_{C} \frac{dz}{z-i} = PV \oint_{|z|=1} \frac{dz}{z-i} + i \epsilon \int_{2 \pi}^{\pi} \frac{d\phi \, e^{i \phi}}{i + \epsilon e^{i \phi} - i} = 0$$

Thus,

$$PV \oint_{|z|=1} \frac{dz}{z-i} = i \pi$$

On the other hand, if we use a contour $C'$ in which we include the pole, then the integral around $C'$ is equal to $i 2 \pi$ times the residue of the integrand at the pole, which residue is simply $1$. Thus,

$$\oint_{C'} \frac{dz}{z-i} = PV \oint_{|z|=1} \frac{dz}{z-i} + i \epsilon \int_{0}^{\pi} \frac{d\phi \, e^{i \phi}}{i + \epsilon e^{i \phi} - i} = i 2 \pi$$

which produces the previous result.

Now, if you enlarge the circle so that you are no longer on the unit circle but on a circle of larger radius that encloses the pole at $z=i$, then, yes, the integral about that circle is indeed $i 2 \pi$. Or if you shrink the circle so that the pole is outside the circle, the integral about that circle is zero. But for the unit circle, on which the pole lies, the integral over the circle makes sense only when speaking of its Cauchy PV, which is $i \pi$.

The integral does not converge. To show this in the most informative way, you should recall the definition of a contour integral: Let $\gamma(t)$ be a contour defined over $\mathbb{C}$ with $t\in I$. Then

\begin{align} \oint_{\gamma}{f(z)dz}:=\int_I{f(\gamma(t))\gamma'(t)dt}. \end{align}

So applying this to the integral in question (I've missed out a bit of algebra), we get

\begin{align} \oint_{S^1}\frac{1}{z-\imath}dz&=\int_0^{2\pi}{\frac{\imath}{e^{\imath t}-\imath}}dt\\ &=\int_0^{2\pi}\frac{\sin(t)-1+\imath\cos(t)}{1-2\sin(t)}dt. \end{align} Now, one can (after a little work) compute that \begin{align} \int\frac{\sin(t)-1+\imath\cos(t)}{1-2\sin(t)}dt=\frac{\tanh^{-1}\left(\frac{\tan\left(\frac{t}{2}\right)-2}{\sqrt{3}}\right)}{\sqrt{3}}-\frac{t}{2}-\frac{\imath}{2}\log(1-2\sin(t))+z_0. \end{align}

This function is undefined at both $0$ and $2\pi$ and hence the integral has no limit, or formally it diverges.