Almost equivalent definitions of the Riemann–Stieltjes integral

Depending on how the Riemann-Stieltjes integral is defined (at least two ways that you mention) there are a variety of joint conditions on the integrand $f$ and integrator $g$ that guarantee existence. There is some but not complete overlap.

The Riemann-Stieltjes integral is less flexible than the Riemann integral. One impediment is that the continuity of $g$ comes into play. A function that is continuous is Riemann integrable and so too is one that is discontinuous only on a set of measure zero. Taking definition (2), if the integrator is increasing, then if $f$ is continuous, the Riemann-Stieltjes integral exists, but may not exist if $f$ is only continuous almost everywhere. That is because it is necessary that the integrand and integrator have no common points at which they are discontinuous.

Some basic relationships (of which I am aware) are:

Definition (1) holds if and only if definition (2) holds for Riemann integrals where $g(x) = x$.

Definition (1) implies definition (2) for Riemann-Stieltjes integrals when $f$ is bounded and $g$ is increasing.

Definition (2) implies definition (1) for Riemann-Stieltjes integrals when $g$ is increasing, and either $f$ or $g$ is continuous. You found a counterexample if the continuity requirement is relaxed.

To prove the third implication, first consider that $f$ is continuous and R-S integrable with respect to $g$ under definition (2). Then for any partition $P = (x_0,x_1, \ldots,x_n)$ and choice of tags we have

$$\left|S(P,f,g) - \int_a^bf \, dg\right| = \left|\sum_{j=1}^n f(\xi_j)[g(x_j) - g(x_{j-1})] - \sum_{j=1}^n \int_{x_{j-1}}^{x_j}f \, dg\right|$$

Since $f$ is continuous we can apply the integral mean value theorem to find points $\eta_j$ such that

$$\left|S(P,f,g) - \int_a^bf \, dg\right| = \left|\sum_{j=1}^n [f(\xi_j)-f(\eta_j)]\,[g(x_j) - g(x_{j-1})] \right| \\ \leqslant \sum_{j=1}^n |f(\xi_j)-f(\eta_j)|\,[g(x_j) - g(x_{j-1})]. $$

By uniform continuity of $f$, for any $\epsilon >0$ there is a $\delta > 0$ such that if $\|P\| < \delta$ then $|f(\xi_j)-f(\eta_j)| < \epsilon/(g(b) - g(a))$ and

$$\left|S(P,f,g) - \int_a^bf \, dg\right| < \epsilon.$$

Proof of the implication assuming that the integrator $g$ is continuous, rather than $f$, is lengthier. In brief, we choose a partition $P' =(x_0,x_1,\ldots,x_n)$ such that the upper sum $U(P',f,g)$ and lower sum $L(P',f,g)$ are within $\epsilon/2$ of the integral. Using the uniform continuity of $g$, we find $\delta >0$ such that $|g(x) - g(y)| < \epsilon/(2nM)$ when $|x-y| < \delta$, where $M$ bounds $f$. Then a partition $P$ with $\|P\| < \delta$ is constructed through a tedious process such that

$$\int_a^b f \, dg - \epsilon < L(P',f,g) < L(P,f,g) \leqslant S(P,f,g) \leqslant U(P,f,g) < U(P',f,g) < \int_a^b f \, dg + \epsilon. $$

Asking that $f$ be continuous is sufficient. In fact, all you need is that $f$ be left continuous when $g$ is right-discontinuous, and vice versa. This is a rather easy exercise that can be done by considering the difference between the maximum and the minimum values attainable by choosing sample points on a given partition. As a corollary of this, $f$ must be continuous where $g$ is both left- and right-discontinuous.

Where $g$ is continuous, it can be shown fairly easily that $f$ is allowed to have finitely many discontinuities. In fact, as long as the set of discontinuities of $f$ is measure zero (in the sense of Lebesgue measure), the integral ought converge. This proof is identical to that on the Riemann integral.