Evalute the definite integral: $\int_{0}^{3} (x^2+1) d[x]$
The result depends on the definition of "integrating with respect to $[x]$."
One possible definition of the integral at stake is as follows: $[x]$ defines a measure on the real line. The measure is concentrated on the integers. For example $[x](\{2\})=1$, $[x](\{1,2\})=2$, but $[x]((1,2))=0$. So $$\int_A f(x)d[x]=\sum_{i\in A}f(i).$$ In our special case
$$\int_0^3 x^2+1\ d[x]=\int_{[0,3]}x^2+1\ d[x]=1+2+5+10=18.$$
If we mean integrating over $[0,3]$ when we write $\int_0^3$.
Or the result is $17$ if we have in mind $(0,3]$ when we write $\int_0^3$.
Interpretation as a Lebesgue-Stieltjes integral
According to the formal definition of the Lebesgue-Stieltjes integral the result is $17$. Why? Because the measure generated by the "integrator" is defined as follows: First we define the measure for an interval the following way:
$$\mu([s,t))=[t]-[s].$$
...
Interpretation as a Riemann-Stieltjes integral:
By the definition we need a partition of the integrating domain $$\mathbb P=\{x_0=0<x_1<x_2<\cdots<x_n=3\}$$ then the approximating sum is
$$S(\mathbb P,x^2+1,[x])=\sum_{i=0}^{n}(c_i^2+1)([x_{i+1}]-[x_i])$$ where $c_i\in [x_{i+1},x_i].$ The Riemann-Stieltjes is said to be existing if (roughly speaking) the limit of $S$ exists when the length of even the wider interval in the partition above tends to zero.
The limit exists and it equals $17$ (not $18$) because the "integrator differences" $[x_{i+1}]-[x_i]$ are all zeros if there is no integer in the interval $[x_i,x_{i+1}]$ and they are all one otherwise. Within $[0,3]$ there are four integers: $0,1,2,3$. $0$, however, even if it is in $[0,x_1]$, does not count because the corresponding "integrator increment" $[x_{1}]-[0]=0$ (if the partition is dense enough), while $3$ contributes since $[3]-[x_{n-1}]=1$ (if the partition is dense enough. So contribute $1$ and $2$.
The OP formally applies the method of integration by part with the following choice $$\frac{d[x]}{dx}=v', x^2+1=u, \text{ and }\int_0^3 uv'\ dx=[uv]_0^3-\int_0^3u'v\ dx.$$ From here, following the formal process, we get $$\int_0^3 x^2+1\ d[x]=\big[[x](1+x^2)\big]_0^3-2\int_0^3x[x]\ dx.$$
So far so good. But then there is a sign mistake (circled in red) and then I cannot follow the reasoning.
Integrating by part
The method exists:
$$\int_0^3(x^2+1)\ d[x]=(3^2+1)[3]-(0^2+1)[0]-\int_0^3 [x]\ df(x)=$$ $$=30-\int_0^3[x]2x\ dx=30-2\left(\int_0^1 0\ x\ dx+\int_1^2 \ 1\ x \ dx+\int_2^3\ 2 x\ dx\right)=$$ $$=30-2\left(0+\frac32+5\right)=17.$$
(Pheww!)
Now, I know that this is what the OP did. But there was a sign mistake unfortunately. Thanks to the OP; I learned a lot.