Definition of subcoalgebra over a commutative ring

I'm surprised nobody has answered yet so let me state (what seems to me to be) the obvious: (2) is the only reasonable definition of sub-coalgebra which makes sense in general. Said differently you want $\Delta:D\rightarrow C\otimes C$ to factor through $D\otimes D \rightarrow C\otimes C$. The definition in your notes is either an oversight or an abuse of language, I'd say.

As მამუკა ჯიბლაძე suggests, coalgebras in $R$-mod are the same as algebras in $R-mod^{op}$, and you don't require the latter map to be injective in the same way that for a quotient algebra $A\rightarrow B$ you don't usually require $A\otimes A\rightarrow B\otimes B$ to be surjective as well (why would you ?). Of course for algebras over a commutative ring this is automatically true, but it's not true for symmetric monoidal categories whose tensor product is not right exact, which is exactly what happens for $R-mod^{op}$.

The downside is that the coalgebra structure on a sub-coalgebra might not be uniquely determined, i.e. it is possible to have two different coalgebra structures on $D$ for which the inclusion $D\rightarrow C$ is a coalgebra map. In other words, the factorisation of the map $D\rightarrow C\otimes C$ might not be unique. Somewhat related, if $C$ is not flat then its category of comodules might not be abelian.

See e.g. Wischnewsky, On linear representations of affine groups. I. for a reference where this is carefully phrased.

You are right.
In the case of an $R$-submodule $D$ of an $R$-coalgebra $C$, the correct definition for $D$ being a subcoalgebra of $C$ is your definition (2) and not the one posted in your notes. This is standard in the contemporary literature: see for example p.11, sect. 2.7 of 1.
The definition mentioned in your notes is valid (as you have already mentioned in the OP) for vector spaces and special cases for $R$-coalgebras (when $D$ is flat or pure as an $R$-submodule for example).

The problem is indeed the one you mention: In general, the restriction of the comultiplication of $C$ to $D$, that is $\Delta\vert _D: D \to C \otimes C$, does not necessarily "lift" to a map $\Delta\vert _D: D \to D \otimes D$. This can happen, even in cases in which $\Delta(D)$ is contained inside the image of the canonical map $D\otimes_R D\to C\otimes_R C$.
A relevant example (attributed to Warren Nichols) is the following: Consider the $\mathbb{Z}$-module $C=\mathbb{Z}/8\mathbb{Z}\oplus \mathbb{Z}/2\mathbb{Z}$ and denote $x=(1,0)$, and $z=(0,1)$. A comultiplication $\Delta:C\to C\otimes_\mathbb{Z} C$ is defined by $\Delta(x)=0$ and $\Delta(z)=4x\otimes x$. So $\Delta(z)$ has order $2$ in $C\otimes_\mathbb{Z} C$. Then, let $y=2x$ and take the $\mathbb{Z}$-submodule $D=\mathbb{Z}y+\mathbb{Z}z\subset C$. Now, $\Delta(z)=y\otimes y$ and $\Delta(D)$ is contained in the image of $D\otimes_\mathbb{Z} D\to C\otimes_\mathbb{Z} C$ but the restriction $\Delta|_{D}$ does not lift (or: does not factor) to a comultiplication $\Delta:D\to D\otimes_\mathbb{Z} D$, since we can see that any preimage of $y\otimes y$ in $D\otimes_\mathbb{Z} D$ has order $4$.
(This example was first suggested by Nichols in 2, p.56 and is also cited as an exercise in 1).

Edit: Another interesting phenomenon, related to the definition, of an $R$-subcoalgebra (definition (2) of the OP), is the fact that following this, the $R$-subcoalgebra $D$ of the $R$-coalgebra $C$ is not uniquely determined: In other words, the same $R$-submodule may correspond to non-isomorphic subcoalgebras. This has already been mentioned and sufficiently explained in Adrien's answer. I thought it might be interesting to include an example of this "phenomenon" (which is again suggested in 2 and cited as an exercise in 1):
Consider the $\mathbb{Z}$-module $C=\mathbb{Z}\oplus\mathbb{Z}/4\mathbb{Z}$ and make it a coalgebra by setting $g=(1,0)$ to be a grouplike and $x=(0,1)$ to be $g$-primitive. Then, take the submodule $D\subset C$ which is spanned by $g$ and $2x$. So, $D\cong\mathbb{Z}\oplus\mathbb{Z}/2\mathbb{Z}\cong\mathbb{Z}\oplus 2\mathbb{Z}/4\mathbb{Z}$. There are (at least) two different ways to make $D$ into a coalgebra: Either consider $g$ to be a grouplike and $2x$ to be $g$-primitive (this corresponds to the restriction of the original map of $C$ in $D$), or take again $g$ to be grouplike and define $$ \Delta(2x)=g\otimes(2x)+(2x)\otimes(2x)+(2x)\otimes g \ \ \ and \ \ \ \epsilon(2x)=0 $$ Both options make $D$ into a coalgebra and in both cases one can easily show that the inclusion $D\hookrightarrow C$ is a coalgebra morphism. So, $D$ becomes a $\mathbb{Z}$-subcoalgebra of $C$ in (at least) two different ways. Furthermore, these are not isomorphic to each other, since the second structure has an additional grouplike element; this is $g+(2x)$.

References:

1. Corings and comodules, T. Brzezinski, R. Wisbauer,
2. Nichols, W.D., Sweedler, M., Hopf algebras and combinatorics, AMS, in: "Umbral Calculus and Hopf algebras", Contemp. Math. 6, 49–84 (1982)