Undefined Hamiltonian for this particular Lagrangian

The reason the Hamiltonian is undefined is that converting a Lagrangian to a Hamiltonian requires:

• Finding $p$
• Writing $H=p\dot q-L$
• Expressing $H$ in terms of $p$ and $q$, eliminating all dependence on $\dot q$.

For the third step to be possible, you need to be able to define a coordinate transformation between the $(q,\dot q)$ coordinate system and the $(q,p)$ coordinate system. This requires that $(q,p)$ is a good coordinate system, in that a state of the system can be uniquely represented by a pair of $(q,p)$ coordinates. In turn, this means that the transformation $(q,\dot q)\rightarrow (q,p)$ must have nonzero Jacobian.

However, it's easy to see this is not the case. We know that $(q,p)=(q,\frac{1}{2}\sin^2 q)$. Thus, the Jacobian is

$$ J=\left|\begin{smallmatrix}1 & 0\\\sin(q)\cos(q) & 0\end{smallmatrix}\right| = 0 $$

That means we CAN'T write $p$ as a function of $q,\dot q$, and there can't be a Hamiltonian $H(p,q)$ that is a function of only the coordinates $(p,q)$ because the coordinates $(p,q)$ don't specify the state of the system.

1. OP's Lagrangian $$L~:=~\frac{\dot{q}}{2}\sin^2(q)~=~\frac{d}{dt}\left(\frac{q}{4}-\frac{1}{8}\sin(2q)\right)\tag{1}$$ is a total time derivative, so the Euler-Lagrange (EL) equation is a triviality $0=0$, cf. e.g. this Phys.SE post, and hence not suitable to determine physical trajectories. In other words, $q$ is a gauge degree of freedom. If $q$ is supposed to be a physical observable, then the Lagrangian (and Hamiltonian) formulations are ill-defined.

2. The Hamiltonian is defined as the Legendre transform of the Lagrangian, and hence formally defined as $$ \begin{align} H(q,p)&:=~ \sup_{v\in\mathbb{R}}(vp-L(q,v))\cr &=~ \sup_{v\in\mathbb{R}}v\left(p-\frac{1}{2}\sin^2(q)\right)\cr &=~ \left\{\begin{array}{l}\infty\text{ if } p~\neq~\frac{1}{2}\sin^2(q),\\ 0\text{ otherwise}. \end{array}\right. \tag{2} \end{align}$$

3. If we instead perform the Dirac-Bergmann analysis, we find that $$p~\approx~\frac{1}{2}\sin^2(q)\tag{3}$$ is a primary (and first class) constraint. First class constraints are a hallmark of a gauge system. The Hamiltonian becomes $$H(q,p,\lambda)~=~\lambda\left(p-\frac{1}{2}\sin^2(q)\right),\tag{4}$$ where $\lambda$ is a Lagrange multiplier. One Hamilton's equation becomes $$ \dot{q}~\approx~\frac{\partial H}{\partial p}~\stackrel{(4)}{=}~\lambda, \tag{5}$$ so not surprisingly the Lagrange multiplier is equal to the velocity. The other Hamilton's equation is trivial $0=0$ once we enforce the constraint (3). The Hamiltonian Lagrangian becomes $$ L_H(q,\dot{q},p,\lambda)~:=~p\dot{q}-H ~\stackrel{(5)}{=}~p\dot{q}-\lambda\left(p-\frac{1}{2}\sin^2(q)\right).\tag{6}$$ As a check, note that the Hamiltonian Lagrangian (6) turns into OP's original Lagrangian (1) if we eliminate/integrate out $\lambda$ and $p$, thereby performing the (singular) Legendre transformation in the reversed direction.

References:

1. M. Henneaux & C. Teitelboim, Quantization of Gauge Systems, 1994.