How would one calculate the cosine of an obtuse angle?

You would use the Law of cosines

Cosine=adjacent/hypotenuse is only true in right triangles, as that is the only time there is a hypotenuse. $\cos \theta$ is well defined for $\theta \gt 90^{\circ}$ and we have $\cos \theta=-\cos (180^{\circ}-\theta)$.