modulo version of the quadratic formula and Euler's criterion

Consider the quadratic congruence $ax^2+bx+c\equiv 0\pmod{p}$, where $p$ is an odd prime. Multiplying through by $4a$, we obtain the equivalent congruence $$4a^2x^2+4abx+4ac\equiv 0\pmod{p},$$ which, by completing the square, can be rewritten as $$(2ax+b)^2\equiv b^2-4ac\pmod{p}.\tag{$1$}$$ So in order for the original congruence to be solvable, $b^2-4ac$ must be a square modulo $p$. Conversely, but we will not need this, if $b^2-4ac$ is congruent to a square modulo $p$, then we can use $(1)$ to solve the original congruence, in a manner which is essentially the Quadratic Formula.

In our case, $4b^2-4ac=25-64=-39\equiv -2\pmod{37}$. So we want $-2$ to be a quadratic residue of $37$. By Euler's Criterion, which you have been asked to use, this is the case precisely if $$(-2)^{\frac{37-1}{2}}\equiv 1\pmod{37}.$$ Since we are calculating an even power, we don't need to worry about the minus sign. Note that $2^5\equiv -5\pmod{37}$, so $2^{10}\equiv 25\pmod{37}$, and therefore $2^{13}\equiv 200\equiv 15\pmod{37}$. It follows that $2^{18}\equiv -75\equiv -1\pmod{37}$, and therefore by Euler's Criterion $-2$ is not a quadratic residue of $37$. We could also have used the calculator, by computing $2^{18}$, and dividing by $37$: the remainder is not $1$.

You can get this in a simpler way. Since $-1$ is a QR of $37$, we find that $-2$ is a QR iff $2$ is. But $37$ has the shape $8k+5$, and $2$ is a QR of primes of the shape $8k\pm 1$, and a NR of primes of shape $8k\pm 3$.

The modular version of the quadratic formula is, for all practical purposes, just the quadratic formula (provided twice the coefficient of $x^2$ is relatively prime to the modulus). As usual, you wind up looking at $b^2-4ac$, which here is $5^2-(4)(2)(8)=25-64=-39$. Since we're working modulo $37$, we can replace that $-39$ by $-2$ or by $35$, if we find it convenient. If $-39$ is a quadratic residue (a "square") modulo $37$, the congruence has solutions; if not, not. So, presumably you have some way to work out whether $-39$ is a quadratic residue modulo $37$ --- that should be where Euler's criterion comes in.

Hint $\rm\bmod\ 37\:$ it has discriminant $\,-39\equiv -2\:$ which is not a square since, by Euler's criterion

$$\rm\bmod\ 37\!:\,\ (-2)^{18}\!=\, 4\cdot (\color{#C00}{2^8})^2 \equiv 4\cdot 9\equiv -1,\ \ \ by\ \ \ \color{#C00}{2^8}\! = 16^2\! = 256\equiv -3,\ \ \ by\ \ \ 7\cdot 37 = 259$$

Recall: if a quadratic $\rm\:f(x)\in R[x]\:$ has a root in a ring $\rm R,\,$ then its discriminant $\rm\,\Delta\,$is a square in $\rm R.\,$ Said contrapositively, if the discriminant is not a square in $\rm R,\,$ then the quadratic has no root in $\rm R.\,$ The proof, by completing the square, works in any ring $\rm\,R\,$ (so in $ \mathbb Z/37 = $ integers mod $37$), viz.

$$\rm\ \ \ \ \ \color{#0a0}{4a}\:(a\:x^2\! + b\:x + c\, =\, 0)\ \Rightarrow\ (2a\:x+b)^2 =\: b^2\! - 4ac =: \Delta$$

Remark $ $ Conversely if $\rm\,\Delta = d^2\,$ and $\rm\,\color{#0a0}{2a}\,$ is invertible then the above arrow reverses and the roots of $\rm\,(2ax+b)^2= d^2\,$ are roots of $\rm\,4a\,f\,$ so roots of $\rm\,f\,$ (by cancelling $\,\color{#0a0}2$ and $\rm\,\color{#0a0}a),\,$ which yields the classical quadratic formula $\rm\,x = (-b\pm d)/(2a).\,$ In the OP we are working over ring $\rm\, R = \Bbb Z_n,\,$ and recall that $\rm\,\color{#0a0}{2a}\,$ is invertible in $\Bbb Z_n\! \!\iff\! \rm\gcd(2a,n)=1\iff n\,$ is odd and coprime to $\,\rm a.$

Beware that generally a modular quadratic can have more than $2$ roots, e.g. $\,x^2\equiv 1\,$ has the roots $\,x\equiv \pm1,\pm3\pmod{\!8},\,$ and we can find them all by CRT. In fact the iterating the Factor Theorem easily shows that a commutative ring is an integral domain iff polynomials over it have no more roots than their degree.

When learning (modular) arithmetic in new rings it is essential to keep in mind that, like above, any proofs from familiar concrete rings (e.g. $\mathbb Q,\mathbb R,\mathbb C)$ will generalize to every ring if they are purely ring theoretic, i.e. if the proof uses only universal ring properties, i.e. laws that hold true in every ring, e.g. commutative, associative, distributive laws. Thus many familar identities (e.g. Binomial Theorem, difference of squares factorization) are universal, i.e. hold true in every ring. The above implication - that a solvable quadratic over $\rm\,R\,$ has a discriminant being a square in $\rm\,R,\,$ used only commutative ring laws so it remains true in every commutative ring. But we need further hypotheses ($\rm R\,$ is a domain and $\,\rm\color{#a00}{2a}\,$ invertible) in order to go further and obtain the quadratic formula and a proof that those are the only roots.

This is one of the great benefits provided by axiomatization: abstracting the common properties of familiar number systems into the abstract notion of a ring allows one to give universal proofs of ring theorems. It is not necessary to reprove these common ring properties every time one studies a new ring (such reproofs occurred frequently before rings was axiomatized).