Is there a non-degenerate quadratic form on every finite abelian group?

Yes. It's necessary and sufficient to show that every finite abelian group admits a nondegenerate quadratic form valued in a finite cyclic group. The following slightly stronger statement is true: every finite abelian $p$-group admits a nondegenerate quadratic form valued in $C_{p^k}$ for some $k$ (this suffices by the Chinese remainder theorem). So let's prove this.

If $A$ is a finite abelian $p$-group for $p$ an odd prime, we can pick any isomorphism $A \cong A^{\ast}$ from $A$ to its Pontryagin dual $A^{\ast} = \text{Hom}(A, \mathbb{Q}/\mathbb{Z}) \cong \text{Hom}(A, \mathbb{Q}_p/\mathbb{Z}_p)$ and we'll get a nondegenerate bilinear form $B : A \times A \to C_{p^k}$ whose associated quadratic form $Q : A \to C_{p^k}$ is nondegenerate. As you say in the comments, this almost but doesn't quite work when $p = 2$.

When $p = 2$ the following slight modification works. Again pick an isomorphism $A \cong A^{\ast}$ to the Pontryagin dual and get a nondegenerate bilinear form $B : A \times A \to C_{2^k}$. Now we do something a bit funny. Consider the inclusion (not a group homomorphism!) $C_{2^k} \to C_{2^{k+1}}$ given by $k \mapsto k$, thinking of elements of $C_n$ as elements of $\mathbb{Z}/n$. Composing this inclusion with $B$ gives a map (not a bilinear map!) $B' : A \times A \to C_{2^{k+1}}$. Now I claim that the diagonal $Q(a) = B'(a, a)$ of this map is a nondegenerate quadratic form. We clearly have $Q(-a) = B'(-a, -a) = B'(a, a)$, and the associated bilinear form $Q(a + b) - Q(a) - Q(b)$ recovers $B$, now taking values in $2 C_{2^{k+1}} \cong C_{2^k}$.

In particular, when $A = C_2$ we get the quadratic form $Q : C_2 \to C_4$ given by $Q(0) = 0, Q(1) = 1$. This quadratic form can be interpreted as a cohomology class in $H^4(B^2 C_2, C_4)$ and so in turn a cohomology operation $H^2(-, C_2) \to H^4(-, C_4)$ which I believe is exactly the Pontryagin square.

Thanks to the Fundamental Theorem of Abelian Groups, let $$G:=\prod_{k=1}^{n}\{z:z^{m_k}=1\,,z\in\mathbb{S}\}\,,$$ and let $\chi(m)=2$ if $m$ is odd and $\chi(m)=1$ if $m$ is even. Then define $$q\colon G\to\mathbb{C}^*\,,~\,~\,~\,(e^{2\pi i\frac{a_k}{m_k}})_k\mapsto \exp(\pi i\sum_k \chi(m_k)\frac{a_k^2}{m_k})\,.$$ We observe that $q$ is a quadratic form because $$e^{\chi(m)\pi i\frac{(a+m)^2}{m}}=e^{\chi(m)\pi i\frac{a^2}{m}}e^{2\pi i(a\chi(m)+\frac{m\chi(m)}{2})}= e^{\chi(m)\pi i\frac{a^2}{m}}\,.$$ The associated bi-character then becomes $$B((e^{2\pi i\frac{a_k}{m_k}})_k,(e^{2\pi i\frac{b_k}{m_k}})_k)= \exp(2\pi i\sum_k \chi(m_k)\frac{a_kb_k}{m_k})\,.$$ Since the degeneracy of the bi-character $B$ is equivalent to $\sum_k\chi(m_k)\frac{a_kb_k}{m_k}\in\mathbb{Z}$ say for all admissible $(b_k)_k$, this forces $a_k\equiv 0\mod m_k$; thus $B$ is non-degenerate!