What does $(a,b)_{\zeta}$ correspond to in $\mathrm{Br}(\mathbb{Q}_p)=\mathbb{Q}/\mathbb{Z}$

Qiaochu Yuan's answer gives good insight into computing the Hilbert symbol here, giving the result as an element of $\mu_{p-1}(\Bbb Q_p)$; but it avoids the machinery of cyclic (division) algebras which are sort of the original setting to define the Hasse invariant as element of $Br(\Bbb Q_p) \simeq \Bbb Q/\Bbb Z$. I want to amend that. It turns out that in the example at the end of his answer, whether the result is $\frac14$ or $\frac34$ depends on (some conventions, and:) which primitive $(p-1)$-th root of unity $\zeta$ you choose in the original question.

References: I learned a lot of this stuff from R. Pierce: Associative Algebras (GTM 88, Springer, 1982) and I. Reiner: Maximal Orders (LMS Monographs 5, Academic Press, 1975) some years ago, although I don't have them at hand by now, so I cannot even guarantee that I follow the exact same conventions.

Namely, let us restrict to $p \neq 2$ for simplicity, and to the basic interesting case $\nu(a) = 0, \nu(b) =1$, i.e. $a\in \Bbb Z_p^\times \simeq \mu_{p-1} \times (1+\Bbb Z_p)$. Since elements in the second factor have $(p-1)$-th roots, we can easily restrict to the case $a \in \mu_{p-1}$; however, for your definition of the cyclic algebra to work, I think we should further restrict to the case that $a$ is a primitive $(p-1)$-th root of unity. Namely, now one would usually define the cyclic algebra as follows: Let $r \in \overline{\Bbb Q_p}$ satisfy $r^{p-1}=a$. The extension $L= \Bbb Q_p(r) \vert \Bbb Q_p$ is the unique unramified extension of degree $p-1$; its Galois group is cyclic and has a distinct generator, namely the (lift of the) Frobenius, let's call it $\sigma$. Then the cyclic algebra you define can be realised as a subalgebra of $M_{(p-1) \times (p-1)}(L)$, generated by

$x:= \pmatrix{r&0& &\\ 0&\zeta^{-1} r& &\\ && \ddots &0\\ &&0&\zeta^{2-p}r}$ and $y:= \pmatrix{0&1&0& &\\ 0&0&1 &&\\ &&\ddots& \ddots &\\ &&& &1\\ b&&&0&0}.$

Note that we have $x^{p-1}=a$ and $y^{p-1}=b$ as well as $yxy^{-1}= \zeta^{-1}x$, as demanded. One checks that this is a division algebra, which contains as a subfield the diagonal matrices

$$\pmatrix{z&0& &\\ 0&\tau(z)& &\\ && \ddots &0\\ &&0&\tau^{p-1}(z)}$$

where $z \in L$ and $\tau \in Gal(L\vert \Bbb Q_p)$ is the automorphism induced by $r\mapsto \zeta^{-1}r.$ We identify this subfield with $L$. This division algebra, which in this case corresponds exactly to your definition of $(a,b)_\zeta$, is usually denoted by something like $(L\vert \Bbb Q_p, \tau, b)$.

Now the original definition of the Hasse invariant goes like this: One can extend the $p$-adic valuation $\nu$ to a valuation of the division algebra. Note that e.g. $\nu(x) = 0$ and $\nu(y) = \dfrac{1}{p-1}$. In the division algebra, there exist elements $\gamma$ such that the Frobenius $\sigma$ on $L$ is induced by conjugation with $\gamma$, i.e.

$$\gamma z \gamma^{-1} = \sigma(z) \text{ for all } z \in L . (*)$$

Then the Hasse invariant is (well-)defined as $\nu(\gamma) + \Bbb Z \in \Bbb Q/ \Bbb Z$.

A quick calculation shows that $\sigma(r) = ar$. Now find $1\le k \le p-1$ such that $\tau^k = \sigma$, or in other words, $\zeta^{-k} = a$. Then one checks that $\gamma := y^k$ satisfies $(*)$ and hence

$$(a,b)_\zeta = \dfrac{k}{n} + \Bbb Z \in \Bbb Q / \Bbb Z.$$

In the example $a=2, b=p=5$ of the other answer, it depends: There are two primitive $4$-th roots of unity in $\Bbb Q_5$, namely $\omega(2)$ and $\omega(3)$ ($\omega$ denoting the Teichmüller map). With your definitions we have

$$(2,5)_{\omega(2)} = \dfrac{3}{4} + \Bbb Z$$ $$(2,5)_{\omega(3)} = \dfrac{1}{4} + \Bbb Z$$

Unravelling many restrictions we made:

For $a \in \Bbb Z_p^\times, b \in p \Bbb Z_p$, we have $$(a,b)_{\zeta} = \dfrac{k}{p-1} + \Bbb Z \in \Bbb Q/\Bbb Z \simeq Br(\Bbb Q_p)$$ where $\zeta^{-k} \equiv a$ mod $p$.

And from there one easily generalises to (consistent with the formula in Qiaochu's answer):

For $a \in \Bbb Z_p^\times, b \in \Bbb Q_p$, we have $$(a,b)_{\zeta} = \nu(b)\dfrac{k}{p-1} + \Bbb Z \in \Bbb Q/\Bbb Z \simeq Br(\Bbb Q_p)$$ where $\zeta^{-k} \equiv a$ mod $p$.

(The only class field theory that goes into this case is basic Kummer theory, relating $\sigma$ and $\tau$ and $\mu_{p-1}$.)

Note that if one rewrote conjugation in $(*)$ as $\gamma^{-1}z\gamma$, everything would be multiplied by $(-1)$ in $\Bbb Q/\Bbb Z$ which in Qiaochu's example would exactly flip the two values $\frac14$ and $\frac34$. So in a way, it's really a convention which one is which. However, if one looks at the full Brauer group instead of just the $p-1$-torsion part as here, one indeed has to fix such a convention.

I should clarify going into this that I don't know any class field theory, so I might say some silly things. The number you're trying to compute is the Hasse invariant of the cyclic algebra, and it should also correspond to a Hilbert symbol as in the comments. I found a nice formula for a special case of the Hilbert symbol in Section 3.3 of these notes (Lemma 3.22). Specialized to this case it gives

$$(a, b)_{\xi} = \omega \left( (-1)^{\nu(a) \nu(b)} \frac{b^{\nu(a)}}{a^{\nu(b)}} \bmod p \right)$$

where $\nu : \mathbb{Q}_p \to \mathbb{Z}$ is the valuation and $\omega : \mathbb{F}_p^{\times} \to \mu_{p-1}(\mathbb{Q}_p)$ is the Teichmüller character.

You asked in the comments about $p = 5, a = 2, b = 3$. In this case $\nu(a) = \nu(b) = 0$ so the Hilbert symbol is trivial. In order for it to be nontrivial at least one of $\nu(a)$ and $\nu(b)$ must be nonzero. For example, sticking to $p = 5$, if we take $a = 2, b = 5$ then we get

$$(1, 5)_{\xi} = \omega \left( \frac{1}{2} \bmod 5 \right) = \omega(3)$$

which is the unique $4^{th}$ root of unity in $\mathbb{Z}_5$ congruent to $3 \bmod 5$, and in particular primitive. Unfortunately I don't know exactly how this is identified with an element of $\mathbb{Q}/\mathbb{Z}$, although it must be either $\frac{1}{4}$ or $\frac{3}{4}$.