How can I properly write this equation in Latex?

My welcome to TeX.SE and you here have a my proposal done as pure LaTeX. There is a bit of difference from your image because there is much text in a specific part of the formula (Maximum predicted reward, given new state and all possible actions).

Some small suggestions:

1. In LaTeX there is a specific command to indicate the maximum and it is \max that is not written in italics (max) like each letter is a variable. The same discussion for "New", which I wrote as text into a formula \text{New}, not in italics.
2. An important indication: if you change something in the formula that you have inserted as an image, the spaces may change. This can be easily resolved by reducing or increasing the distance with the command \mkern-30mu (for example).

\documentclass[a4paper,12pt]{article}
\usepackage{amssymb,mathtools}
\begin{document}
\[\underbrace{\text{New}Q(s,a)}_{\scriptstyle\text{New Q-Value}}=Q(s,a)+\mkern-34mu\underset{\text{New Q-Value}}{\underset{\Bigl|}{\alpha}}\mkern-30mu[\underbrace{R(s,a)}_{\scriptstyle\text{Reward}}+\mkern-30mu\underset{\text{Discount rate}}{\underset{\Biggl|}{\gamma}}\mkern-75mu\overbrace{\max Q'(s',a')}^{\scriptstyle\substack{\text{Maximum predicted reward, given} \\ \text{new state and all possible actions}}}\mkern-45mu-Q(s,a)]\]
\end{document}

An alternative, with use of tikzmark library:

\documentclass{article}
\usepackage{mathtools}
\usepackage{tikz}
\usetikzlibrary{tikzmark}

\usepackage{lipsum}
\begin{document}
\lipsum[66]
\[
\underbrace{\text{New }Q(s,a)}_{\substack{\text{New}\\
                                          \text{Q-Value}}}
    = \underbrace{Q(s,a)}_{\substack{\text{Current}\\
                                     \text{Q-Value}}}
    + \tikzmarknode{A}{\alpha}
    \Bigl[
        \underbrace{R(s,a)}_{\text{Reward}}
    + \tikzmarknode{B}{\gamma}
        \overbrace{\max Q'(s',a')}^{\mathclap{%
            \substack{\text{Maximum predicted reward, given} \\
                      \text{new state and all possible actions}}
                                            }}
    - Q(s,a)
    \Bigr]
\begin{tikzpicture}[overlay, remember picture,shorten <=1mm, font=\footnotesize, align=center]
\draw (A.south) -- ++ (0,-.8) node (C) [below] {Learning\\ rate};
\draw (B.south) -- (B |- C.north)  node[below] {Discount\\ rate};
\end{tikzpicture}
\vspace{4ex}
\]
\lipsum[66]
\end{document}

After two compilation the result is: