How can I make siunitx's \num bold?

You can locally redefine \bfseries to use \fontseries{b}, rather than \fontseries{bx} as usual.

You can also avoid all those phantoms by using more columns.

\documentclass[a4paper]{scrartcl}
\usepackage[ngerman]{babel}
\usepackage[utf8]{inputenc}
\usepackage{amssymb,amsmath}
\usepackage{booktabs, makecell,etoolbox}
\renewcommand\theadfont{\normalsize}
\usepackage[binary-units,group-separator={,}]{siunitx}
\sisetup{per-mode=fraction,
         binary-units=true,
         group-separator = {\,},
         range-phrase=-}
\DeclareSIUnit\pixel{px}
\DeclareSIUnit\float{float}
\DeclareSIUnit\floats{floats}

\begin{document}

\begin{table}[ht]
\renewrobustcmd{\bfseries}{\fontseries{b}\selectfont}
\sisetup{detect-weight,mode=text,group-minimum-digits = 4}

\centering

\small
\addtolength{\tabcolsep}{-1.5pt}% decide at the end
\begin{tabular}{
  @{}
  c
  l
  S[table-format=3.0]@{\,}c@{\,}l@{\,}c@{\,}l
  S[table-format=5.0]
  S[table-format=8.0]
  S[table-format=4.0]@{\,}c@{\,}S[table-format=2.0]@{}>{${}}c<{{}$}@{}S[table-format=2.0]
  @{}
    }
\toprule
\# & Type
   & \multicolumn{5}{c}{\begin{tabular}[t]{l}Filters @\\ Patch size / stride\end{tabular}}
   & {Parameters}
   & {FLOPs}
   & \multicolumn{5}{c}{Output size}
\\
\midrule
  &    Input                    
       &&&&&   &           0       &       0   & 1 & @ & 32 & \times & 32    \\
1 &    Convolution
       & 6 & @ & $5 \times 5 \times 1$ & / & 1 
       & 156 & 307 800 & \bfseries 6 & @ & \bfseries 28 & \times & \bfseries 28 \\
2 &    Scaled average pooling   
       &&& $2 \times 2$ & / & 2
       & 2 & 336 & 6 & @ & 14 & \times & 14 \\
3 &    Convolution
       & 16 & @ & $5 \times 5 \times 6$ & / & 1
       & 2 416 & \bfseries 942 400 & 16 & @ & 10 & \times & 10 \\
4 &    Scaled average pooling
       &&& $2 \times 2$ & / & 2
       & 2 & 1 600 & 16 & @ & 5 & \times & 5 \\
5 &    Fully Connected
        & 120 & \multicolumn{4}{@{}l}{\ neurons}
        & \bfseries 48 120 & 240 000   & 120 \\
6 &    Fully Connected
        & 84 & \multicolumn{4}{@{}l}{\ neurons}
        & 10 164 & 20 580  & 84 \\
7 & Fully Connected (output)
        & 10 & \multicolumn{4}{@{}l}{\ neurons}
        & 850 & 1 730 & 10 \\ \midrule
$\sum$  &&&&&&& 61 710 & 15 144 446 & 9118\\
\bottomrule
\end{tabular}

\end{table}
\end{document}

Not perfectly aligned since bold fonts are wider than non bold. Bold fonts are achieved by siunitx package option detect-weight and use of S column type in third and fourth column:

\documentclass[a4paper]{scrartcl}
\usepackage[ngerman]{babel}
\usepackage[utf8]{inputenc}
\usepackage{amssymb,amsmath}
\usepackage{booktabs, makecell}
\renewcommand\theadfont{\normalsize}
\usepackage[binary-units,group-separator={,}]{siunitx}
\sisetup{per-mode=fraction,
         binary-units=true,
         group-separator = {\,},
         range-phrase=-}
\DeclareSIUnit\pixel{px}
\DeclareSIUnit\float{float}
\DeclareSIUnit\floats{floats}

\usepackage{etoolbox}% <-- for bold fonts
\newcommand{\ubold}{\fontseries{b}\selectfont}% <-- for bold fonts
\robustify\ubold% <-- for bold fonts

\begin{document}

\begin{table}[ht]
    \centering
    \begin{tabular}{cll 
                    S[detect-weight,group-minimum-digits = 4,table-format=5.0]% <-- changed
                    S[detect-weight,group-minimum-digits = 4,table-format=8.0]% <-- changed
                    l}
    \toprule
\# &    Type                     
        &   \thead[tl]{Filters @\\ Patch size / stride} 
            &   {Parameters}    &   {FLOPs}   & Output size  \\\midrule
   &    Input                    
        &   &           0       &       0   & \hphantom{0}1 @ $32\times 32$    \\
1  &    Convolution              
        &   \hphantom{0}6 @ $5 \times 5 \times 1$ / 1 
            &       156         &   307 800 & \hphantom{0}\textbf{6}\,@\,$\mathbf{28\times 28}$    \\
2  &    Scaled average pooling   
        & \hphantom{00 @} $2 \times 2 \hphantom{\times 00}$ / 2 
            &        2          &       336 & \hphantom{0}6 @ $14\times 14$    \\
3  &    Convolution              
        &   16 @ $5 \times 5 \times 6$ / 1            
            &   2 416           & \ubold 942 400% <-- boldface
                                    & 16 @ $10\times 10$ \\
4  &    Scaled average pooling   
        &   \hphantom{00 @} $2 \times 2 \hphantom{\times 00}$ / 2 
            &        2          &     1 600 & 16 @ \hphantom{0}$5\times \hphantom{0}5$   \\
5  &    Fully Connected          
        &   120 neurons                      
            & \ubold 48 120% <-- boldface
                & 240 000   & 120 \\
6  &    Fully Connected          
        &   \hphantom{0}84 neurons           
                &       10 164  &   20 580  & \hphantom{0}84 \\
7  & Fully Connected (output) 
        &   \hphantom{0}10 neurons           
            &               850 &   1 730   & \hphantom{0}10 \\ \midrule
$\sum$  &   &   &        61 710 & 15 144 446& \num{9118}\\
    \bottomrule
    \end{tabular}
\end{table}
\end{document}

Note: use of \bf is depreciated for + 20 years.

Edit:

Meanwhile (after many years and comments below) many of us learned how to managed boldface digits in S column tables. In original answer was not added option mode=text to \sisetup and definition of ubold can also be improved:

...
\si    \usepackage{etoolbox}% <-- for robustify bold fonts in S columns
    \newrobustcmd\ubold{\DeclareFontSeriesDefault[rm]{bf}{b}\bfseries}% <-- for bold fonts

    \begin{document}
    \begin{table}[ht]
        \centering
    \sisetup{detect-weight,
             mode=text,
             group-minimum-digits = 4}
        \begin{tabular}{cll
                        S[table-format=5.0]% <-- changed
                        S[table-format=8.0]% <-- changed
                        l}
setup{per-mode=fraction,
         binary-units=true,
         group-separator = {\,},
         range-phrase=-,
         mode=text    % <---  
         }
...

Considerinmg above in MWE gives desired result: