Why $3$-dim isotropic harmonic oscillator's symmetry is not $O(6)$ but $U(3)$?

The statement holds both in the classical and the quantum case:

Clasical mechanics

The symmetry group of the function $H:\mathbb{R}^3\times\mathbb{R}^3\rightarrow\mathbb{R}$ given by $H(q,p)=\sum_{i=1}^3(q_i^2+p_i^2)$ is indeed $O(6)$, as you guessed. However, when talking about symmetries an extra condition is needed: they should preserve the symplectic structure.

To understand what is meant by this, remember that Hamilton's equations are $\dot{q_i}=\partial H/\partial p_i$ and $\dot{p_i}=-\partial H/\partial q_i$. They may be written as \begin{equation} \left(\begin{array}{c}\dot{q}\\\dot{p}\end{array}\right)= \left(\begin{array}{cc}0 & I \\ -I & 0\end{array}\right) \left(\begin{array}{cc}\frac{\partial H}{\partial q} \\ \frac{\partial H}{\partial p}\end{array}\right) \end{equation} where $I$ is the $3\times 3$ identity matrix. Thus, if we want the form of Hamilton's equations to be invariant under a transformation \begin{equation} \left(\begin{array}{c}q\\p\end{array}\right)\mapsto M\left(\begin{array}{cc}q \\ p\end{array}\right) \end{equation} it should satisfy $M^T \Omega M = \Omega$ where \begin{equation} \Omega = \left(\begin{array}{cc}0 & I \\ -I & 0\end{array}\right). \end{equation}

The group of such matrices $M$ is the symplectic group $Sp(6,\mathbb{R})$. The group of symmetries should now be the intersection of $O(6)$ and $Sp(6,\mathbb{R})$. Using the general property $U(n)=O(2n)\cap Sp(2n,\mathbb{R})$ we get that the group we are looking for is $U(3)$.

Quantum mechanics

An explanation of why the symmetry group can't be $O(6)$ (in a more general case) can be found in the answer to this question, as pointed out in the comments.

Now we want to see that the symmetry group is $U(3)$. We require the invariance of $H$ and of commutation relations $[q_i,q_j]=[p_i,p_j]=0$, $[q_i,p_j]=i\delta_{ij}$, which can be written collectively as $[(q,p)^T,(q,p)]=i\Omega$. Therefore the commutator preserving transformations are $Sp(2,\mathbb{R})$ and by the same reasoning as before the group of symmetries is $U(3)$.