SU(3) vs SO(3) color gauge

My sense is that you might first need to appreciate why the SU(3) color group is just right for the quark model with flavor SU(3). An easy way to see it, pretty much the historical route, is to consider why the $\Delta^{++}$ in the baryon decuplet can exist. It has spin 3/2, so it is uuu with all 3 quark spins up. But then, spin-wise this is symmetric, whereas, quarks being fermions, its wavefunction must be totally antisymmetric. In our world, this is arranged by giving these quarks a color, say RGB, and making them color triplets, in the fundamental rep of color SU(3).

When you compose three color triplets in SU(3) you get $$ \mathbf{3}\otimes\mathbf{3}\otimes\mathbf{3}=\mathbf{10}_S\oplus\mathbf{8}_M\oplus\mathbf{8}_M\oplus\mathbf{1}_A, $$ where the subscripts denote the symmetry of the representation (symmetric, mixed, or antisymmetric). All we care about here is that the last term, the singlet, the colorless hadron, is antisymmetric. (It corresponds to the Young tableau consisting of a tower of three stacked boxes, if you care to visualize it.) As a consequence, the spin-color wavefunction is antisymmetric, just the ticket for fermions, so it exists.

Now, you ask, what if the quarks were color triplets, but now in the adjoint of SO(3), instead? Is there a fully antisymmetric singlet in the product? Well, you know how to compose SO(3) reps: what do you get by combining three spin-1s? (I write the dimensions 2 j +1 of the reps, not the spins, for comparison): $$ \mathbf{3}\otimes\mathbf{3}\otimes\mathbf{3}=(\mathbf{5}_S\oplus\mathbf{3}_A\oplus\mathbf{1}_S)\otimes\mathbf{3}= \mathbf{7}_S\oplus\mathbf{5}_M\oplus\mathbf{3}_M\oplus\mathbf{5}_M\oplus \mathbf{3}_A +\mathbf{1}_M+ \mathbf{3}_M $$ The upshot is that the singlet is of mixed symmetry, so the $\Delta^{++}$ does not exist, and, so, at the very least, the baryon decuplet doesn't... No $\Omega^-$, then. Bummer...

I would not spoil the fun of working out what types of multiplets might or might not exist for SO(3)... Any decent book on group theory or the WP articles linked might lead you to explore your question.

From a physical perspective if QCD were $SO(3)$, or as I will argue really $SU(2)$, it might be a strong coupling version of the weak interactions. This is because $SU(2)$ is a double cover over $SO(3)$. You also may not have color triplet, but more like color doublets. Why might this be the case? $SO(3)$ has $3$ dimensions. One outstanding problem is the family problem: The dimension of the gauge group, and thus the number of gauge bosons, is the same as the fermion sector. This would be a reduced number of quarks. There are some exotic physics such as possibly color superconductivity that have flavor-color mode locking.