What is the significance of Lie groups $SO(3)$ and $SU(2)$ to particle physics?

QuantumMechanic's links turn up a dizzying array of meanings for $SU(2)$ in physics, so your question probably turns out to be too broad for a simple answer. Nonetheless, I do like it, and similar questions that grope for pithy overviews of things, so I'll try to answer it with my non particle physicist's understanding.

Probably the "main" meaning of $SU(2)$ you're going to find is as the (or highly nontrivial part of the) gauge group of certain Yang-Mills-kind theories (see Yang-Mills Wiki page), notably:

1. The $SU(2) \times SU(1)$ gauge group of the electroweak interaction (see Wiki page of this name). Here the three orthonormal (wrt the Killing form) Lie algebra basis vectors of $SU(2)$ correspond to the three W-bosons.
2. Isospin Symmetry (see Wiki page "Isospin" The group of approximate symmetries leaving invariant the strong interaction Hamiltonian, as formulated by Heisenberg in 1932. The proton and neutron "live in" the fundamental representation of $SU(2)$ (I'm guessing you're a mathematician - so in case you haven't yet picked this up, physicists are wont to mean the vector space that images of group members under the representation act on as the "representation" - it took me a while to grasp this), whereas the three pions live in the adjoint representation of $SU(2)$, i.e. they are transformed by corresponding members of $SO(3)$. The protons and neutrons have spin $1/2$, being spinors, and they can be thought of as basis vectors for a $\mathbb{C}^2$ state space, which is acted on by a group member $\gamma$ simply through $X\in\mathbb{C}^2\mapsto \gamma\,X$. The three neutral pions are basis vectors in an $\mathbb{R}^3$ state space which is acted on by $Y\in\mathbb{R}^3\mapsto\mathrm{Ad}(\gamma)\,Y$.

In the case of gauge theories, my understanding of their importance (the ones of the Yang-Mills kind with a finite dimensional structure group) is in this answer. If you're a bit slow on the uptake like me, you may need someone to point out to you that "all we're doing" in constructing a gauge theory is building a fibration on a physical observable theory and the gauge group is nothing more than the fibre bundle's structure group: we put hair on a theory and see what beautiful braids we can make with it. (Yes, I really did need someone to point this out explicitly to me, even though I have a reasonable grasp of fibre bundles!) But why do we do this, i.e. on the surface seem to add complexity, when it would seem the aim of physics to simplify things, not kit them with more hair (complexity)? There are two answers here:

1. There is a known classical gauge theory - Maxwell's electromagnetism with the $U(1)$ symmetry - whose curious gauge symmetry we seek to take to other physics, just as a "suck and see" mathematical physics analogy;

2. There are either (i) experimentally observed continuous symmetries or (ii) conserved quantities in physically observed processes, so we add the fibration as a way to beget these symmetries or conserved quantites in theory. In the case of observed conserved quantities, this procedure works through Noether's theorem, but it is important to understand the implication through Noether's theorem is only one way: a Lagrangian with continuous symmetries implies the same number of conserved quantities, but a conserved quantity doesn't needfully imply a continuous symmetry. Again, it is a suck and see approach - we know one way to force a conserved quantity in a theory - to wit: adding a fibration or gauge symmetry - so we try it and see what happens, and it so happens that experimentally the theories built in this way work rather well (the Standard Model).

Other resources that I found helpful - particularly if you haven't already delved into gauge theories - are the following:

1. John C. Baez and John Huerta, "The Algebra of Grand Unified Theories"
2. Gerard 't Hooft's "Lie Groups in Physics"
3. Terrence Tao's blog "What is a gauge?"
4. "Gauge Theory" Wikipedia Page
5. "Introduction to Gauge Theory" Wikipedia Page
6. Summary of Gerard 't Hooft's 1999 Nobel Lecture
7. Relevant chapters in Roger Penrose's "Road To Reality" (I don't have it before me at the moment).

I found the first two papers by Baez/Huerta and 't Hooft invaluable here. Like I said, I am not a particle physicist but after reading this I feel I can at least follow many discussions in this field without too much (let's say < 80%) going over my head. Thanks to John Baez and his wonderful literature, I think that withering away in a nursing home is not going to be too bad, if I can still read by then! (this isn't in the offing BTW). I find almost anything written on physics and its relationship with mathematics by Baez, 't Hooft and Penrose well worth reading. There was (likely still is) an excellent introduction to gauge theory on Gerard 't Hooft's webpage but the webpage itself is a bit hard to navigate and I can't find it at the moment - I guess such disorganisation is inevitable for someone as polymathematic as 't Hooft is wanting to share so much varied material.

But maybe the deepest, simplest and (for me, the most beautiful) meaning of all for $SO(3)$ and $SU(2)$ is the simple relationship between the two groups, the one being the universal cover of the other (see my answer here), as was taught to me indirectly by a seven year old boy (the depth of physical meaning, rather than the universal cover property) when I was demonstrating the Dirac belt trick and cup tricks at my daughter's school and he asked the question "can you make a fancier arrangement of ribbons so that you have to spin her [the doll] three times rather than twice to get back to the start?" (I use a doll on a ribbon rather than just a marked card with children because, as social animals, we're hard wired to ken a face, so keeping track of the spins is unmistakable with a doll. Many smallish children of about six years old and over find the belt trick really enthralling, BTW.)

I was thoroughly impressed by his question and wished I could answer it better for him. But as far as particles are concerned, the answer is the same: an emphatic no: there are only half integer spins, not spin $1/3$ and so forth, because $SU(2)$ is the universal cover of $SO(3)$. There are only bosons and fermions in the World, and the double cover relationship between $SO(3)$ and $SU(2)$ is why - "a simply connected topological space admits no non trivial coverings" to quote from W. S. Massey, "Algebraic Topology: An Introduction" - so the universal cover is the whole gig! The belt trick works because the evolution of the Serret-Frenet frames along the twisted ribbon encodes a continuous path through $SO(3)$ from the identity to the transformation defined by the doll's orientation in space and so the ribbon precisely encodes the homotopy class of this path. If you can loop it over the doll (deform the path continuously) and undo the twists, the ribbon is still encoding the same homotopy class. The belt trick is a precise physical analogy to the mathematical procedure for building a universal cover. So this humble observation about the relationship between $SO(3)$ and $SU(2)$ explains all the following:

1. There are no other spin $1/3$, or any $1/N$ aside from $1/2$, ribbons realisable in a Dirac belt trick;

2. Spinors and tensors exhaust the list of everything that transforms compatibly with rotations. Actually the idea broadens from the $SO(3)$ with $SO(2)$ relationship to general proper Lorentz transformations: we add boosts to the mix and get the identity connected component of the Lorentz group $O(3,1) \cong PSL(2, \mathbb{C}) \cong \operatorname{Aut}(\mathbb{C})$ (the latter being the group of invertible Möbius transformations) and the double cover of this beast is $SL(2, \mathbb{C})$, so spinors and tensors exhaust the list of everything that transforms compatibly with rotations, boosts and general combinations thereof; and

3. There are only bosons and fermions - i.e. only particles with half integer or whole number spins in the World.

Truly I find this simple relationship is a little jewel to behold. There is a footnote in Chapter 17 of the third volume "the Feynman Lectures on Physics" where Feynman says he had been trying to find a simple demonstration that there are only half integer spins and had failed - "We'll have to talk about it with Prof. Wigner, who knows all about such things"!, he ends the footnote. I rather think Feynman, from what I know of his work and sense of humour, would have been delighted to have the explanation suggested to him by a seven year old, were he alive.

Lastly, I'd just like to mention how $SU(2)$ and $SO(3)$ show up in my own field of optics and electromagnetism. It's not quite what people wontedly mean by "particle physics" but it is an application in the physics of the photon. The general polarisation state of a one-mode electromagnetic field $\Psi = \left(\begin{array}{c}\psi_+\\\psi_-\end{array}\right)$ can be encoded in two complex amplitudes, one for each circular polarisation's basis state (or, amplitudes of the two Riemann-Silberstein vectors for a given wavevector more as discussed in my answer here ). A lossless polarisation transformer (waveplate, mirror system, and so forth) must impart a general unitary transformation on these two amplitudes, for the sum of their square magnitudes is the wave's power. Often, we're not worried about phase that is common to both polarisation eigenstates, so we can think of the matrix of our polarisation transformer as living in $SU(2)$ rather than $U(2) \cong SU(2) \otimes U(1)$, but the Jones calculus actually handles $U(2)$ as well:

$$\Psi \mapsto \Psi^\prime = \mathbf{U}\,\Psi;\;U\in SU(2)$$

In this context, $\mathbf{U}$ is called the transformer's Jones Matrix. We can also represent the polarisation state by the Stokes parameters:

$$s_j(\Psi) = \Psi^\dagger \sigma_j \Psi$$ $$\begin{array}{l} s_0 = \Psi^\dagger\,\Psi = |\Psi|^2\\ s_1 = 2 \operatorname{Re}(\psi_+^*\,\psi_-)\\ s_2 = 2 \operatorname{Im}(\psi_+^*\,\psi_-)\\ s_3 = |\psi_+|^2 - |\psi_-|^2 \end{array}$$

where $\sigma_j$ are the Pauli spin matrices (here $\sigma_j;\,j=1,\,2,\,3$ are the matrices on the Pauli Matrix Wiki page and $\sigma_0$ is the $2\times2$ identity); $i\,\sigma_j;\,j=1,\,2,\,3$ of course span $\mathfrak{su}(2)$ and $i\,\sigma_j;\,j=0,\,1,\,2,\,3$ span $U(2)$. This definition of the Stokes parameters is slightly different to that wontedly given in optics (e.g. section 1.4 of Born and Wolf, "Principles of Optics" sixth edition ); there is an unimportant sign switch and a renumbering. The Pauli spin matrices $i \sigma_1,\,i \sigma_2,\, i \sigma_3$ are a basis for $\mathfrak{su}(2)$ and $U$ can be written as $U = \exp(-i \theta \sum \gamma_j \sigma_j/2);\;\theta,\,\gamma_j\in\mathbb{R},\;\sum\gamma_j^2 = 1$. If the system input is $\Psi$, then, after transformation by $U$, its Stokes parameters are transformed by the spinor map:

$$s_k = \Psi^\dagger U^\dagger \sigma_k U \Psi = \Psi^\dagger U^{-1} \sigma_k U \Psi = - i \Psi^\dagger \exp\left(i \frac{\theta}{2} \sum_j \gamma_j \sigma_j\right) i \sigma_k \exp\left(-i \frac{\theta}{2} \sum_j \gamma_j \sigma_j\right) \Psi $$

or, alternatively, the unit sphere of Stokes vectors $(s_1,\,s_2,\,s_3)^T$ is transformed by precisely the rotation $\exp(\theta \,\mathbf{H})$ matrix corresponding to $\mathbb{U}$ when the latter is mapped by the standard adjoint representation homomorphism:

$$\exp(\theta \,\mathbf{H}) = \exp\left(\theta \left(\begin{array}{ccc}0&\gamma_z&-\gamma_y\\-\gamma_z&0&\gamma_x\\\gamma_y&-\gamma_x&0\end{array}\right)\right)$$

so that we can visualise polarisation state changes as rotations of the unit sphere, as long as we are happy to be blind to the difference between a transformation $\mathbf{U}$ and its negative $-\mathbf{U}$, i.e. we are happy to see only cosets of this homomorphism's kernel.

A slight generalisation of this procedure is to use Mueller Calculus(see Wiki page "Mueller calculus", which is the density matrix notation in disguise and can deal with partially polarised light states, which are classical statistical mixtures of pure quantum states. I describe this aspect of the Mueller Calculus in my answer here.

I will insert here an experimentalists introduction to SU(2) and SU(3).

Back in the sixties we were organizing the exciting resonance data we got from a multiplicity of experiments into spin Regge poles . Spin was organized in SU(2) multiplets since nuclear physics studies, and the analogue was recognized in isotopic spin ( proton up, neutron down) and used extensively for the new particles discovered. So spin and isotopic spin, i.e. SU(2) had an early significance for experimental physics, on which any more extensive theories could be based.

Then came the revelation of the eightfold way,

The meson octet. Particles along the same horizontal line share the same strangeness, s, while those on the same diagonals share the same charge, q.

It was very exciting at the time to see that the mesons, painstakingly organized into SU(2) multiplets, had an extra symmetry when a new quantum number was used, a symmetry that was beautifully fulfilled by the multiplets of SU(3) groups. Groups became very important to the organization of physics data, and were inputs to the theoretical foundations that might describe those data after 1970. Thus higher groups, like SO(3), were explored and/or utilized .