For a connected, simply connected Lie group what does the Lie algebra tell me?
One thing to realize is that the group 'actually acting' in the adjoint representation is often not the group itself, but one of its, not-simply connected quotients. Your example is excellent: the simply connected group is $SU(2)$ but the group acting on $\mathbb{R}^3$ in the adjoint representation is $SO(3)$: the group elements $I$ and $-I$ in $SU(2)$ act the same.
Related example: the Universal Cover of $SL_{2}(\mathbb{R})$ has no faithful finite-dimensional representation so the group acting on $\mathbb{R}^3$ in the adjoint representation is $PSL(2, \mathbb{R})$.
However this can be a blessing in disguise. The group that acts in the adjoint representation can be found by using the ordinary matrix exponential on the adjoint representation of the Lie algebra.