What prevents this magnetic perpetuum mobile from working?

You do not need to invoke friction. The magnetic forces are in equilibrium by themselves so if you place the magnets in that configuration, they will not spontaneously begin to move. The reason is that there is a corresponding force on the magnets when they are vertical that matches the ones you've already drawn.

Let me make a simple model. First of all, start by upping the game and including two big magnets, which can only make it better:

If red is a north pole, then each rotating magnet, when horizontal, has a north pole repelling its backside north pole and a north pole attracting its front south pole. Focusing on the big magnets for the moment, the fact that their north poles face each other suggests that we can trade them for a pair of anti-Helmholtz coils. This means the important character of their field is its quadrupolar nature, and we can approximate the magnetic field as $$\mathbf B=\frac{B_0}a\begin{pmatrix}x\\y\\-2z\end{pmatrix},$$ where the $z$ axis goes from one big magnet to the other, $a$ is some characteristic length, and $B_0$ is some characteristic field strength.

Now, for the little magnets, I think it is uncontroversial to model them as point dipoles. If $\theta$ is the angle the wheel spoke makes with the $x$ axis (with the wheel in the $x,z$ plane) then each magnet is a dipole with moment $$\mathbf m=m \begin{pmatrix}-\sin(\theta)\\0\\\cos(\theta)\end{pmatrix} \text{ at } \mathbf r=R \begin{pmatrix}\cos(\theta)\\0\\\sin(\theta)\end{pmatrix}.$$ With this, the potential energy of each spoke magnet is $$U=-\mathbf m\cdot\mathbf B=3\frac R a mB_0\sin(\theta)\cos(\theta)=\frac32\frac Ra mB_0\sin(2\theta).$$

To see how this behaves, here is a colour plot of the energy, with negative energy in red and positive energy in blue.

You can see there is a gradient pointing up on the right and down on the left. However, these are matched by clockwise gradients when the spokes are vertical. A single magnet will settle on the lower left or the upper right; a pair of magnets will settle on that diagonal. For a symmetrical wheel with three or more magnets, the total potential energy is flat at zero, $$U=\sum_{k=1}^n\frac32\frac Ra mB_0\sin\left(2\left(\theta_0+\frac{2\pi}{n}k\right)\right) =\frac3{4}\frac Ra mB_0\text{Im}\left[e^{2i\theta}\sum_{k=1}^n(e^{2\pi i/n})^k\right] =\frac3{4}\frac Ra mB_0\text{Im}\left[e^{2i\theta+2\pi i/n}\frac{1-(e^{2\pi i/n})^n}{1-e^{2\pi i/n}}\right] =0 $$ and there is no resultant magnetic force.

I will try to explain what I think is the solution. It is obviously far from complete, it is just an idea, a "progress report". So I will not select it as the correct answer, unless I will be able to finish it.

To explain the solution, I will refer to the following image, which is a simplified version of the original device from the question. It looks more complicated, but I think it makes the main idea simpler to understand.

Each magnet moves along the band ABCD, cyclically. Each magnet, when moving between A and B, and between C and D, doesn't rotate, but it is accelerated by the magnetic force. This is obvious, as anyone who played with magnets knows. Therefore, if the movement is slowed down somewhere, it has to be when moving between B and C, and between D and A, as seen in the picture below.

Between the points B-C, respectively D-A, the magnet rotates. Some of the field lines joining it with the large magnet are broken, and others are created. If we put a small magnet near a large one, we see that it changes its orientation, and locks itself along a preferred orientation. We also know this by looking at the needle of a compass in the magnetic field of the Earth. So, my guess is that it is there where the kinetic energy accumulated when moving along AB and CD is consumed. But I don't know how to calculate the balance. Anyway, my guess is that the system will find a preferred position and lock into it, before being able to make at least a complete rotation.