How can we experimentally tell the difference between particles with and without rest mass?

I'm just going to answer the title question as asked.

There are at least three categories of ways to detect non-zero mass for particles, and each has variations.

The basic methods are

• Measure the kinetic energy to momentum ratio ($T/p$).

• Measure the speed. Any value well distinguished from $c$ implies mass.

• Creation or decay kinematics.

• Observe mixing.

Spectrometer-Calorimeter

(Spectrometers measure momentum; calorimeters measure energy)

Relativity makes it clear that the ratio of total energy to momentum of a particle is $R_0 = T/p = E/p = c$ for a massless particle and $R_m = \frac{\gamma-1}{\beta\gamma} c$ for a particle with mass.

Reliably distinguishing that ratio is easy for particles with large mass.

Speed

If a particle is massless it moves at $c$; if massive it move at less than $c$, so any measurement of speed less than $c$ implies non-zero mass. Getting a value for that mass requires a little more work.

Speed can be measured

• By time-of-flight, either between two time-resolving detectors or from a known creation time to a single time-resolving detector. For charged particle it is easy to build detectors with nano-second time resolution, so this a straight-forward for particles even as light as electrons.

• For charged particles with a velocity-threshold detector such as a Cerenkov or transition radiation detector.

For neutral particle you have to get them to interact with a charged object in your detector to spot them, which makes this more difficult. None the less, accelerator neutrino speeds can be constrained to be very close to that of light with existing hardware.

Creation/Decay Kinematics

The conservation of four-momentum at a creation or decay vertex means that with good enough information on the motion of all the involved particle and if all the masses but one are known you can find the final mass.

This is easiest to see in a creation context where the annihilation of a particle with its anti-particle $$ e^+ + e^- \longrightarrow X + \bar{X} \;,$$ can only proceed if the total (center-of-mass) energy is at least twice the mass of species $X$. Actual measurements generally involve the shape of the production cross-section versus total (CoM) energy rather than seeking the actual threshold where the production rate vanishes.

Attempts to obtain the neutrino masses this way have, so far, been frustrated by the difficulty of the experiment and the low rate near the end-point. However, new measurements are contemplated.

Mixing

This is the means by which we know neutrinos must have some mass. In essence, mixing requires time and no proper time passes between points on a luminal trajectory, so anything that mixes can't be following luminal trajectories and therefore must have mass.