Is moving an "all-in-one" PC when turned on/off dangerous?

Unless the machine is designed for movement (ie, a laptop) the general advice is don't move it while it's running - it is probably OK most of the time. Mechanical hard drives are usually the vunerable component here - a hard drive that's spinning is much more vunerable to even slight impacts, or changes in orientation, than it would be if it was parked and powered down - when it can sustain much stronger impacts without damage. You've already mentioned this, but it's here for completeness.

Movement while correctly powered down should be fine in general, as long as you are gentle. The problem with moving the machine regularly is there is an increase chance of accidental damage caused by poor handling.

• Avoid all impacts (including "dropping" the machine "lightly" on a desk).
• Avoid sudden movements (eg, don't pick it up violently).
• Avoid sudden changes in momentum (eg, don't spin round quickly while carrying it).

In other words, minimise all accelerating forces that you place the machine (and thus its components) under.

As long as you're being moderately careful about avoiding jarring impacts and dropping it, I wouldn't worry about.

The main part that could be damaged by any sort of movement would be the hard drive, but it would take a pretty solid impact to damage it. (If you couldn't move computers at all while they were on I think laptops would be much less popular.)