Do you feel more impact force when falling straight down or tipping over?

Both you and your ancestors are wrong. But I bet you would never guess the real answer!

Assuming the base of the ladder doesn't slide you have a rotating system. Just like a freely falling man you convert potential energy to kinetic energy, but for a rotating system the kinetic energy is given by:

$$ T = \frac{1}{2}I\omega^2 $$

where $I$ is the angular momentum and $\omega$ is the angular velocity. Note that $v = r\omega$, where $r$ is the radius (i.e. the length of the ladder). We'll need this shortly.

Let's start by ignoring the mass of the ladder. In that case the moment of inertia of the system is just due to the man and assuming we treat the man as a point mass $I = ml^2$, where $m$ is the mass of the man and $l$ the length of the ladder. Setting the change in potential energy $mgl$ equal to the kinetic energy we get:

$$ mgl = \frac{1}{2}I\omega^2 = \frac{1}{2}ml^2\omega^2 = \frac{1}{2}mv^2 $$

where we get the last step by noting that $l\omega = v$. So:

$$ v^2 = 2gl $$

This is exactly the same result as we get for the man falling straight down, so you hit the ground with the same speed whether you fall straight down or whether you hold onto the ladder.

But now let's include the mass of the ladder, $m_L$. This adds to the potential energy because the centre of gravity of the ladder falls by $0.5l$, so:

$$ V = mgl + \frac{1}{2}m_Lgl $$

Now lets work out the kinetic energy. Since the man and ladder are rotating at the same angular velocity we get:

$$ T = \frac{1}{2}I\omega^2 + \frac{1}{2}I_L\omega^2 $$

For a rod of mass $m$ and length $l$ the moment of inertia is:

$$ I_L = \frac{1}{3}m_Ll^2 $$

So let's set the potential and kinetic energy equal, as as before we'll substitute for $I$ and $I_L$ and set $\omega = v/l$. We get:

$$ mgl + \frac{1}{2}m_Lgl = \frac{1}{2}mv^2 + \frac{1}{6}m_Lv^2$$

and rearranging this gives:

$$ v^2 = 2gl \frac{m + \frac{1}{2}m_L}{m + \frac{1}{3}m_L} $$

and if $m_L \gt 0$ the top of the fraction is greater than the bottom i.e. the velocity is greater than $2gl$. If you hold onto the ladder you actually hit the ground faster than if you let go!

This seems counterintuitive, but it's because left to itself the ladder would rotate faster than the combined system of you and the ladder. In effect the ladder is accelerating you as you and the ladder fall. That's why the final velocity is higher.