Why do non-Newtonian fluids go hard when having a sudden force exerted on them?

A quick comment on your terminology. The description "non-Newtonian" just means the stress/flow rate graph is not linear i.e. there isn't a single constant viscosity coefficient. The fluid you describe is what we colloid scientists call "dilatant", and it is certainly non-Newtonian. However there are lots of other non-Newtonian fluids such as tomato ketchup and shampoo that behave in different ways. See Are there good home experiments to get a feel for the behavior of yield-stress liquids? for a related question.

Anyhow, kleingordon has explained why the dilatant effect occurs, but let me try a slightly different approach to the explanation.

Oobleck is a suspension of solid (starch) particles in water. Suppose you had a very dilute suspension i.e. lots of water and a little starch. In this case the spacing between the starch grains is large so the grains can flow around without hitting each other, and the suspension just behaves like water. As you increase the amount of starch the spacing between the grains decreases, until at some point the spacing between the grains becomes less than the size of a grain. At this point, when you try apply a large force to suspension the starch grains bump into each other and lock together to form a framework. The water in the suspension now has to flow through the small pores in the starch grain "framework" and this requires a lot of force. Hence you can stand on the suspension for a moment. If the apply a small force the water/starch grains move slowly and this gives time for the starch grains to slide around between each other so they will flow. This is why the chap in the white shirt could run on the oobleck, but when he stood still he gradually sank.

I can address one class of non-Newtonian fluids consisting of solid particles dispersed in a liquid medium, such as the cornstarch and water mixture commonly called "oobleck." In more scientific language, I am talking about concentrated colloidal suspensions of particles. Here is an image of oobleck, taken from Dounas-Frazer et al 2012.

These fluids tend to be shear-thickening, as you describe. Their viscosity increases substantially once the fluid velocity increases above a critical value. This tends to make them stiffen in response to impacts, which allows for fun activities such as running across the fluid surface (watch http://youtu.be/yHlAcASsf6U and related videos. The Faraday waves are also especially engrossing).

Here's how the shear-thickening comes about: Imagine a velocity gradient in the fluid. Then grains in one layer of the fluid will have to "roll over" particles in another layer of the fluid, colliding with each other as they do so. The steeper the velocity gradient, the more the fluid will tend to "dilate" in the direction normal to the gradient. But once the dilation effect gets sufficiently large, the water's surface tension provides a confining force that resists further dilation. This makes it much harder to maintain the velocity gradient and so the viscosity goes way up. (This explanation is based on the discussion in Fall et al 2007).