Why exactly do atomic bombs explode?

I don't understand why exactly it leads to a powerful explosion instead of just a burst of ionising radiation.

This radiation, representing most of the initial energy output by a nuclear weapon, is swiftly absorbed by the surrounding matter. The latter in turn heats almost instantly to extremely high temperature, so you have the almost instantaneous creation of a ball of extremely high kinetic energy plasma. This in turn means a prodigious rise in pressure, and it is this pressure that gives rise the blast wave.

The same argument applies to the neutrons and other fission fragments / fusion products immediately produced by the reaction. But it is the initial burst of radiation that overwhelmingly creates the fireball in an atmospheric detonation, and the fireball that expands to produce most of the blast wave.

In an implosion fission bomb, the bulk of the nuclear reactions (100 GJ) occur in the final microsecond--it is a tremendous amount of power--$10^{17}$ Watts. This power is dumped into the compressed Pu pit--which is a few kilogram with a density of 100 g/cm^3-it is small.

The nuclear material is heated to an extremely high temperature, tens of millions of K--it black body radiates (X-rays) to the nearest material: the "cold" bomb components (air quotes because they have just been involved in a conventional explosion). They are heated to a lesser temperature, which blackbody radiates isotropically, heating material further away from the bomb. Note that the atmosphere is opaque to these X-rays, so the process of radiative diffusion continues for several hundred of meters. This is called the fireball--it's all about photons and radiative diffusion. As it diffuses outward, at some point it becomes slower than a shockwave, so that a shockwave separates from the fireball, which is now on the order of a few hundred thousand K. (In photographs of the fireball in the 1st few milliseconds, when it's 100's of meters, the 'shadows' of the bomb components can still be seen).

The shockwave heats air as it passes, but it is not as hot as the fireball. It is at least 50,000K, maybe 100,000K--so it's at least as radiant as a lightning bolt--but it is not transient, and it could subtend a much larger solid angle--hence the phenomenal thermal damage. Nevertheless, it is significantly colder than the material behind it.

As the shockwave propagates, it weakens an eventually no longer heats air to luminance--at this point the traditional fireball has reached it's maximum size. (I say traditional, because its what we see in test footage, but it should be distinguished from the initial fireball which is radiatively diffusing photons with some plasma thrown in--and radiation is the dominant mode of energy transfer, even though the bomb casing can have hypersonic velocities.)

Now for an airburst the shockwave that is reflected from the ground move through heated air, and is thus faster than the direct shockwave: it catches up and the two combine to for a more powerful shock, called the Mach stem. This goes on to produce blast damage, as buoyant forces lift the fireball, producing the infamous mushroom cloud.

For tests like Starfish Prime, which occurred in space--the initial X-rays (very hard X-rays) from the bomb components aren't absorbed by air--they continue to the upper atmosphere where wide scale Compton scattering produces a huge and sudden current, leading to continental scale EMP.

A point of clarification: since the OP asked about shockwave formation--as others pointed out, the temperature rise leads to huge pressure, which leads to a shockwave, but it does not form in the fireball--the radiative diffusion is at first much faster--and it's only as the fireball petters out that the shockwave separates from it.

The easiest way to answer this is to direct you to an explanation of the products of nuclear fission and their energy components. You are correct in thinking that ionizing radiation is released. However, it is only a small part of the energy release. For U235, the energy released in a single fission is about 195 MeV. Of that, 170 MeV is in the kinetic energy of the two fission fragments (physics is nice in limiting things to two fragments most of the time). Another 12 MeV is in the kinetic energy of the released neutrons which can cause the self-perpetuating chain reaction. Only about 8 MeV is found in the released gamma radiation. The heat generated is then due to the transfer of kinetic energy in collisions with other matter. Since most of the initial energy release occurs within 10E-12 seconds of the absorption of the initial neutron, and the collisions begin shortly after that, the heat generation begins within fractions of a second. The plasma expansion, pressure rise, and blast wave follow.