Neutron discovery, why it couldn't be gamma ray? or other photon?

This is the illustration of the Chadwick experiment (from Wikipedia):

The key point is that the scattering is close to the forward direction. So if we draw the diagram for the Compton scattering as:

The angle $\theta$ is small. This diagram shows the hypothetical incoming gamma ray hitting a proton in the paraffin and scattering it. The equation for the change in wavelength of the gamma ray (due to energy transfer to the scattered proton) is given by the usual equation for Compton scattering:

$$ \lambda' - \lambda = \frac{h}{m_p\,c}\left(1 - \cos\theta\right) $$

Since $\theta$ is small $\cos\theta\approx 1$ and therefore the the $1-\cos\theta$ term is very small. This means that the change in wavelength of the gamma photon is small, and therefore that only a small fraction of the gamma photon energy can be transferred to the proton.

And this is the key point. Chadwick measured the energy of the emitted protons ($5.5$MeV) and their scattering angle, and used the equation above to calculate what energy the gamma ray had to be in order to transfer $5.5$MeV to the protons. The result was that that the initial gamma ray had to have an energy of at least 50MeV. But the initil alpha particles didn't have this much energy, so they couldn't possibly create a gamma ray with such a high energy, and that ruled out the presence of gamma rays.

In 1930 it was discovered that Beryllium, when bombarded by alpha particles, emitted a very energetic stream of radiation. This stream was originally thought to be gamma radiation. However, further investigations into the properties of the radiation revealed contradictory results. Like gamma rays, these rays were extremely penetrating and since they were not deflected upon passing through a magnetic field, neutral. However, unlike gamma rays, these rays did not discharge charged electroscopes (the photoelectric effect). Irene Curie and her husband discovered that when a beam of this radiation hit a substance rich in protons, for example paraffin, protons were knocked loose which could be easily detected by a Geiger counter.

In 1932, Chadwick proposed that this particle was Rutherford's neutron. In 1935, he was awarded the Nobel Prize for his discovery. Using kinematics, Chadwick was able to determine the velocity of the protons. Then through conservation of momentum techniques, he was able to determine that the mass of the neutral radiation was almost exactly the same as that of a proton. This is Chadwick's equation:

italics mine.

The answers to your questions:

paraffin slab is rich in protons so is there a need of high threshold energy?

The need for energy comes from the measured energy of the scattered by "X" particle

How they calculated the 55 Mev?

they measured the energy of the scattered proton.

Can't photo electric effect happen there so that only 5.7 Mev photon is needed?

The photoelectric effect is at the atom level, not the nucleus level and the energies are of order keV not MeV. In addition (italics in quote) the "X" did not interact electromagnetically.