Fuse 125V/4A can be replaced with 250V/4A?
Fuses are used for over-current protection, first please diagnose the mistake what causing the fuse to blow (like any shortages, overloading or some faults) and solve that before you replace new fuse.
As long the voltage of fuse is more than the system voltage. you can use it.
Since, what you are saying is right.. fuses protects the system/equipment from over-current not over-voltage.
Yes.
The voltage rating on a fuse tells you how high a voltage it can interrupt without arcing or otherwise allowing current to continue to flow after the fuse blows. A fuse of equal or higher voltage rating is fine as a replacement (after fixing whatever caused the old fuse to blow).
The conditions that will cause a fuse to blow are solely related to current (and time, for a slow-blow fuse). A replacement fuse needs to match current rating and slow-blow or not; no more and no less, so it doesn't blow spuriously and doesn't allow more current than desired without blowing.
Fuses are similar to a switch or relay that has to be able hold off a high voltage across itself. Interrupting a high current with an inductive load can create transient high voltages, which could create an arc. When a fuse first melts apart, you presumably do get some arcing over the tiny initial gap, but it presumably extinguishes quickly as more metal melts and the gap gets too long.