Why is the work done on a charge calculated from infinity?

Consider the form of the potential energy between two point charges in the case that I use a reference distance $r_0$ as the zero (written here in SI units). $$ U_{r_0} = \frac{q_1 \,q_2}{4 \pi \epsilon_0} \left( \frac{1}{r} - \frac{1}{r_0} \right) \;.$$ This is quite general, but it will get to be very messy to write down and manipulate very quickly indeed. It also means that the sign of the energy depends on the the relative sign of the charges and the relative size of $r$ and $r_0$

Now, the special case of taking $r_0$ as arbitrarily distant, gets us the familiar form \begin{align*} U_\infty &= \lim_\limits{r_0 \to \infty} U_{r_0}\\ &= \frac{q_1 \,q_2}{(4 \pi \epsilon_0) r} \;, \end{align*} which is algebraically simpler and the sign of which can be known at any distance just from the relative sign of the charges.

The conventional form is simply easier to use in the majority of cases.

But it gets better, because the same kind of consideration applies to Newtonian gravitation, and the convention of zero energy at infinite remove means that the total energy bound bodies is negative while that of free bodies is positive (with zero the parabolic edge case).

It really is a natural choice after you've looked at the ways you're going to be using the quantity.

See, infinity is the place that is considered to have no charges, and is at 0 potential all the time.

So, potential at a point in an electric field is defined as work done in bringing an unit positive charge from infinite distance to that point. Actually, we are measuring the potential difference between infinity and the required point. But we've named it potential of the point as it is with reference point infinity.

Basically, infinity is considered as a reference place that is fixed. If other points are considered, then one has to define the other point first and the potential at that point to find the potential at a new point.

N.B.: In practical application, we can measure potential difference and not strictly potential at a point.