What does one second after big bang mean?
We know that time passes differently for different observers, and the question is how can a time be given without telling which frame it is in. The answer is that there's a preferred reference frame in cosmology, the comoving frame, because of the fact that there's matter and radiation in it.
Intuitively, the special frame is the one that's "static" with respect to this matter and radiation content. More precisely, it is the one in which all observers that see an isotropic universe are static. Time measured in this system is called comoving time. The time from the beginning of the universe is usually given in this way, as a comoving time.
To get some intuition about the comoving frame one might consider the comoving observers, the ones that see isotropy and therefore have constant comoving coordinates. A comoving observer is such that when it looks around and adds the motion of the objects it sees zero net motion.
For example, we can look at the cosmic microwave background and detect some variation in the redshift depending on the direction. It's caused by Doppler effect and it means that we have some velocity relative to the comoving frame. On the other hand, a comoving observer sees the same redshift in any direction.
Another example: we can choose to measure the distances and velocities of galaxies. By Hubble's law, we expect the velocity to be proportional to the distance. If we find a deviation from this behavior, we know that the galaxy is moving with respect to the comoving frame, and thus has a peculiar velocity (we also have a peculiar velocity). If all galaxies had constant comoving coordinates, we would see perfect agreement with Hubble's law: the relative motions of galaxies would be due only to the expansion of the universe.
While not at all obvious, it turns out that our best models of cosmology suggest that there exists a special frame of reference in which the distribution of the entire universe's matter and energy appears extremely uniform on very large (i.e. cosmological) scales. When we talk about the age of the universe, we always mean the age as viewed in this special frame. You are completely correct that particles that are moving very quickly with respect to this special frame will measure a very different age of the universe.
(Note that the existence of this special frame is completely compatible with special and general relativity, which say that the laws of physics themselves look the same in any inertial reference frame, not that the distribution of matter does.)