Isn't my ground different from the factory's ground?

Components are damaged by two or more of their pins being at a large enough potential difference. If the component has a conductive case, or pad, then that counts as a 'pin' too.

It's possible to break them by trying to charge them up to a new potential through one sensitive pin, while the voltage of the other pins is held more or less constant through capacitance to ground. That can be the situation where you, perhaps charged to 15kV with respect to ground, pick up a component that's at ground potential by (say) the gate lead.

Conductive packaging shorts all the pins together. What you do is to bring the conductive bag to your potential first. Any charging current that has to flow into the component does so through all pins, so does not damage the component.

Let's say an insulated carton of components in conductive bags charged to 100kV arrives at your workstation. You and the workstation are grounded. You open the carton, and as soon as you touch a component bag, a current flows between you and the bag to discharge it down to ground potential. Meanwhile, the bag has maintained all the component pins at the same potential, so no damaging voltage is applied across the component. Now you and component are at the same potential, you can open and touch.

Why did the component arrive at 100kV? Surely the other factory ground is not that different to yours? No, but the last bit of the trip might have been carried by a guy with nylon shoes. When stuff is properly packed, it doesn't matter if intermediate stages of the journey take it to potential way different from ground.

Hopefully your parts are packaged in an ESD-dissipative tray or bag. Then when you set them down on your ESD mat in your lab, any charge that's built up on them can drain away through the packaging and the mat. They won't discharge quickly enough to damage the components because both the bag and the mat have substantial resistance (1 megohm to ground is common for ESD mats and wrist straps).

My understanding is that ESD safety things are designed to bring everything that can touch a component to the same electrical potential energy

That's when you're right.

ground.

And here's when you're wrong. There is no such thing as "universal ground". Not even Earth is. You just pick a point of a circuit and say "Hereby, by the power vested in me by Science of Electrical Engineering, I declare you as The Ground and all other grounds as null and void." and instead of a sword, you touch it with an ESD safety thing. That's it.

It doesn't matter if your ground is same or 1000V different than ground they've used at the factory. As you've said, the ESD safe equipment is designed to bring the part safely (read: slowly) to your ground.