How can I find out which wires are the primary wires on this transformer?
Since transformers by their nature are bi-directional, the selection of the primary side totally depends on your input voltage and desired output voltage.
The transformer you describe likely has multiple taps on the "primary" side, may have multiple windings on the "primary" side and likely has multiple windings on the secondary side. Start with a low range DMM, and check for continuity between different leads on each side of the transformer. Once you have mapped continuity, check resistance between the same leads. You should be prepared for the transformer to be as complex as this:
The "secondary" side may be a single coil with multiple taps, or it may have multiple outputs more like the above example.
Once you've reverse-engineered the coil arrangement, you'll need to determine the turns ratio between each set of coils. I would NOT recommend your 120VAC test for this. Start with a much lower (and safer) voltage. Find a small "wall-wart" type power supply that you can sacrifice. The lower the output voltage the better. You want it just for its transformer, not the rectification and regulation components, so if you can find an AC-output wall-wart, you can use it's output as-is. What you want is a low voltage AC source that you can use to test individual windings. Note that applying a low voltage AC source to the "secondary" may result in lethal voltages on the "primary", so be careful!
Find one set of windings to apply your AC input to, and measure the resulting output on each set of coils and on each tap. Transformers are ratiometric, so the relative voltages will be the same using your low voltage AC test vs. when you identify the intended primary winding and apply 115VAC to it.
Doing this, you should have a good sense as to what windings are present that the relative turns ratio between each. Good luck!
Electrician, butting in.
In the field, we frequently test transformer ratios as a service to industrial customers. Normally you'd have an automatic or semi-automatic test set that would give you the complete characteristic of, say, a current transformer. But since you don't have that, you have to get creative.
Using a voltage source can be very unpredictable and quite dangerous, as HikeOnPast and Martin James pointed out. A quick multimeter reading from terminal to terminal on each coil will tell you which one is longer - that is, the high-voltage side. Short it from end to end using an ammeter. Then pass a controlled current source through the low-voltage side. This can actually give you incredibly accurate results. If you find that you have a multiple winding transformer, all of them have to be shorted out.
This obviously has the disadvantage that now, you need a source of ac that produce a controlled amount of current. These aren't exactly just laying around, but they could be built fairly simply. So, yeah, what's the point?
No more dangerous voltages, and no need for high voltage probes. The relatively low-impedance coils will not generate any significant amount of voltage in this arrangement - as long as you short all of the other coils out. Do NOT forget that part, or you will be very sad.
If you do this sort of thing a lot, I strongly recommend keeping around an AC current source anyway. There is no better way of getting highly accurate resistance readings on low-resistance stuff, and no safer way of testing unknown transformers and coils.
'To test I (very quickly) hooked up 120V to the pairs and measured the other wires with my scope.'
Don't do this, even for a second. Apart from the danger of death by stepped-up voltages, an inadvertent gross step-up resulting from such experimentation is quite likely to result in internal arc-over, straining the insulation and resulting in either immediate, or subsequent, failure of windings.
Find a pair and shove on a much. much lower voltage. A variac would be a good start - notch up carefully and see what is happening on all the other windings.