What makes a pull-up/down resistor strong or weak?

Strong means low resistance. Weak means high resistance. Of course low and high are relative terms, and so are strong and weak. The reference for this relationship must be inferred from context.

A strong or low resistance pull-up/down resistor is good because the time constant formed the load capacitance (often, the input gate capacitance, and the PCB trace capacitance) is small, so rise/fall times will be short.

A strong pull-up/down resistor is good because noise currents from unintended coupling and EMI will result in smaller noise voltages. (Think about Ohm's law)

A weak or high resistance pull-up/down resistor is good because it will not require much current from the driving circuitry to work against the resistor. Batteries will thus last longer, parts can be smaller and don't get as hot.

Of course, you usually want all of these things, but a resistor can't be both. A discussion about strong vs. weak is usually clarifying which of these concerns (or perhaps others) are more important for a particular application.

A "weak" pull resistor is usually a high value resistor that only allows a small amount of current through, and can quickly be overwritten, but takes longer to reassert itself.

A "strong" pull resistor is usually a low value resistor, allows more current through, takes longer to be overwritten, but can quickly reassert a line.

They are completely relative to your needs, not just other pull resistors like internal ones.

In your button scenario, the time it takes to switch from one state to the other isn't important, so weak vs strong doesn't apply there. But weak vs strong does apply in the practical matter of Current Consumption. A strong pull resistor would, when the button is pressed, cause a large drain of current from vcc through the resistor to ground. A weak pull resistor would cause a small drain of current. Theoretically any resistor would work, but for practical purposes, a weak resistor is used because unnecessary high current drains can cause issues and can easily be avoided by sizing the resistor correctly.

Does "strong" vs "weak" only apply when one such resistor is being compared to other resistances in the circuit, such as an internal pull-down resistor?

Yes, this is exactly it. Strong and weak simply refer to the relative drive strength of the component. A pull up/down resistor's value has no association to whether it is strong or weak. Only in knowing the context of the other connections to the net can you determine if a pull-up is strong or weak.