What determines the speed of a brushless DC motor

First let's consider just a ordinary brushed DC motor. The hardware mechanically ensures that the windings are switched (commutated) such that the magnetic field is always trying to pull the motor along. The magnetic field strength is directly proportional to current, so the torque is proportional to current. So at a very basic level, the speed is whatever results in enough mechanical resistance to ballance the torque. However, that is not useful in most cases since it's not obvious what the current is.

For a stalled motor, the current is the applied voltage divided by the resistance of whatever windings are switched in. However, as the motor spins it also acts like a generator. The voltage the generator produces is proportional to speed, and apposes the external applied voltage. At some speed this equals the external voltage, in which case the effective voltage driving the motor is zero and the motor current is zero. That also means the torque is zero, so a unloaded motor can't spin that fast since there is always some friction. What happens is that the motor spins at a little lower speed. The amount it spins slower is just enough to leave a little effective voltage on the motor, which is the amount to create just enough current to create the torque to ballance the small friction in the system.

This is why the speed of a unloaded motor doesn't just increase until it flies apart. The unloaded speed is pretty much proportional to the external voltage, and is just below the speed at which the motor internally generates that voltage. This also explains why a fast spinning motor draws less current than a stalled motor at the same external voltage. For the stalled motor, current is applied voltage divided by resistance. For the spinning motor, current is applied voltage minus the generator voltage divided by the resistance.

Now to your question about a brushless DC motor. The only difference is that the windings are not automatically switched in and out according to the rotation angle of the motor. If you switch them optimally as the brush system in a brushed DC motor is intended to do, then you get the same thing. In that case the unloaded current will be even lower since there is no friction from the brushes to overcome. That allows less current to drive the motor at a particular speed, which will be closer to where the generator voltage matches the external applied voltage.

With a brushless motor you have other options. I recently did a project where the customer needed very accurate motor speed. In that case I communtated the windings at precisely the desired speed derived from a crystal oscillator. I used the Hall effect position feedback signals only to clip the applied magnetic field to within ±90° of the position. This works fine as long as the load on the shaft is less than the torque applied when the magnetic field is at 90°.

Usually, however, you commutate a brushless DC motor optimally, just like the mechanical brushes would try to do. This means keeping the magnetic field at 90° from the current position in the direction of desired rotation. The overall applied voltage is then adjusted to modulate speed. This is efficient since only the minimum voltage is used to make the motor spin the desired speed.

Yes, PWM works fine for driving the coils. After a few 100 Hz or so for most motors, the windings only "see" the average applied voltage, not the individual pulses. The mechanical system can't respond anywhere near that fast. However, these windings make magnetic fields which apply force. There is a little bit of force on every turn of wire. While the motor may operate fine at a few 100 Hz PWM, individual turns of the winding can be a little loose and vibrate at that frequency. This is not good for two reasons. First, the mechanical motion of the wires can eventually cause insulation to rub off, although that's rather a long shot. Second, and this is quite real, the small mechanical vibrations become sound that can be rather annoying. Motor windings are therefore commonly driven with PWM just above the audible range, like 25-30 kHz.

Speed of the brushless DC motor depends on the same parameters as in a brushed DC motor. The speed is directly proportional to the voltage that is applied to the phases (e.g A,B,C incase of a 3 phase motor). The speed of the bldc motor is inversely proportional to the torque on the rotor shaft when it is set up for constant power. . The current flowing through the windings is directly proportional to the torque. Hence in a simple way, the speed of the brushless motor increases in with increase in voltage OR decrease in the winding current (assuming one of these parameters as a constant).

The applied voltage here refers to the "average" voltage of the phases. This in turn is dictated by the width of the PWM pulses applied to the FETs (in case of a bridge driver) that drive the phases.