Does the number of connected devices—regardless of activity—kill Wi-Fi speed?

I spoke with a person who actually designs Wi-Fi hardware on low level signal processing et al.

The simple answer is: "No, the number of devices connected to an AP have little to no effect if they're not doing anything".

Longer answer gets a bit involved but basically Wi-Fi physical layer is entirely asynchronous technology and the clients do not "say" anything much "often" unless they want something. Obviously "often" is in machine terms, to put things into perspective the client Wi-Fi chip has only about 76us to react during a transaction so almost all of the low level functionality is handled in dedicated hardware even for host-based solutions. But very little real overhead is created by idle clients.

Wrt several devices taxing the network, it is more pronounced in a near-far case where the clients cannot "hear" each other. In this case they can both (or several of them) access the AP at the same time causing stalls and resynchronization, not very different from old-school coaxial ethernet collision issues. But if you have a room full of guys with their Android phones connected to the same AP, they can "hear" the band is in use and wait for their turn.