Is it possible to disable a single default feature in Cargo?

This is not currently possible, but your motivation sounds reasonable. Ideas for improvements to Cargo can be submitted as GitHub issues or (for more major things than this) RFCs.

Currently, No. You have to disable all "default" features, then opt back in for the ones you want. So you have to know what the "default" list is, and opt-back-in for all but the one you want to omit.

This GH issue describes the status and discusses possibilities, but remains open (3y later).