How are we supposed to use debug logs for a specific Apex class only

Realistically, all you can do is set the user trace flags to a debug level with all levels set to "none", then set the class's log to a debug level that you want. Yes, you'll still be generating a ton of logs, but the irrelevant logs will all show up as 1Kb, while the ones you want will be significantly larger. If you're using the Developer Console, you should even be able to click on the Size column header to sort by size; all of the ones you care about will be near the top/bottom of the list.

The short answer is, you can't. The class specific debug levels only control how verbose the logging for those classes are.

When the user executes an action, all the classes within the same context are logged. Theoretically, you can make this easier on yourself by lowering the debug level of all of the classes you don't care about that you expect to show up within the same context but that's still not even close to what you want.

You can try to implement a custom logging tool but even then, you're not going to get true logging. You'll only get the information you tell it to give you (which may work for your needs).

What I typically do is talk to the user I'm trying to debug. Tell them to stop clicking on things, delete all the debug logs, then tell them to click the one button that executes the action you want logged.