Hiding a C++ class in a header without using the unnamed namespace

The right way to go about it in C++ is PIMPL idiom. Alternative solution is to put the class you want to hide into a nested namespace, which is usually called detail. But that will not make it totally private as users will still be exposed to its dependencies, and will be able to use it directly.

You could do an inner class:

class B
{
  class A { /* ... */ };
  A a_;
}

Document that this class is not part of the public API and should not be used.

In C++ you have to trusted programs that link with your library code because you have little other choice. C++ has limited "access control" features many of which can be bypassed or abused so you're better of treating your API clients with respect and building trust.

If you design your API to be easy to use correctly and hard to use unintentionally incorrectly then you will be helping your clients and it is hardly your fault if your clients abuse your interface.