Create one object instance per class in hierarchy

But the next question was how would I enforce this approach on each class? As per my approach each constructor would have to carefully populate the map otherwise it will break the constraint.

Simply by putting that map, and the code around it into some distinct Utility class. So that each and of your classes could do something like:

public WhateverClassConstructor() {
  synchroized(someLock) {
     SingeltonChecker.ensureUnique(this);

with

 public static void ensureUnique(Object objectToAdd) {
   Class<?> classToAdd = o.getClass();
   ...

And given the fact that C extends B extends A, you probably only need that call in the constructor of class A. On the other hand, imagine that your first call to new C() causes an exception with the C-constructor, but a second call would not. Sure, that is a problem in itself, but still, the question remains: how do you ensure an object was fully and correctly initialized before adding it to such a map?!

Thus: there is a ton of things to consider when writing such utility code. Thus my answer would focus on the impractically, almost stupid-ness of the given design point, and suggest to throw it away and go for other solutions, for example something as simple as:

public enum SingletonObjectHolder {
  private final Object hold;
  A_INSTANCE(new A()), B_INSTANCE(new B())...

  public Object getSingleton() { return hold; }
  private SingletonObjectHolder(Object o) { hold = o };

Don't waste too much time trying to give people a technical answer, when the real point is to not shoot yourself into the foot. And make no mistake: getting that map-based "singleton" approach to work robust and correct, for all kinds of contexts, consider that really hard.

In other words: if I would ask you this question in an interview, I would want to hear an answer that challenges that horrible design point. Sure, spent 10% of your time outlining a solution for the given scenario. But spend 90% of the time explaining why it is so bad, and why other solutions would be much better.

This pattern doesn't make much sense, but you can use the implicit super() invocation in default constructors to your advantage here.

Given:

class A{
        static final Set<Class<?>> classes = new HashSet<>();
        public A() {
            if (!classes.add(getClass())) 
               throw new IllegalStateException(
                  "One instance of " + getClass() + " already created."
               );
        }
    }
class B extends A{}
class C extends B{}

And somewhere...

new A();
new B();
new C();
new C();

The last invocation will throw the exception because the implicit C constructor invokes super().

Making this thread-safe would be the next step, which you can do in a number of ways.