Java: Stack with weak references

Yes. What you're describing is a property of weak references in general, not WeakHashMap specifically.

From the API:

Suppose that the garbage collector determines at a certain point in time that an object is weakly reachable. At that time it will atomically clear all weak references to that object...