How to fill empty space when component is not visible or hidden?

You need to call setManaged(false) as well.

You can take advantage of bindings to set managed property automatically based on visibility. E.g. component.managedProperty().bind(component.visibleProperty());