Is it possible to build custom GUI like this in Java?

Is not only possible but quite easy, you don't have to go all the way to create a custom Look and Feel.

This is what I've made in 20 minutes:

There is a lot of information on the web on how to customize the components without having to create a whole new L&F.

Understanding the Swing Architecture helps a lot.

Just in case you haven't read it the Swing Tutorial is here.

Finally you'll need the doc: Java doc

Oh dear, no-no-no! If you want your users' eyes to bleed, then go for it. Otherwise, follow whatever UI guidelines are appropriate for your platform.

To answer your question: this is certainly doable in any modern windowing system.

Here's what generally happens when programmers design UIs:

_{(source: jensroesner.de)}

_{(source: bulkrenameutility.co.uk)}

You can and even you can change it dynamically - see Look-n-Feel feature of swing