How to improve look and feel of JAVA swing GUI?

You can set the look and feel to reflect the platform:

try { 
    UIManager.setLookAndFeel(UIManager.getSystemLookAndFeelClassName());
} catch (Exception e) {
    e.printStackTrace();
}

If this is not nice enough for you, take a look at SWT for Eclipse.

if you're good with Photo shop you could declare the JFrame as undecorated and create your own image that will server as your custom GUI.

in this example refer to the JFrame as frame.

JFrame frame = new JFrame();
frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
frame.setSize(800, 600);
//If you want full screen enter frame.setExtendedState(JFrame.MAXIMIZE_BOTH);
frame.setUndecorated(true);

now that the JFrame has its own border and custom GUI, if you defined your own custom buttons for minimize, maximize, and exit you need to lay a JPanel over the buttons and declare what those buttons will do in a action listener function(lol sorry...I don't know if it's called function like c++ or class...) for example we will refer to the JPanel as panel and we will set up a exit button.

panel.addActionListener(new ActionListener(){
@Override
public void actionPerformed(ActionEvent e){
System.exit(JFrame.EXIT_ON_CLOSE);
});
}