JavaFX Minimizing Undecorated Stage

You can just set the appropriate window style. It works in XP but should be ok in windows 7 32 bit. I think (but can't test) if you use 64 bit then change to the Ptr windows functions, ie. GetWindowLongPtr.

import com.sun.jna.Native;
import com.sun.jna.Pointer;
import com.sun.jna.platform.win32.User32;
import com.sun.jna.platform.win32.WinDef.HWND;
import com.sun.jna.platform.win32.WinUser;
import static com.sun.jna.platform.win32.WinUser.GWL_STYLE;
import javafx.application.Application;
import javafx.scene.Scene;
import javafx.scene.control.TextArea;
import javafx.scene.layout.VBox;
import javafx.stage.Stage;

public class JNATest extends Application {
    public static void main(String[] args) { launch(args); }

    @Override
    public void start(Stage stage) {
        TextArea ta = new TextArea("output\n");
        VBox root = new VBox(5,ta);
        Scene scene = new Scene(root,800,200);
        stage.setTitle("Find this window");
        stage.setScene(scene);
        stage.show();
        //gets this window (stage)
        long lhwnd = com.sun.glass.ui.Window.getWindows().get(0).getNativeWindow();
        Pointer lpVoid = new Pointer(lhwnd);
        //gets the foreground (focused) window
        final User32 user32 = User32.INSTANCE;
        char[] windowText = new char[512];
        HWND hwnd = user32.GetForegroundWindow();
        //see what the title is
        user32.GetWindowText(hwnd, windowText, 512);
        //user32.GetWindowText(new HWND(lpVoid), windowText, 512);//to use the hwnd from stage
        String text=(Native.toString(windowText));
        //see if it's the same pointer
        ta.appendText("HWND java:" + lpVoid + " HWND user32:"+hwnd+" text:"+text+"\n");
        //change the window style if it's the right title
        if (text.equals(stage.getTitle())){
            //the style to change 
            int WS_DLGFRAME = 0x00400000;//s/b long I think
            //not the same constant here??
            ta.appendText("windows api:"+WS_DLGFRAME+" JNA: "+WinUser.SM_CXDLGFRAME);
            int oldStyle = user32.GetWindowLong(hwnd, GWL_STYLE);
            int newStyle = oldStyle & ~0x00400000; //bitwise not WS_DLGFRAME means remove the style
            newStyle = newStyle & ~0x00040000;//WS_THICKFRAME   
            user32.SetWindowLong(hwnd, GWL_STYLE, newStyle);
        }
    }

}

My guess is you replace the last 3 lines with

            long oldStyleLong = user32.GetWindowLongPtr(hwnd, GWL_STYLE).longValue();
            long newStyleLong = oldStyleLong & ~ 0x00400000l;
            user32.SetWindowLongPtr(hwnd, GWL_STYLE, new BaseTSD.LONG_PTR(newStyleLong));

for 64 bit. I think I don't have those functions in my User32.dll, so I can't test it. There's lots of extraneous code in there, mainly for testing or teaching. Remove the unused lines once you figure out what you want to do.

ps. Don't add newStyle = newStyle & ~0x00020000;//WS_MINIMIZEBOX. That's one of the style flags JavaFX doesn't use for undecorated. That's why the minimize isn't available. Maybe if you try setting stage undecorated and adding (using |, not &~) the minimize box flag, you'll get the same result. There are tools to look up all the style flags from any window.

Here's the simplest amount of code that just changes an undecorated stage using the stage's HWND.

    public void start(Stage stage) {
        Scene scene = new Scene(new Pane(new Label("Hello World")));
        stage.initStyle(StageStyle.UNDECORATED);
        stage.setTitle("Find this window");
        stage.setScene(scene);
        stage.show();
        long lhwnd = com.sun.glass.ui.Window.getWindows().get(0).getNativeWindow();
        Pointer lpVoid = new Pointer(lhwnd);
        HWND hwnd = new HWND(lpVoid);
        final User32 user32 = User32.INSTANCE;
        int oldStyle = user32.GetWindowLong(hwnd, GWL_STYLE);
        System.out.println(Integer.toBinaryString(oldStyle));
        int newStyle = oldStyle | 0x00020000;//WS_MINIMIZEBOX
        System.out.println(Integer.toBinaryString(newStyle));
        user32.SetWindowLong(hwnd, GWL_STYLE, newStyle);
    }

It prints out the style flags before and after so you can look up what styles are set.

Two things to note here.

First, it doesn't look like these libraries are in the latest version of JNA, 5.50 as of now, adding from the Maven repository. I had to add the 4.2.1 library instead.

Second, you may encounter this exception, like I did on Windows 10 and Java 11: Error with package com.sun.glass.ui while learning Java Native Access

The solution is go to your VM options in your IDE (Run -> Edit Configurations..., in IntelliJ) and add this:

--add-exports
javafx.graphics/com.sun.glass.ui=ALL-UNNAMED

It should work after that.

I would like to see someone implement the native Windows animations for minimizing and un-minimizing an undecorated window, but I haven't searched too thoroughly yet to see if this has already been discussed. I'll update this if I come across a solution.

Edit:

Upon further research on the Windows animations, it looks like a solution could be hacked together, but I gave up at trying to implement this C# hack below. It seems to be more of an OS issue and not just JavaFX.

I was able to get the initial window to stay undecorated while minimizing and with the animation by modifying this in start():

int newStyle = oldStyle | 0x00020000 | 0x00C00000;

But, after minimizing and reopening, the Windows border appears oddly enough.

Then, I tried to use a ChangeListener to swap Windows styles when iconifying.

stage.iconifiedProperty().addListener(new ChangeListener<Boolean>() {

        @Override
        public void changed(ObservableValue<? extends Boolean> ov, Boolean t, Boolean t1) {
            if (t1.booleanValue() == true) {
                int newStyle = oldStyle | 0x00020000 | 0x00C00000;
                user32.SetWindowLong(hwnd, GWL_STYLE, newStyle);
            } else if (t1.booleanValue() == false) {
                int newStyle = oldStyle | 0x00020000;
                user32.SetWindowLong(hwnd, GWL_STYLE, newStyle);
            }
        }
    });

This successfully gets the windows un-minimize animation to work fine consistently, while leaving the (visible) stage borderless.

It looks like I can get minimization animations working once I find out the best way to re-apply:

int newStyle = oldStyle | 0x00020000 | 0x00C00000;
user32.SetWindowLong(hwnd, GWL_STYLE, newStyle);

just before the stage is iconified, and the border isn't visible to the user. Once implemented, this might work similarly to the C# solution in the first link below. Basically, what the above ChangeListener does in reverse.

Links to do with solving borderless/undecorated animations:

Use windows animations on borderless form

https://exceptionshub.com/borderless-window-using-areo-snap-shadow-minimize-animation-and-shake.html

http://pinvoke.net/default.aspx/Constants/Window%20styles.html

JavaFX Minimizing & Maximing undecorated stage with animations