Showing a modal NSWindow, without activating the other application windows

Instead of creating an NSWindow, create an NSPanel with the style NSNonactivatingPanelMask. You can then do the usual makeKeyAndOrderFront: and orderOut: to show/hide panel as needed.

NSApp's beginModalSessionForWindow, runModalSession, endModalSession are methods you need.

Have a look here for example how to use it: Creating a fully customized NSAlert