How to know if a NSWindow is fullscreen in Mac OS X Lion?

You need to use an & bitwise operator to test that that option is being used. Not tested but probably something like this:

- (BOOL) inFullScreenMode {
    NSApplicationPresentationOptions opts = [[NSApplication sharedApplication ] presentationOptions];
    if ( opts & NSApplicationPresentationFullScreen) {
       return YES;
    }
    return NO;
}

To see if any of your windows are in full screen mode simply check the style mask of the window.

NSUInteger masks = [someNSWindow styleMask]
if ( masks & NSFullScreenWindowMask) {
 // Do something
}

The way I handled it in pre-10.7 (where neither NSApplicationPresentationFullScreen nor NSFullScreenWindowMask was available) was to check

if ([mainWindow frame].size.height == [[mainWindow screen] frame].size.height)
{
    // window is fullscreen
}

And this piece of really old code seem to still work not only on "Lion" but also on today's - at the time of writing 10.14.x - OS.

I was just looking for a solution myself and based on Matthieu's answer I created a category on NSWindow that works fine for me.

@interface NSWindow (FullScreen)

- (BOOL)mn_isFullScreen;

@end

@implementation NSWindow (FullScreen)

- (BOOL)mn_isFullScreen
{
    return (([self styleMask] & NSFullScreenWindowMask) == NSFullScreenWindowMask);
}

@end

For Swift 3.0

if let window = NSApp.mainWindow {
    let isWindowFullscreen = window.styleMask.contains(NSFullScreenWindowMask)
}

Obviously, for the original question, you'd replace NSApp.mainWindow with whichever document window you're wanting to check.