How to create a transparent window with non-rectangular buttons?

What about overriding the NSResponder event methods (mouseUp:, mouseDown:, etc) in your custom buttons (presumably you'd be subclassing NSControl or NSButton to make these)? In those methods you could properly calculate your bounding rectangle (or circle, in this case) and do a simple hit test with the coordinates of the click to see if the click should be handled.

You might also be able to find some help in Apple's event handling docs, specifically about mouse events and custom views.

What about making it all a single control? It might make drawing a little bit complex, but it looks like you're doing custom drawing anyway.

If you did it as a single control, you could first check and see if the click point is within the center circle. If it's not, then all you have to do is identify which quadrant it's in to know which "button" it belongs to (while also verifying that the click fits within the outer circle).

Alternatively, you could create a transparent view over your buttons that captures the click events and forwards them on to the appropriate control based on the logic I just specified.

Similar to Marc W's response, in your own event method you can check the alpha value of the clicked on the bitmap and ignore if the alpha is lower than a certain value.

To get the bitmap, read this.

Now you should have a bitmap pointer like this (pseudocode - you'll have to fill in the pieces):

pixels = CGBitmapContextGetData( ctx ); // there's actually two ways to get pixels in the above Apple tech note

Then, you can do this to get the pixel you are interested in, and test it:

// I'm assuming each pixel is 24 bits of color and one byte of alpha
#define getRed(p) ((p) & 0x000000FF)
#define getGreen(p) ((p) & 0x0000FF00) >> 8
#define getBlue(p) ((p) & 0x00FF0000) >> 16
#define getAlpha(p) ((p) & 0xFF000000) >> 24

CGPoint pixPt;
long pixel;
pixPt = getMouseClick();

pixel = pixels[pixPt.x + pixPt.y * bytesPerRow];

if (getAlpha(pixel) < 25)
  // you've clicked on transparent pixels, alpha of 10% or less (alpha ranges from 0-255)

This opens up some possibilities for you, such as having a ring around your inside circle that is inert and won't belong to any of the quadrants or the middle circle. Of course, there's always other ways to do these things without resorting to pixel-wrangling :-)