Change background color of NSButton
Here is another possible way to do this:
let button = NSButton()
button.title = ""
button.bezelStyle = .texturedSquare
button.isBordered = false //Important
button.wantsLayer = true
button.layer?.backgroundColor = NSColor.red.cgColor
The standard Aqua (pill-shaped) buttons are drawn by the system. They don't have a background color as such. In fact, they are composed of images that Cocoa stitches together to make a coherent button image. So Cocoa can't recolor the images to your liking. Only the default button (the one with Return set as its key equivalent) will have a blue pulsing background.
What it sounds like you want to do will involve subclassing NSButtonCell
and doing your own drawing. If you wanted to recolor the button images to get the effect you want, you can use the excellent Theme Park utility to extract copies of Apple's button images and use those. I'll admit to having done this myself in order to "steal" certain interface elements that aren't otherwise accessible, such as the scrollers from iTunes.
Assuming everything is hooked up in IB for your borderless button.
// *.h file
IBOutlet NSButton* myButton;
// *.m file
[[myButton cell] setBackgroundColor:[NSColor redColor]];
Note from the setBackgroundColor documentation:
"The background color is used only when drawing borderless buttons."
If this won't do it for you then you'll need to override NSButton and implement the drawing yourself.
Good Luck.