Customize right click highlight on view-based NSTableView

Stop Default Drawing

Several answers describe how to draw a custom contextual-click highlight. However, AppKit will continue to draw the default one. There is an easy trick to stop that and I didn't want it to get lost in a comment: subclass NSTableView and override -menuForEvent:

// NSTableView subclass
override func menu(for event: NSEvent) -> NSMenu?
{
    // DO NOT call super's implementation.
    return self.menu
}

Here, I assume that you've assigned a menu to the tableView in IB or have set the tableView's menu property programatically. NSTableView's default implementation of -menuForEvent: is what draws the contextual menu highlight.

---

Solve Bad Apple Engineering

Now that we're not calling super's implementation of menuForEvent:, the clickedRow property of our tableView will always be -1 when we right-click, which means our menuItems won't target the correct row of our tableView.

But fear not, we can do Apple Engineering's job for them. On our custom NSTableView subclass, we override the clickedRow property:

class MyTableView: NSTableView
{
    private var _clickedRow: Int = -1
    override var clickedRow: Int {
        get { return _clickedRow }
        set { _clickedRow = newValue }
    }
}

Now we update the -menuForEvent: method:

// NSTableView subclass
override func menu(for event: NSEvent) -> NSMenu?
{
    let location: CGPoint = convert(event.locationInWindow, from: nil)
    clickedRow = row(at: location)

    return self.menu
}

Great. We solved that problem. Onwards to the next thing:

---

Tell Your RowView To Do Custom Drawing

As others have suggested, add a custom Bool property to your NSTableRowView subclass. Then, in your drawing code, inspect that value to decide whether to draw your custom contextual highlight. However, the correct place to set that value is in the same NSTableView method:

// NSTableView subclass
override func menu(for event: NSEvent) -> NSMenu?
    {
        let location: CGPoint = convert(event.locationInWindow, from: nil)
        clickedRow = row(at: location)
        
        if clickedRow > 0,
           let rowView: MyCustomRowView = rowView(atRow: tableRow, makeIfNecessary: false) as? MyCustomRowView
        {
            rowView.isContextualMenuTarget = true
        }
        
        return self.menu
    }

Above, I've created MyCustomRowView (a subclass of NSTableRowView) and have added a custom property: isContextualMenuTarget. That custom property looks like this:

// NSTableRowView subclass
var isContextualMenuTarget: Bool = false {
    didSet {
        needsDisplay = true
    }
}

In my drawing method, I inspect the value of that property and, if it's true, draw my custom highlight.

---

Clean Up When The Menu Closes

You have a controller that implements the datasource and delegate methods for your tableView. That controller is also likely the delegate for the tableView's menu. (You can assign that in IB or programatically.)

Whatever object is your menu's delegate, implement the menuDidClose: method. Here, I'm working in Objective-C because my controller is still ObjC:

// NSMenuDelegate object
- (void) menuDidClose:(NSMenu *)menu
{
    // We use a custom flag on our rowViews to draw our own contextual menu highlight, so we need to reset that.
    [_outlineView enumerateAvailableRowViewsUsingBlock:^(__kindof MyCustomRowView * _Nonnull rowView, NSInteger row) {
        
        rowView.isContextualMenuTarget = NO;
            
    }];
}

Performance Note: My tableView will never have more than about 50 entries. If you have a table with THOUSANDS of visible rows, you would be better served to save the rowView that you set isContextualMenuTarget=true on, then access that rowView directly in -menuDidClose: so you don't have to enumerate all rowViews.

Single-Column: This example assumes a single column tableView that has the same NSMenu for each row. You could adapt the same technique for multi-column and/or varying NSMenus per row.

And that's how you beat AppKit in the face until it does what you want.

I know I'm a bit late to offer any help to the OP, but hopefully this can spare some other folks a little bit of time. I subclassed NSTableRowView to achieve the right-click contextual menu highlight (why Apple doesn't have a public drawing method to override this is beyond me). Here it is in all its glory:

BSDSourceListRowView.h

#import <Cocoa/Cocoa.h>

@interface BSDSourceListRowView : NSTableRowView

// This needs to be set when a context menu is shown.
@property (nonatomic, assign, getter = isShowingMenu) BOOL showingMenu;

@end

BSDSourceListRowView.m

#import "BSDSourceListRowView.h"

@implementation BSDSourceListRowView

- (void)drawBackgroundInRect:(NSRect)dirtyRect
{
    [super drawBackgroundInRect:dirtyRect];

    // Context menu highlight:
    if ( self.isShowingMenu ) {
        [self drawContextMenuHighlight];
    }
}

- (void)drawContextMenuHighlight
{
    BOOL selected = self.isSelected;
    CGFloat insetY = ( selected ) ? 2.f : 1.f;
    NSBezierPath *path = [NSBezierPath bezierPathWithRoundedRect:NSInsetRect(self.bounds, 2.f, insetY) xRadius:6.f yRadius:6.f];
    NSColor *fillColor, *strokeColor;

    if ( selected ) {
        fillColor = [NSColor clearColor];
        strokeColor = [NSColor whiteColor];
    } else {
        fillColor = [NSColor colorWithCalibratedRed:95.f/255.f green:159.f/255.f blue:1.f alpha:0.12f];
        strokeColor = [NSColor alternateSelectedControlColor];
    }

    [fillColor setFill];
    [strokeColor setStroke];

    [path setLineWidth:2.f];
    [path fill];
    [path stroke];
}

- (void)drawSelectionInRect:(NSRect)dirtyRect
{
    [super drawSelectionInRect:dirtyRect];
    if ( self.isShowingMenu ) {
        [self drawContextMenuHighlight];
    }
}

- (void)setShowingMenu:(BOOL)showingMenu
{
    if ( showingMenu == _showingMenu )
        return;
    _showingMenu = showingMenu;
    [self setNeedsDisplay:YES];
}

@end

---

Feel free to use any of it, change any of it, or do whatever you want with any of it. Have fun!

---

Updated for Swift 3.x:

SourceListRowView.swift

import Cocoa

open class SourceListRowView : NSTableRowView {

    open var isShowingMenu: Bool = false {
        didSet {
            if isShowingMenu != oldValue {
                needsDisplay = true
            }
        }
    }

    override open func drawBackground(in dirtyRect: NSRect) {
        super.drawBackground(in: dirtyRect)
        if isShowingMenu {
            drawContextMenuHighlight()
        }
    }

    override open func drawSelection(in dirtyRect: NSRect) {
        super.drawSelection(in: dirtyRect)
        if isShowingMenu {
            drawContextMenuHighlight()
        }
    }

    private func drawContextMenuHighlight() {

        let insetY: CGFloat = isSelected ? 2 : 1
        let path = NSBezierPath(roundedRect: bounds.insetBy(dx: 2, dy: insetY), xRadius: 6, yRadius: 6)
        let fillColor, strokeColor: NSColor

        if isSelected {
            fillColor = .clear
            strokeColor = .white
        } else {
            fillColor = NSColor(calibratedRed: 95/255, green: 159/255, blue: 1, alpha: 0.12)
            strokeColor = .alternateSelectedControlColor
        }

        fillColor.setFill()
        strokeColor.setStroke()

        path.lineWidth = 2
        path.fill()
        path.stroke()
    }

}

Note: I haven't actually run this, but I'm pretty sure this should do the trick in Swift.