which method is called when selecting a cell in the NSTableView in Cocoa OS X?

In Swift 5.4:

Just use tableView.action = #selector(YOUR_METHOD), then in your method, try to do something with tableView.selectedRow.

This would simply do the trick. Take a look at the demo below:

override func viewDidLoad() {
    super.viewDidLoad()
    // Do view setup here.

    tableView.action = #selector(didSelectRow)
}

@objc private func didSelectRow() {
    print("Selected row at \(tableView.selectedRow)")
}

But please be aware that if nothing is selected, or you clicked outside of a cell, then you'll get a tableView.selectedRow of -1. So you might wanna check if the index is out of range before you use it. :)

Swift 3 (from Eimantas' answer):

func tableViewSelectionDidChange(_ notification: NSNotification) {
    let table = notification.object as! NSTableView
    print(table.selectedRow);
}

What is tableViewController object? Only NSTableView instances respond to selectedRow. You can get current table view (the one that sent the notification) from notification's object property:

Objective-C:

-(void)tableViewSelectionDidChange:(NSNotification *)notification{
    NSLog(@"%d",[[notification object] selectedRow]);
}

Swift:

func tableViewSelectionDidChange(notification: NSNotification) {
    let table = notification.object as! NSTableView
    print(table.selectedRow);
}

my 2 cents for Xcode 10/swift 4.2

  func tableViewSelectionDidChange(_ notification: Notification) {
        guard let table = notification.object as? NSTableView else {
            return
        }
        let row = table.selectedRow
        print(row)
    }