Is there a high-performance local area "file sharing" protocol?

What is wrong with NFS 4?

Modern security -- ideally, uses public/private keys like SSH. Tunneling over TLS would be great.

NFS 4 offers modern authentication. Data security should be done via a VPN.

High performance -- scanning a file system or reading bulk data should run at the speed that the server and network can support.

NFS 4 is reasonable high performance for the general case. Lots of caching, aggregation of commands, etc.

Native clients for Windows and Linux -- others would be gravy but not interesting to me.

I am not aware of anything here. Any reasonable wide-spread solution comes from one of the corners and is only supported as secondary citizien in the other world. That said: I think it is fair to say that e.g. SMB is better supported in the Unix world (via Samba) than e.g. NFS is in the Windows world.

File locking compatible with Linux and Windows.

NFS gives you this.

Change notification compatible with Linux and Windows.

Good question. I don't think that NFS supports this in a good way. At least I am not aware of any command that e.g. would call a callback if something changes on the (meta data) server.

To your 2 MB/s with SMB? There should be something misconfigured. SMB is certainly capable of more than that. 100 MB/s (GB wirespeed) should be no problem in a reasonable configuration.

The fastest file sharing protocol in the world isn't going to be able to make up for the fact that some apps aren't written with remote filesystems in mind, so they dispatch lots of small reads or writes, and wait for one operation to complete before starting the next one. This is fine on the relatively low latency of a local disk, but it's painful over a network, where the latency is much higher.

If you want scanning of a remote media library to go fast, you need to have code running on the server do the scanning and report the results back to the client process in one big lump.