What is the advantage of iSCSI over SMB?

Mike is correct, iSCSI and SMB/CIFS operate at two different layers of abstraction.

You can think of SMB/CIFS as exporting a file system that other machines can access. The directory structure, security metadata, and such is already there. Client machines can read and write files to this file system, but that is the extent of their access.

With iSCSI, the client machines see what amounts to the same thing as a physical volume from the operating system point of view. From a Windows client machine, you would first configure an iSCSI initiator pointing at the iSCSI host (target). After that, you would see a new physical disk in the Windows disk management control panel. You would then initialize it, partition it, then format it with whatever file system you wished. No other machine would have access to this area (at least simultaneously and ignoring advanced things like cluster file systems).

The process would be a little different from ESXi but the concept is the same - the iSCSI target would appear like another disk on which you could build a VMFS file system.

One thing you may consider if you are building your own storage server is that it is possible to do both things at once. Software such as OpenFiler and FreeNAS allow you to aggregate a pool of storage and allow both iSCSI and SMB/CIFS access. The iSCSI space must be pre-allocated (much a like a virtual machine hard disk) and is not visible to the CIFS clients. The two areas are separate. You could also stay with a Windows server and install software to export an iSCSI target, such as that made by StarWinds.

iSCSI in general will perform better because there will not be as many layers of abstraction in the way - the SMB protocol in this case. Also, since it is a block-level protocol, some software that will not work when installed to a network share will work with iSCSI.

I was thinking that ESXi may not allow access to SMB shares, NFS only, so you may want to look into that as well. If that is true, you will need some way to export NFS shares but the above discussion remains the same. OpenFiler/FreeNAS have NFS support and Windows can be setup with it too.

File Level VS block level

Block Level (iSCSI) You can format a remote volume with your local operating systems file system.

Let's say, you want to have a volume for Apple Time-machine and you wanted this as an Apple Journalized file system. or you wanted a remote filesystem formatted as NTFS. to format a disk you need to have block access to the disk.

SMB are already formatted by the device hosting SMB. these file systems are typically EXT2&3 and a Linux Subsystem is allowing you access to write on its own local file system over the SMB protocol.

I think SMB is your typical network share whereas iSCSI is presented as a exclusive direct-attached block-based storage (like a locally installed HDD) and therefore allows your choice of file system with its own associated pros and cons. (But I've been wrong before :)