Copying terabytes of hundreds of thousands of files in folder slow

The root cause for the slow transfer rate is, possibly, the fact that the workstation M2 drives need to do a lot of random reads.

The fast NVMe M2s (that you are most likely using, I think) are advertised with up to multiple GB/s r/w speeds. That is true for sequential reads for big files, but in your situation you will have random reads instead. Random read rates for common consumer/prosumer NVMe M2 SSDs range from 70MB/s to 110MB/s, that is within your rate of 600Mbps. Reviews of SSDs will often include random read speed results which is where I got that range from.

There are SSDs such as Intel Optane SSDs that can deliver random read speeds in the ballpark of around 500MB/s.

Furthermore you state that you connect the drives via USB-C. Depending on what technology is used, USB3.0, 3.1, 3.2 or Thunderbolt, this connection might cause slowdowns as well. Internal NVMe M2 drives (or other faster PCI-e based ones) might resolve the problem.

To prove or invalidate my assumption, you can use the Windows 10 task manager or the performance monitor. The task manager will give you a percentage of how busy the drives are. If the drive(s) in question sit at 100% or at anything above 80%, then they are likely limiting the speeds. On the other hand, if it is idling, then it is not limiting. Disclaimer: I do not know how reliable the busy percentages of the windows task manager are, especially not for external drives.

If it turns out that the drives on the source side are not busy at all, you might want to check the destination side and see how the drives are doing there (you can use the tool iostat for that).

If none of this helps you because you were able to exclude both the drives on the source and the destination sides as being the root cause for the issue, then I suggest you start with basic troubleshooting steps. For example, you can transfer a big file across and see if this transfer suffers the same limitations. You could reverse the transfer direction and copy some of the small files back onto the workstations. If just the reversal leads to much better speeds, then perhaps there is one component that limits only when reading and not when writing, or vice versa.

Or try to rule out some components by attaching devices directly with no extra switch in between or whatever you can remove from the scenario for testing.

Ok looks like the issue is resolved now. Here was the solution.

In the /etc/samba/smb-shares.conf.local

This line was added to the share we are using

case sensitive = yes

Now we are transferring at a stable 200MBps. While not the ideal speed, it is not decreasing in speed overtime. This fixes the speed decreasing issue.

Without extensive profiling on both source and destination, it is difficult to give a definitive answer. That said, I do not think that the source NVMe drive is the bottleneck; after all, you are reading quite large files, with a considerable amount or sequential reads.

Due to the high number of files involved, I lean more towards inefficiencies in NTFS and/or the SMB protocol itself.

I suggest you to try the following:

• on the destination host, create a dedicated dataset with disabled sync, checksum and compression (ie: zfs set sync=disabled <dataset>, etc). Note: you should consider this as a test and/or a temporary solution only, I do not suggest to permanently run with these settings off;

• on the source host, try to boot with a linux live cd/usb and to transfer files with the NFS protocol (rather than SMB). You should basically do the following:

  • boot with the live cd;
  • install nfs and ntfs-3g utilities;
  • mount the NTFS filesystem (ie: in /mnt/localdir);
  • configure a NFS export on destination;
  • mount it on source host (ie: mount x.x.x.x:/dstdir /mnt/localdir);
  • use cp or rsync to transfer these files;
  • on another terminal, try running dstat -d -f -n on both hosts to monitor the file transfer.