Python copy larger file too slow
Your problem has nothing to do with Python. In fact, the Windows copy process is really poor compared to the Linux system.
You can improve this by using xcopy
or robocopy
according to this post: Is (Ubuntu) Linux file copying algorithm better than Windows 7?. But in this case, you have to make different calls for Linux and Windows...
import os
import shutil
import sys
source = "source\to\large\file"
target = "destination\directory"
if sys.platform == 'win32':
os.system('xcopy "%s" "%s"' % (source, target))
else:
shutil.copy(source, target)
See also:
- Actual Performance, Perceived Performance, Jeff Atwood blog post about this subject
- Windows xcopy is not working in python, which gives the syntax for using xcopy on Windows in fact
Just to add some interesting information: WIndows does not like the tiny buffer used on the internals of the shutil implementation.
I've quick tried the following:
- Copied the original shutil.py file to the example script folder and renamed it to myshutil.py
- Changed the 1st line to
import myshutil
- Edited the myshutil.py file and changed the copyfileobj from
def copyfileobj(fsrc, fdst, length=16*1024):
to
def copyfileobj(fsrc, fdst, length=16*1024*1024):
Using a 16 MB buffer instead of 16 KB caused a huge performance improvement.
Maybe Python needs some tuning targeting Windows internal filesystem characteristics?
Edit:
Came to a better solution here. At the start of your file, add the following:
import shutil
def _copyfileobj_patched(fsrc, fdst, length=16*1024*1024):
"""Patches shutil method to hugely improve copy speed"""
while 1:
buf = fsrc.read(length)
if not buf:
break
fdst.write(buf)
shutil.copyfileobj = _copyfileobj_patched
This is a simple patch to the current implementation and worked flawlessly here.
Python 3.8+: Python 3.8 includes a major overhaul of shutil, including increasing the windows buffer from 16KB to 1MB (still less than the 16MB suggested in this ticket). See ticket , diff