How can I partially read a huge CSV file?
In addition to EdChums answer I find the nrows
argument useful which simply defines the number of rows you want to import. Thereby you don't get an iterator but rather can just import a part of the whole file of size nrows
. It works with skiprows
too.
df = pd.read_csv('matrix.txt',sep=',', header = None, skiprows= 1000, nrows=1000)
Use chunksize
:
for df in pd.read_csv('matrix.txt',sep=',', header = None, chunksize=1):
#do something
To answer your second part do this:
df = pd.read_csv('matrix.txt',sep=',', header = None, skiprows=1000, chunksize=1000)
This will skip the first 1000 rows and then only read the next 1000 rows giving you rows 1000-2000, unclear if you require the end points to be included or not but you can fiddle the numbers to get what you want.