Viewing a very large CSV file?
Yes, there is.
You can use OpenRefine (or Google Refine). OpenRefine is like a spreadsheet on steroids.
The file size that you can manipulate depend on your computer's memory.
Since it is a CSV file.
- Download http://openrefine.org/download.html
- It is open source. Unzip openrefine.zip.
- Run openrefine-2.7-rc.1\openrefine.exe.
- It is a web app. So open http://127.0.0.1:3333/ in Chrome.
- Upload the large csv file. In my case the file size was 3.61 GB and it got opened successfully.
https://snag.gy/02WECq.jpg
You could try PostgreSQL 9.1+ and its file_fdw (File Foreign Data Wrapper) which would pretend that the CSV file is a table. If you replaced the CSV file with another CSV file of the same name, then you would see the new info immediately in the database.
You can improve performance by using a materialized view (PG 9.3+) which essentially creates a real database table from the CSV data. You could use pgAgent to refresh the materialized view on a schedule.
Another alternative would be to use the COPY statement:
/* the columns in this table are the same as the columns in your csv: */
create table if not exists my_csv (
some_field text, ...
);
/* COPY appends, so truncate the table if loading fresh data again: */
truncate table my_csv;
/*
you need to be a postgres superuser to use COPY
use psql \copy if you can't be superuser
put the csv file in /srv/vendor-name/
*/
copy
my_csv
from
'/srv/vendor-name/my.csv'
with (
format csv
);