Viewing a very large CSV file?

Yes, there is.

You can use OpenRefine (or Google Refine). OpenRefine is like a spreadsheet on steroids.

The file size that you can manipulate depend on your computer's memory.


Since it is a CSV file.

  1. Download http://openrefine.org/download.html
  2. It is open source. Unzip openrefine.zip.
  3. Run openrefine-2.7-rc.1\openrefine.exe.
  4. It is a web app. So open http://127.0.0.1:3333/ in Chrome.
  5. Upload the large csv file. In my case the file size was 3.61 GB and it got opened successfully.

https://snag.gy/02WECq.jpg


You could try PostgreSQL 9.1+ and its file_fdw (File Foreign Data Wrapper) which would pretend that the CSV file is a table. If you replaced the CSV file with another CSV file of the same name, then you would see the new info immediately in the database.

You can improve performance by using a materialized view (PG 9.3+) which essentially creates a real database table from the CSV data. You could use pgAgent to refresh the materialized view on a schedule.

Another alternative would be to use the COPY statement:

/* the columns in this table are the same as the columns in your csv: */
create table if not exists my_csv (
  some_field text, ...
);

/* COPY appends, so truncate the table if loading fresh data again: */
truncate table my_csv;

/* 
you need to be a postgres superuser to use COPY 
use psql \copy if you can't be superuser 
put the csv file in /srv/vendor-name/
*/

copy 
  my_csv 
from 
  '/srv/vendor-name/my.csv'
with (
  format csv
);

Tags:

Database

Csv