Split a large json file into multiple smaller files

Use this code in linux command prompt

split -b 53750k <your-file>
cat xa* > <your-file>

Refer to this link: https://askubuntu.com/questions/28847/text-editor-to-edit-large-4-3-gb-plain-text-file


consider to use jq to preprocessing your json files

it could split and stream your large json files

jq is like sed for JSON data - you can use it to slice 
and filter and map and transform structured data with 
the same ease that sed, awk, grep and friends let you play with text.

see the official documentation and this questions for more.

extra: for your first questions jq is written by C, it's faster than python/node isn't it ?


Answering the question whether Python or Node will be better for the task would be an opinion and we are not allowed to voice our opinions on Stack Overflow. You have to decide yourself what you have more experience in and what you want to work with - Python or Node.

If you go with Node, there are some modules that can help you with that task, that do streaming JSON parsing. E.g. those modules:

  • https://www.npmjs.com/package/JSONStream
  • https://www.npmjs.com/package/stream-json
  • https://www.npmjs.com/package/json-stream

If you go with Python, there are streaming JSON parsers here as well:

  • https://github.com/kashifrazzaqui/json-streamer
  • https://github.com/danielyule/naya
  • http://www.enricozini.org/blog/2011/tips/python-stream-json/