Split a large json file into multiple smaller files
Use this code in linux command prompt
split -b 53750k <your-file>
cat xa* > <your-file>
Refer to this link: https://askubuntu.com/questions/28847/text-editor-to-edit-large-4-3-gb-plain-text-file
consider to use jq to preprocessing your json files
it could split and stream your large json files
jq is like sed for JSON data - you can use it to slice
and filter and map and transform structured data with
the same ease that sed, awk, grep and friends let you play with text.
see the official documentation and this questions for more.
extra: for your first questions jq is written by C, it's faster than python/node isn't it ?
Answering the question whether Python or Node will be better for the task would be an opinion and we are not allowed to voice our opinions on Stack Overflow. You have to decide yourself what you have more experience in and what you want to work with - Python or Node.
If you go with Node, there are some modules that can help you with that task, that do streaming JSON parsing. E.g. those modules:
- https://www.npmjs.com/package/JSONStream
- https://www.npmjs.com/package/stream-json
- https://www.npmjs.com/package/json-stream
If you go with Python, there are streaming JSON parsers here as well:
- https://github.com/kashifrazzaqui/json-streamer
- https://github.com/danielyule/naya
- http://www.enricozini.org/blog/2011/tips/python-stream-json/