How to import data from cloud firestore to the local emulator?
My method is somewhat manual but it does the trick. I've shared it in this useful Github thread but I'll list the steps I did here if you find them useful:
- Go to my local Firebase project path.
- Start the emulators using:
firebase emulators:start
- Create manually some mockup data using the GUI at http://localhost:4000/firestore using the buttons provided: + Start Collection and + Add Document.
- Export this data locally using:
emulators:export ./mydirectory
- About the project data located at Firebase Database / Cloud Firestore, I exported a single collection like this:
gcloud firestore export gs://my-project-bucket-id.appspot.com --collection-ids=myCollection
The export is now located under Firebase Storage in a folder with a timestamp as name (I didn't use a prefix for my test) - Download this folder to local drive with:
gsutil cp -r gs://my-project-bucket-id.appspot.com/myCollection ./production_data_export
NOTE: I did this in a Windows environment... gsutil will throw this error: "OSError: The filename, directory name, or volume label syntax is incorrect" if the folder has invalid characters for a folder name in Windows (i.e. colons) or this error: "OSError: Invalid argument.9.0 B]" if an inner file in the folder has invalid characters too. To be able to download the export locally, rename these with a valid Windows name (i.e. removing the colons) like this:gsutil mv gs://my-project-bucket-id.appspot.com/2020-05-22T02:01:06_86152 gs://my-project-bucket-id.appspot.com/myCollection
- Once downloaded, imitate the local export structure renaming the folder to
firestore_export
and copying thefirebase-export-metadata.json
file from the local export folder. Just to be visual, here's the structure I got:
$ tree .
.
├── local_data_export
│ ├── firebase-export-metadata.json
│ └── firestore_export
│ ├── all_namespaces
│ │ └── all_kinds
│ │ ├── all_namespaces_all_kinds.export_metadata
│ │ └── output-0
│ └── firestore_export.overall_export_metadata
└── production_data_export
├── firebase-export-metadata.json
└── firestore_export
├── all_namespaces
│ └── kind_myCollection
│ ├── all_namespaces_kind_myCollection.export_metadata
│ ├── output-0
│ └── output-1
└── firestore_export.overall_export_metadata
8 directories, 9 files
- Finally, start the local emulator pointing to this production data to be imported:
firebase emulators:start --import=./mock_up_data/production_data_export/
- You should see the imported data at: http://localhost:4000/firestore/
This should assist readers for now, while we await a more robust solution from the Firebase folks.
This can be accomplished through a set of commands in terminal on the existing project:
1. Login to firebase and Gcloud:
firebase login
gcloud auth login
2. See a list of your projects and connect to one:
firebase projects:list
firebase use your-project-name
gcloud projects list
gcloud config set project your-project-name
3. Export your production data to gcloud bucket with chosen name:
gcloud firestore export gs://your-project-name.appspot.com/your-choosen-folder-name
4. Now copy this folder to your local machine, I do that in functions folder directly:
cd functions
gsutil -m cp -r gs://your-project-name.appspot.com/your-choosen-folder-name .
5. Now we just want to import this folder. This should work with the basic command, thanks to latest update from Firebase team https://github.com/firebase/firebase-tools/pull/2519.
firebase emulators:start --import ./your-choosen-folder-name
Check out my article on Medium about it and a shorthanded script to do the job for you https://medium.com/firebase-developers/how-to-import-production-data-from-cloud-firestore-to-the-local-emulator-e82ae1c6ed8
Note: Its better to use a different bucket for it, as copying into your project bucket will result in the folder created in your firebase storage.
If you are interested in gsutil arguments like -m
, you can see them described by executing gsutil --help
.