Getting Bigquery to auto detect schema
Here the solution for node js .
const filename = "path of file to be uploaded";
const metadata = {
sourceFormat: 'CSV',
skipLeadingRows: 1,
autodetect: true,
};
const dataset =await
bigQuery.dataset('datasetname').table('tablename').load(filename,metadata);
As stated in a previous answer, schema auto-detection is a part of federated data source querying. It is not explicitly stated under the CSV and JSON schema auto-detection heading that this does not apply to bq load
. If you feel this is unclear in the documentation, I would strongly suggest clicking Send feedback at the top righthand corner of that documentation page and describing this ambiguity in detail.
As for the bq load
command, according to the bq load
documentation, the table schema is a required parameter. Omitting it does result in the error message you are facing.
EDIT
Thanks to polleyg for the update. In this blog post, it was announced that the schema should also be detected at load. As mentioned by Chris Sears, the --autodetect
flag should meet your needs.
The --autodetect
flag is probably what you want. It works for CSV and (newline delimited) JSON input files.
For example...
bq load --source_format=NEWLINE_DELIMITED_JSON --autodetect yourdataset.yourtable inputfile.json
See documentation here: https://cloud.google.com/bigquery/bq-command-line-tool#creatingtablefromfile
Note that this has nothing to do with federated data sources.