Bigquery: Partitioning data past 2000 limit (Update: Now 4000 limit)

We are in the process of enforcing the 2000 partitions limitations (so we documented that a bit early to give our users early notice). So it is a soft limit at this point.

Creating large numbers of partitions has performance implications, so we suggest limiting tables to 2000 partitions. We have some room here depending on the schema of the table, so it would be reasonable to ask support if an increase is possible. We will review it depending on how much resources we think would be needed for operations against that table.

We hope to support a larger number of partitions (upto 10K) at some point in the future, but we are working through the design and implementation changes necessary for that (we don't have an ETA on it at this point).

Regarding your question "Is it possible to partition by week or month instead?", there is a feature request to get more flexibility on the type of partitions: https://issuetracker.google.com/issues/35905817

If we are able to also have INT as a type of partition, then it would be easy to define 'months partitions' in the way YYYYMM.