Is there a way to configure QGIS to prevent the st_extent call when adding a PostGIS table?

Have a look at the settings of PostGIS connection. There is a selection "Use estimated metadata" that is unselected by default. Check it and QGIS will use ST_Estimated_Extent http://postgis.net/docs/manual-2.0/ST_Estimated_Extent.html. Remember to run VACUUM ANALYZE before.

The "use estimated metadata" option on the connection works for newly created layers, but it doesn't change it on existing layers that connect to that database, nor did I find any way to change it in the layer properties (I'm using 3.8.3). I found it relatively quick to change all my existing layers by editing the QGS file to add estimatedmetadata=true to all PostGIS data source descriptions.

Here are the steps I followed using the Linux terminal. Similar steps can be done in any OS, as long as you have a way to search and replace text.

1. Extract QGS file from the QGZ:

   $ unzip map.qgz map.qgs
   

2. Add estimatedmetadata=true to all PostGIS data source descriptions in the QGS. In my particular file, I found these in XML <datasource> and <layer-tree-layer> tags.

   For my particular file, I checked and saw that all such lines had no estimatedmetadata=, so rather than switching estimatedmetadata=false to estimatedmetadata=true, I had to add in a new estimatedmetadata=true. I looked for a string that was in the middle of all of the data source descriptions, so I could do the equivalent of a search and insert by using a text replace. The string I found in my file was checkPrimaryKeyUnicity=, so I ran the following command:

   $ sed -i 's/checkPrimaryKeyUnicity/estimatedmetadata=true checkPrimaryKeyUnicity/g' map.qgs
   

   You may have to adapt this to the details of your specific file, make the changes in a more manual way, or use a tool with XML-specific editing features.

3. Replace the QGS file in the QGZ:

   $ zip -u map.qgz map.qgs