SSL/https breaking when using www

Browsers don’t further access the server when the certificate is not trusted/valid, so the .htaccess redirect can’t work (it should work as soon as the user adds the certificate as an exception).

The best solution is to get a certificate for the hostname with www, too. This does not only help for redirecting users to the correct hostname when following links from search results or bookmarks, it also helps those users that type your hostname with www (which is commonly done, even if it’s advertised without www), which might still happen long after search engines have removed the hostname with www from their indexes.

If that’s not possible, you can at least signal search engines that they should prefer the hostname without www. A 301 redirect is the best way here. If that’s not possible, the second best is the canonical link type. Of course both can only work for search engines that ignore the bad certificate (I guess most do).

You could also try to use the search engine’s webmaster tools to set the preferred hostname. For Google, see Set your preferred domain (www or non-www) (but I don’t know if this works for hosts without a trusted certificate).

In any case, you have to wait. After some time, the hostname with www should disappear from search results.

The "bad cert thing" is triggered by the browser during the SSL handshake. This is the very first thing that happens and occurs long before .htaccess or any of your server-side logic has a chance to do anything. As you suggest, "the browser is refusing the connection before htaccess has a chance to act".

So, if your SSL cert is only valid for the apex domain and Google has somehow indexed https://www.example.com/ or people follow links to https://www.example.com/ then the only thing you can do to resolve this immediate issue is to "fix" your cert.

a way to keep Google from indexing the pages with www

Bit of a puzzle why it should "keep" doing this, unless the HTTPS/www version is being linked to in various places? (Did you once use the http://www.example.com version, before using the SSL cert?)

With the properties verified in Google Search Console (GSC) you can at least set a preference between www and non-www.

While it is a good idea to get the certificate for your www subdomain as well, you could try setting up a redirect at your nameserver level too. Almost all of the DNS solution providers have the feature to enable a name mask or a domain forwarding.

I am not sure whether it is also applicable at the http level, or if it works for requests at the HTTPS protocol, but it is worth a shot.