Drupal - View uses the wrong template when cache is enabled

The answer is simple and possible a bit unfortunate for you.

Do not use node--view-* templates. They are broken.

See this core issue here. The problem is that fixing it would result in a performance regression for other sites that do not use that feature; as more cache variations have to be stored if the same node is displayed in multiple views.

Instead of that, use a unique view mode for your view that you can address as a template through that.