Masking layers by a vector polygon layer in QGIS

You can put the colored polygons on top, with a layer blending mode set to darken

Below, have the building layer with the polygon fill in white.

At the bottom, add a new layer containing one large black polygon.

---

Without the black background:

Here's an alternate method for coloring the colored buildings to match the underlying zones.

With Geometry Generator styling, create a separate symbol layer of buildings that intersect each of the zones.

intersection($geometry, geometry(get_feature( 'zones', 'zoneNo', 1)))
intersection($geometry, geometry(get_feature( 'zones', 'zoneNo', 2)))
etc.

Change the color of each symbol layer to match the zone color. Create a black background, eg by setting the map canvas background color to black, or by changing the zone layer style to black-filled simple polygons.

Disclaimer: My test dataset was very small, so I have no idea if this will render faster than the inverted polygon method.