Checking if feature visible using PyQGIS

Another solution, using willRenderFeature. This works regardless of the renderer type. You need to clone the renderer, and call willRenderFeature between startRender and stopRender, otherwise QGIS crashes (cf. pyQGIS API).

You can create the QgsRenderContexton the fly, but you need to call setFeature() on its expressionContext for each feature.

def getVisibleSelectedFeatures(layer):
    visibleFeatures = []
    renderer = layer.renderer().clone()
    ctx = QgsRenderContext()
    renderer.startRender(ctx, QgsFields())
    for feature in layer.selectedFeatures():
        ctx.expressionContext().setFeature(feature)
        if renderer.willRenderFeature(feature, ctx):
            visibleFeatures.append(feature)
    renderer.stopRender(ctx)
    return visibleFeatures

You can also slightly change the function to make it a generator, in case you need to work with a large set of data:

def getVisibleSelectedFeatures(layer):
    renderer = layer.renderer().clone()
    ctx = QgsRenderContext()
    renderer.startRender(ctx, QgsFields())
    for feature in layer.selectedFeatures():
        ctx.expressionContext().setFeature(feature)
        if renderer.willRenderFeature(feature, ctx):
            yield feature
    renderer.stopRender(ctx)

Usage:

for feature in getVisibleSelectedFeatures(iface.activeLayer()):
   # Do something with the feature

I can't find a simple way of doing what you ask but the following code seems to do the job by looking at the renderer of the active layer, looping through the categories, checking for any where the render state is false and adds the render category value to a list. It loops through the selected layers and tests the attribute used in the categorized symbology to see if it appears on the list, if not it adds the feature to the visibleSelectedFeatures list. The code runs from the python window in QGIS, tested using 3.4:

iface.activeLayer().selectByRect(extent)
selectedFeaturesList = list(iface.activeLayer().selectedFeatures())
visibleSelectedFeatures = []
layer = iface.activeLayer()
renderer = layer.renderer()
invisible = list()
for rc in renderer.categories():
    if rc.renderState() is False:
        invisible.append(rc.value())

columnindex=iface.activeLayer().fields().indexFromName(renderer.classAttribute())
for feat in selectedFeaturesList:
    attrs = feat.attributes()
    if attrs[columnindex] not in invisible:
        visibleSelectedFeatures.append(feat)