Styling specific layer by using polygon mask in QGIS?

Not a perfect solution but you could make use of the Geometry Generator which adds a visualised line to represent the intersection. You could then set this to overlap the original line feature.

Add a new symbol layer by clicking the plus sign and select the Geometry generator as symbol layer type. Set the geoemtry type to LineString / MultiLineString and use the following expression:

intersection($geometry, geometry(get_feature( 'polygonLayer','fieldName','value'))) 

You would need to add details about your specific polygon where:

  • polygonLayer is the name of your polygon layer
  • fieldName is the name of the field
  • value is the feature value of your specific polygon

Style properties

Note that to colour the visual line, you may need to do it from the Draw effects property:

Draw effects properties

This was the result (note that the visual line did not overlap the original line completely so I modified the offset slightly):

Result

And without the polygon:

Result without polygon



Edit:

If you want this to be applied for each line feature intersecting a polygon feature, go to the Function Editor and use the following function (change the name of polygon example_2 to match the name of your polygon layer):

from qgis.core import *
from qgis.gui import *

@qgsfunction(args='auto', group='Custom')
def func(feature, parent):
    polygon_layer = QgsMapLayerRegistry.instance().mapLayersByName( "polygon example_2" )[0]
    feat_list = []
    geoms = QgsGeometry.fromWkt('GEOMETRYCOLLECTION()')
    for polygon_feat in polygon_layer.getFeatures():
        if feature.geometry().intersects(polygon_feat.geometry()):
            intersection = feature.geometry().intersection(polygon_feat.geometry())
            feat_list.append(intersection)
    for x in feat_list:
        geoms = geoms.combine(x)
    return geoms

Function editor

Click Load then go to the Expression tab and type in func(). Hopefully the result should look like the following (using the same style properties mentioned above):

Final result


Extending on Joseph's answer I came up with this function. It accounts for different coordinate systems and I needed to lookup into two masking layers, thus it handles that too. Further more I wanted to be able to either mask the lines inside the polygons or the lines outside the polygons.

from qgis.core import *
from qgis.gui import *
from qgis.utils import iface

@qgsfunction(args='auto', group='Custom')
def mask_line_with_polygon(mask_type, line_layer_name, polygon_layer_name_1, polygon_layer_name_2, feature, parent):
    line_layer = QgsMapLayerRegistry.instance().mapLayersByName( line_layer_name )[0]

    # This is the geometry outside the polygon mask.
    outside = QgsGeometry(feature.geometry())

    polygon_layer_names = [polygon_layer_name_1, polygon_layer_name_2]
    line_feature_extent = outside.boundingBox()

    geoms = QgsGeometry.fromWkt('MultiLineString()')

    for polygon_layer_name in polygon_layer_names:
        if polygon_layer_name is None or len(polygon_layer_name) == 0:
            continue

        # If the line and the polygon layers have different projections, handle them here.
        polygon_layer = QgsMapLayerRegistry.instance().mapLayersByName(polygon_layer_name)[0]
        trs = QgsCoordinateTransform(line_layer.crs(), polygon_layer.crs())
        polygon_extent = trs.transform(line_feature_extent)
        trs = QgsCoordinateTransform(polygon_layer.crs(), line_layer.crs())

        # Go through the features in the polygon layer, but only those within the line feature bounding box.
        for feature in polygon_layer.getFeatures(QgsFeatureRequest().setFilterRect(polygon_extent)):
            polygon_geometry = QgsGeometry(feature.geometry())

            # Transform the polygon to line space.
            polygon_geometry.transform(trs)

            if outside.intersects(polygon_geometry):
                if mask_type.lower() == 'outside':
                    inside = outside.intersection(polygon_geometry)

                    if inside.isMultipart():
                        for x in inside.asMultiPolyline():
                            geoms.addPart(x)
                    else:
                        geoms.addPart(inside.asPolyline())

                outside = outside.difference(polygon_geometry)

    if mask_type.lower() == 'inside':
        if outside.isMultipart():
            for x in outside.asMultiPolyline():
                geoms.addPart(x)
        else:
            geoms.addPart(outside.asPolyline())

    return geoms

This exercise has shown me that QGIS isn't too fond of working with large data sets and this algorithm with QGIS crashing on my way too often. I suspect that the QGIS renderer doesn't like to render Geometry Generators which are time consuming.