Is django prefetch_related supposed to work with GenericRelation

If you want to retrieve Book instances and prefetch the related tags use Book.objects.prefetch_related('tags'). No need to use the reverse relation here.

You can also have a look at the related tests in the Django source code.

Also the Django documentation states that prefetch_related() is supposed to work with GenericForeignKey and GenericRelation:

prefetch_related, on the other hand, does a separate lookup for each relationship, and does the ‘joining’ in Python. This allows it to prefetch many-to-many and many-to-one objects, which cannot be done using select_related, in addition to the foreign key and one-to-one relationships that are supported by select_related. It also supports prefetching of GenericRelation and GenericForeignKey.

UPDATE: To prefetch the content_object for a TaggedItem you can use TaggedItem.objects.all().prefetch_related('content_object'), if you want to limit the result to only tagged Book objects you could additionally filter for the ContentType (not sure if prefetch_related works with the related_query_name). If you also want to get the Author together with the book you need to use select_related() not prefetch_related() as this is a ForeignKey relationship, you can combine this in a custom prefetch_related() query:

from django.contrib.contenttypes.models import ContentType
from django.db.models import Prefetch

book_ct = ContentType.objects.get_for_model(Book)
TaggedItem.objects.filter(content_type=book_ct).prefetch_related(
    Prefetch(
        'content_object',  
        queryset=Book.objects.all().select_related('author')
    )
)

prefetch_related_objects to the rescue.

Starting from Django 1.10 (Note: it still presents in the previous versions, but was not part of the public API.), we can use prefetch_related_objects to divide and conquer our problem.

prefetch_related is an operation, where Django fetches related data after the queryset has been evaluated (doing a second query after the main one has been evaluated). And in order to work, it expects the items in the queryset to be homogeneous (the same type). The main reason the reverse generic generation does not work right now is that we have objects from different content types, and the code is not yet smart enough to separate the flow for different content types.

Now using prefetch_related_objects we do fetches only on a subset of our queryset where all the items will be homogeneous. Here is an example:

from django.db import models
from django.db.models.query import prefetch_related_objects
from django.core.paginator import Paginator
from django.contrib.contenttypes.models import ContentType
from tags.models import TaggedItem, Book, Movie


tagged_items = TaggedItem.objects.all()
paginator = Paginator(tagged_items, 25)
page = paginator.get_page(1)

# prefetch books with their author
# do this only for items where
# tagged_item.content_object is a Book
book_ct = ContentType.objects.get_for_model(Book)
tags_with_books = [item for item in page.object_list if item.content_type_id == book_ct.id]
prefetch_related_objects(tags_with_books, "content_object__author")

# prefetch movies with their director
# do this only for items where
# tagged_item.content_object is a Movie
movie_ct = ContentType.objects.get_for_model(Movie)
tags_with_movies = [item for item in page.object_list if item.content_type_id == movie_ct.id]
prefetch_related_objects(tags_with_movies, "content_object__director")

# This will make 5 queries in total
# 1 for page items
# 1 for books
# 1 for book authors
# 1 for movies
# 1 for movie directors
# Iterating over items wont make other queries
for item in page.object_list:
    # do something with item.content_object
    # and item.content_object.author/director
    print(
        item,
        item.content_object,
        getattr(item.content_object, 'author', None),
        getattr(item.content_object, 'director', None)
    )

Building on Bernhard's answer, which has a code-snippet at the end that throws the below error in reality:

ValueError: Custom queryset can't be used for this lookup.

I've overridden the GenericForeignKey to actually allow the behavior, how bulletproof this implementation is, is unknown to me at this time but it seems to get what I need done, so I'm posting it here, hopefully it'll help out others. Please lookout for START CHANGES and END CHANGES tags to see my changes to the original django code.

from django.contrib.contenttypes.fields import GenericForeignKey as BaseGenericForeignKey

class CustomGenericForeignKey(BaseGenericForeignKey):
    def get_prefetch_queryset(self, instances, queryset=None):
        """
        Enable passing queryset to get_prefetch_queryset when using GenericForeignKeys but only works when a single
        content type is being queried
        """
        # START CHANGES
        # if queryset is not None:
        #     raise ValueError("Custom queryset can't be used for this lookup.")
        # END CHANGES

        # For efficiency, group the instances by content type and then do one
        # query per model
        fk_dict = defaultdict(set)
        # We need one instance for each group in order to get the right db:
        instance_dict = {}
        ct_attname = self.model._meta.get_field(self.ct_field).get_attname()
        for instance in instances:
            # We avoid looking for values if either ct_id or fkey value is None
            ct_id = getattr(instance, ct_attname)
            if ct_id is not None:
                fk_val = getattr(instance, self.fk_field)
                if fk_val is not None:
                    fk_dict[ct_id].add(fk_val)
                    instance_dict[ct_id] = instance

        ret_val = []
        for ct_id, fkeys in fk_dict.items():
            instance = instance_dict[ct_id]
            # START CHANGES
            if queryset is not None:
                assert len(fk_dict) == 1  # only a single content type is allowed, else undefined behavior
                ret_val.extend(queryset.filter(pk__in=fkeys))
            else:
                ct = self.get_content_type(id=ct_id, using=instance._state.db)
                ret_val.extend(ct.get_all_objects_for_this_type(pk__in=fkeys))
            # END CHANGES

        # For doing the join in Python, we have to match both the FK val and the
        # content type, so we use a callable that returns a (fk, class) pair.
        def gfk_key(obj):
            ct_id = getattr(obj, ct_attname)
            if ct_id is None:
                return None
            else:
                model = self.get_content_type(id=ct_id,
                                              using=obj._state.db).model_class()
                return (model._meta.pk.get_prep_value(getattr(obj, self.fk_field)),
                        model)

        return (
            ret_val,
            lambda obj: (obj.pk, obj.__class__),
            gfk_key,
            True,
            self.name,
            True,
        )