Reliable Collection Caching as Cache in Service Fabric

If the Reliable Dictionary is meant to act as a cache, then I don't really see the point of offloading unused items to Azure Storage. If it's a cache, I would expect unused items to be purged, and caller would need to go back to the source of truth for anything that has expired from the cache. But it sounds like you want the Reliable Dictionary to be an up-to-date source of truth. So I think you have to first decide if you're actually building a cache, or a source of truth data store that can page data out of memory. It sounds more like the latter.

In either case, it can be done as you described, but keeping them in sync consistently won't be easy because you don't have a transaction across a Reliable Dictionary and an external store.

Enumerating a collection is fine but it is an expensive operation, so I would not recommend doing it on large amounts of data in a hot path, such as a user request path. It's ok to do it periodically in a scheduled manner.

Do you need to offload data to external storage? Can you offload to local disk? Reliable Collections will soon do offloading of state to disk automatically.

I would use an actor. Give each item it's own actor and store the state there. When the actor is garbage collected, you could save state somewhere else, or simply do that on an actor timer.

Doing this means you won't have to replicate a lot of the actor code to manage lots of instances.

CAVEAT

This makes sense if your overall design make sense. As Vaclav's comment below says, actors are not good for a general purpose cache due to the single threaded model for actors. But if your design has an actor representing a single entity and the caching is related to that entity (such as a user), then treating the actor as a cache can work well.

The Team at SoCreate just released an open source project called Service Fabric Distributed Cache that might help you or other people using Service Fabric and need a cache. We built this so we wouldn't need to run Redis or something like that as a guest exe in Service Fabric. This gived you a way to run, monitor, and manage your cache as a Service Fabric Reliable Service. You can learn more about it here:

http://service-fabric-distributed-cache.socreate.it/

or on GitHub here: https://github.com/SoCreate/service-fabric-distributed-cache