Apple - Prevent the Siri Watch Face from displaying Photo Memories

Photo Memories, along with other "Data Sources" for the Siri Watch Face, can be turned off in the Watch app on iOS.

Watch app > My Watch > My Faces > Siri (Watch Face) > Data Sources

You can also turn off Breathe, Home, Stocks, etc.