Per-directory history in zsh

There is a plugin that claims to do exactly what you are looking for, appropriately named per directory history plugin:

https://github.com/jimhester/oh-my-zsh/commit/baa187e4b903f39422a84b580e6e617ec3738e09

"Per-directory-history - tracks previous command history both per current directory and globally, with the ability to switch between them on the fly, bound to ctrl-g." says their wiki.

I did not test it myself (yet), but according to the comments it should work.