Should I be concerned about a very slow dictionary attack?

Just block them by changing time interval in your jail rule.

bantime  = 10800 ;3 hours
findtime  = 86400 ;1 day
maxretry = 5

If an attacker fails 5 times in a day, he will be blocked for 3 hours or you can increase that number according to your wish.

Prevent as much as possible, before it will be to late.