Sharepoint - Wrong counts for refiner values in SharePoint 2013, Enterprise Search Center
When was the last time you ran a full crawl since this issue? I'll be clear, I don't think that's the issue but managed properties (the refiners) are very unique and a full crawl is needed before you can even search against them. I wouldn't be half-surprised, as unbelievable as it sounds, that you would need to do a full crawl for the refiners to show the correct amounts. Perhaps that's why Microsoft doesn't show them by default. The managed properties may only be pulling items from the last full crawl index.
Aside from that, could you post your search display template that shows the number as well as let us know what types of searches you are running?