How long should it take to refresh a Developer Sandbox?

A status of Pending means, the sandbox refresh still hasn't started because other people's sandbox refresh requests are ahead of your request in the queue. Therefore, Salesforce doesn't provide SLA time. There's nothing much you can do but wait (indefinitely). Yes it is true that sometimes it can range from seconds to days to get to the front of this queue.

After your request reaches the front of the queue, the status would change from Pending to Processing and then to Ready for activation. Just for the sake of completeness, after you click the Activate link, the status would change to Activating and then to Created.

Since it is a developer sandbox, I'd not worry after crossing the Pending status. It should be fairly quick during the Processing status. There is no data to be copied, except for Custom Settings data. There are no slow queues and no fast queues for developer sandboxes - those are only for full sandboxes.

Related question: What is the maximum time taken for the refresh of a full-copy sandbox?