What is the more accepted practice for defining tolerance values?

If the map units are in any sort of projected coordinate system, taking input in meters, miles, or beard-seconds should be relatively easy. If the map units are geographic, you probably ought to project them for an operation like this anyhow to make your algorithm easier (the problem with geographic coordinates is that north-south degrees are not the same ground length as east-west degrees; to make matters worse, the ratio varies depending on your latitude).

When buffering, ArcGIS takes the input in user-selectable units and converts them on-the-fly to map units. This can make more sense, since it's a lot easier to think of a 5 mile buffer than a 26,400 foot buffer. So, I think #1 is probably better from a UX standpoint.

Just to throw something out there, I think most people would expect to be able to specify the tolerance units independent of the units the data is actually stored in.

You could either a) make the unit selection a dropdown list or b) parse it out from the tolerance text input area.

In the case of b) you would for example have a value of 500 Feet entered and you would parse out the word Feet, make sure it matches a list of supported measurement units, and use that unit for your tolerance calculation, converting it to the dataset's measurement unit if necessary. If no unit is entered then assume it is in the same units as the dataset.