How bad is it in practice to over-nest selectors in SASS/SCSS?

It depends on how much dynamic manipulation of the DOM and styles will go on after page load. It's not page loads (mostly) or slow selectors on initial layout that are at issue, it's repaints/reflows.

Now, Steve Souders says that on the average site it's simply not a real concern. However, on a web app or highly interactive site, poorly performing CSS rules can make your repaints slower than they have to be. If you have a lot of repaints...

Experts such as Nicole Sullivan, Paul Irish, and Steve Souders have covered the way CSS interacts with with JavaScript and how to write highly performant CSS selectors. It's more than depth (different selectors have different performance), but a good rule of thumb is to limit both depth and complexity to keep yourself out of trouble--but not so much performance trouble, read on.

However, as jankfree.org notes, it's not so much descendant or specific selectors as it is certain properties in certain contexts (html5rocks.com) that make paints expensive. I see long or complicated selectors more as a maintainability issue (Nicolas Gallagher) than a performance issue--keeping in mind that maintainability interacts with performance. Highly maintainable code can iterate faster and is easier to debug (helping you find and fix performance issues).

Now, as to Sass optimization. Yes, Sass can optimize your CSS. But it cannot optimize your selectors. A 4 level nested block will be output as a 4 level nested selector. Sass cannot change it without possibly making your CSS not work. You, as the author, have to optimize the way you write Sass to optimize your output. I, personally, use nesting only in a limited way (a killer feature in Sass for me is composing styles with @extend and placeholders). However, if you really love nesting you might be able to tweak your output to some degree using the Sass parent selector reference (or the newer @at-root).

So far as I know, neither Sass nor Compass has a built-in tool to analyze selectors and warn about them. It's probably possible to create a tool to do that (set a max depth and have your pre-processor warn you) utilizing an AST. More directly, Google Page Speed does have an existing feature that provides some information. SCSS Lint has a nesting option. There's also CSS Lint. (These can theoretically be added to run in your Compass config's on_stylesheet_saved if you're not already using something like Grunt or Gulp).

Just think about how you would want to write the actual css selector. Don't nest everything just because it is a child of the element.

nav li ul li a { 
    /* over specific, confusing */
}
.sub-menu a {
    /* add a class to nested menus */
}

Once you start chaining that many selectors, it becomes a pain to override and can lead to specificity issues.