Drupal - Where is links__system_main_menu defined?

Drupal 7 core doesn't actually define this function.

links__system_main_menu is a theme hook pattern of the form [base hook]__[context]. When links are themed with theme('links__system_main_menu', $vars), theme() will search for and use theme_links__system_main_menu() if it has been defined. If not, it will use theme_links().

See theme() for the official documentation of this behavior. Chapter 3 of Drupal 7 Module Development does an excellent job of explaining this and provides several examples.

BTW, the same principle applies to template files. If, for example, we call theme('node__article__1', $vars), theme() will search for a node--article--1.tpl.php file, and then for node--article.tpl.php, finally falling back to node.tpl.php if neither of them are defined.

theme_links__system_main_menu() is not currently defined from Drupal, but Drupal will use MYTHEME_links__system_main_menu(), if you define it in your theme.

function MYTHEME_links__system_main_menu($variables) {
  $html = "<div>\n";
  $html .= "  <ul>\n"; 

  foreach ($variables['links'] as $link) {
    $html .= "<li>".l($link['title'], $link['path'], $link)."</li>";
  }

  $html .= "  </ul>\n";
  $html .= "</div>\n";

  return $html;
}

You can find some more info on http://drupal.org/node/1033442#comment-5076932.