Drupal - How do I pass a variable from a custom module to its template file?

Independently from the Drupal version for which you are writing the module, there are two errors in your code:

• You define "Bluemarine" as theme function, but then you call theme('custom'), which would call the "custom" theme function
• If you define "custom" as a theme function that uses a template file, then theme_custom() is never called

If you are writing code for Drupal 6, then the code should be similar to the following one. I take the assumption the name for the theme function is custom.

function custom_menu(){
  $items = array();

  $items['custom'] = array(
    'title' => t('custom!'),
    'page callback' => 'custom_page',
    'access arguments' => array('access content'),
    'type' => MENU_CALLBACK,
  );

  return $items;
}

function custom_theme() {
  return array(
    'custom' => array(
      'arguments' => array('output' => NULL),
      'template' => 'custom',
     ),
  );
}

function custom_page() {
    $output = 'This is a custom module';
    return theme('custom', $output);    
}

function theme_custom($output) {
}

The template file will have access to $output, and to any variables set in template_preprocess_custom(), if your module implements it.

For example, you could implement code similar to the following one:

function template_preprocess_custom(&$variables) {
  if ($variables['output'] == 'This is a custom module') {
    $variables['append'] = ' and I wrote it myself.';
  }
}

With this code, the template file has access to $output and $append.

As example of theme function that uses a template file, you can look at theme_node(), which is defined in node_theme(), and that uses node.tpl.php as template file; the preprocess function implemented by the Node module for that theme function is template_preprocess_node().

You are calling the wrong theme function. Instead of function theme_custom it should be function theme_Bluemarine. You also need to pass an array to the variables piece of hook_theme(). See a simple example here.

Using your example (after changing template and theme function to custom):

function custom_menu(){
  $items = array();

  $items['custom'] = array(
    'title' => t('custom!'),
    'page callback' => 'custom_page',
    'access arguments' => array('access content'),
    'type' => MENU_CALLBACK,
  );

  return $items;
}

function custom_page() {
  $setVar = 'this is custom module';
  return theme('custom', array('output' => $setVar));
}

function custom_theme() {
  $path = drupal_get_path('module', 'custom');
  return array(
    'custom' => array(
        'variables' => array('output' => null),
        'template' => 'custom',
     ),
  );
}

Now in custom.tpl.php just need <?php print $output; ?>