Wordpress - get term by id without taxonomy

I accepted One Trick Pony's answer because it got me on the right track and it does answer my question. However, in the end I am using this, it returns full term object with it's taxonomy fields. Though it's a bit hacky...

/**
 * Get ther without khowing it's taxonomy. Not very nice, though.
 * 
 * @uses type $wpdb
 * @uses get_term()
 * @param int|object $term
 * @param string $output
 * @param string $filter
 */
function &get_term_by_id($term, $output = OBJECT, $filter = 'raw') {
    global $wpdb;
    $null = null;

    if ( empty($term) ) {
        $error = new WP_Error('invalid_term', __('Empty Term'));
        return $error;
    }

    $_tax = $wpdb->get_row( $wpdb->prepare( "SELECT t.* FROM $wpdb->term_taxonomy AS t WHERE t.term_id = %s LIMIT 1", $term) );
    $taxonomy = $_tax->taxonomy;

    return get_term($term, $taxonomy, $output, $filter);

}

Yes, but you need you make your own SQL query.

A modified version of WP's get_term():

function &get_term_by_id_only($term, $output = OBJECT, $filter = 'raw') {
    global $wpdb;
    $null = null;

    if ( empty($term) ) {
        $error = new WP_Error('invalid_term', __('Empty Term'));
        return $error;
    }

    if ( is_object($term) && empty($term->filter) ) {
        wp_cache_add($term->term_id, $term, 'my_custom_queries');
        $_term = $term;
    } else {
        if ( is_object($term) )
            $term = $term->term_id;
        $term = (int) $term;
        if ( ! $_term = wp_cache_get($term, 'my_custom_queries') ) {
            $_term = $wpdb->get_row( $wpdb->prepare( "SELECT t.* FROM $wpdb->terms AS t WHERE t.term_id = %s LIMIT 1", $term) );
            if ( ! $_term )
                return $null;
            wp_cache_add($term, $_term, 'my_custom_queries');
        }
    }

    if ( $output == OBJECT ) {
        return $_term;
    } elseif ( $output == ARRAY_A ) {
        $__term = get_object_vars($_term);
        return $__term;
    } elseif ( $output == ARRAY_N ) {
        $__term = array_values(get_object_vars($_term));
        return $__term;
    } else {
        return $_term;
    }
}

Probably not a good idea. Try to handle the tax too in your meta fields...