Wordpress - How to validate custom fields in custom post type?

You're on the right track. I test the fields in the save_post callback, and then use admin notices to display errors to the user when a field fails validation. They show up just in a highlighted box at the top of the page, just like any errors/messages that WordPress itself generates.

Here's a simple example of creating an admin notice:

function my_admin_notice()
{
    ?>

    <div class="updated">
       <p>Aenean eros ante, porta commodo lacinia.</p>
    </div>

    <?php
}
add_action( 'admin_notices', 'my_admin_notice' );

That's not very practical, though. In a situation like this, you really just want a function that you can pass a message to. Something like,

if( $pizza != 'warm' )
    $notices->enqueue( 'Pizza is not warm', 'error' );

So, you can write that enqueue() function yourself (along with a function to print the notices), or you can include a library like IDAdminNotices.

Here's an example from a plugin I wrote. This uses notice enqueue/print functions that are built into the class itself, rather than including an external library.

public function saveCustomFields( $postID )
{
    // ...

    if( filter_var( $_POST[ self::PREFIX . 'zIndex'], FILTER_VALIDATE_INT ) === FALSE )
    {
        update_post_meta( $post->ID, self::PREFIX . 'zIndex', 0 );
        $this->enqueueMessage( 'The stacking order has to be an integer.', 'error' );
    }   
    else
        update_post_meta( $post->ID, self::PREFIX . 'zIndex', $_POST[ self::PREFIX . 'zIndex'] );

    // ...
}
add_action( 'save_post',    array( $this, 'saveCustomFields' );

I wrote a small plugin that not only validates the input fields on custom post types but also removes the default admin notice, without the use of Javascript.

here is some of the code

/ Validation filters

$title = $album->post_title;
if ( ! $title ) {
    $errors['title'] = "The title is required";
}

// if we have errors lets setup some messages
if (! empty($errors)) {

    // we must remove this action or it will loop for ever
    remove_action('save_post', 'album_save_post');

    // save the errors as option
    update_option('album_errors', $errors);

    // Change post from published to draft
    $album->post_status = 'draft';

    // update the post
    wp_update_post( $album );

    // we must add back this action
    add_action('save_post', 'album_save_post');

    // admin_notice is create by a $_GET['message'] with a number that wordpress uses to
    // display the admin message so we will add a filter for replacing default admin message with a redirect
    add_filter( 'redirect_post_location', 'album_post_redirect_filter' );
}

You can see the full tutorial here