Drupal - Custom validation for a form?

You can add any number of validation functions to any form in hook_form_FORM_ID_alter() like so:

function mymodule_form_article_node_form_alter(&$form, &$form_state, $form_id) {
  // There will already be some validate handlers added so you need to add to the
  // array rather than overwrite it.
  $form['#validate'][] = 'mymodule_article_form_validate';

  // As mentioned above you can add as many as you want
  $form['#validate'][] = 'mymodule_article_form_validate_2';
}

function mymodule_article_form_validate($form, &$form_state) {
  // Random example, if the title is 'test' throw an error
  if ($form_state['values']['title'] == 'test') {
    form_set_error('title', 'Title cannot be "test"');
  }
}

You should use hook_form_alter and add your function to the #validate property.