Drupal - How do I add a custom validation handler to an existing form/field?

Berdir gave the correct answer, that a constraint is the correct way to go about adding validation to a field in Drupal 8. Here is an example.

In the example below, I will be working with a node of type podcast, that has the single value field field_podcast_duration. The value for this field needs to be formatted as HH:MM:SS (hours, minutes and seconds).

To create a constraint, two classes need to be added. The first is the constraint definition, and the second is the constraint validator. Both of these are plugins, in the namespace of Drupal\[MODULENAME]\Plugin\Validation\Constraint.

First, the constraint definition. Note that the plugin ID is given as 'PodcastDuration', in the annotation (comment) of the class. This will be used further down.

namespace Drupal\[MODULENAME]\Plugin\Validation\Constraint;

use Symfony\Component\Validator\Constraint;

/**
 * Checks that the submitted duration is of the format HH:MM:SS
 *
 * @Constraint(
 *   id = "PodcastDuration",
 *   label = @Translation("Podcast Duration", context = "Validation"),
 * )
 */
class PodcastDurationConstraint extends Constraint {

  // The message that will be shown if the format is incorrect.
  public $incorrectDurationFormat = 'The duration must be in the format HH:MM:SS or HHH:MM:SS. You provided %duration';
}

Next, we need to provide the constraint validator. This name of this class will be the class name from above, with Validator appended to it:

namespace Drupal\[MODULENAME]\Plugin\Validation\Constraint;

use Symfony\Component\Validator\Constraint;
use Symfony\Component\Validator\ConstraintValidator;

/**
 * Validates the PodcastDuration constraint.
 */
class PodcastDurationConstraintValidator extends ConstraintValidator {

  /**
   * {@inheritdoc}
   */
  public function validate($items, Constraint $constraint) {
    // This is a single-item field so we only need to
    // validate the first item
    $item = $items->first();

    // Check that the value is in the format HH:MM:SS
    if ($item && !preg_match('/^[0-9]{1,2}:[0-5]{1}[0-9]{1}:[0-5]{1}[0-9]{1}$/', $item->value)) {
      // The value is an incorrect format, so we set a 'violation'
      // aka error. The key we use for the constraint is the key
      // we set in the constraint, in this case $incorrectDurationFormat.
      $this->context->addViolation($constraint->incorrectDurationFormat, ['%duration' => $item->value]);
    }
  }
}

Finally, we need to tell Drupal to use our constraint on field_podcast_duration on the podcast node type. We do this in hook_entity_bundle_field_info_alter():

use Drupal\Core\Entity\EntityTypeInterface;

function HOOK_entity_bundle_field_info_alter(&$fields, EntityTypeInterface $entity_type, $bundle) {
  if (!empty($fields['field_podcast_duration'])) {
    $fields['field_podcast_duration']->addConstraint('PodcastDuration');
  }
}

The #validate property is still used in Drupal 8. (With Adi's solution you will override the existing validator)

If you want to add your custom validator in addition to the default, you will have to add something like this in hook_form_FORM_ID_alter (or similar):

$form['#validate'][] = 'my_test_validate';

The correct way to do this for a content entity like node is to register it as a constraint.

See forum_entity_bundle_field_info_alter() and the corresponding ? ForumLeaf validation constraint (note that there are two classes needed).

That is a bit more complicated at first, but the advantage is that it is integrated into the validation API, so your validation isn't limited to the form system but can, for example, also work with nodes submitted through the REST API.