Drupal - Perform a query with an entity field condition with multiple values

Use two separate andConditionGroup():

$query = \Drupal::entityQuery('node')
  ->condition('status', NODE_PUBLISHED)
  ->condition('type', 'custom_type');
$and = $query->andConditionGroup();
$and->condition('custom_taxonomy', 2);
$query->condition($and);
$and = $query->andConditionGroup();
$and->condition('custom_taxonomy', 8);
$query->condition($and);
$result = $query->execute();

This works no matter how many terms are in the field or in which delta they are.

Edit

This results in this SQL:

SELECT base_table.vid AS vid, base_table.nid AS nid
FROM 
{node} base_table
INNER JOIN {node_field_data} node_field_data ON node_field_data.nid = base_table.nid
INNER JOIN {node__custom_taxonomy} node__custom_taxonomy ON node__custom_taxonomy.entity_id = base_table.nid
INNER JOIN {node__custom_taxonomy} node__custom_taxonomy_2 ON node__custom_taxonomy_2.entity_id = base_table.nid
WHERE  (node_field_data.status = '1') AND (node_field_data.type = 'custom_type') AND( (node__custom_taxonomy.custom_taxonomy_target_id = '2') )AND( (node__custom_taxonomy_2.custom_taxonomy_target_id = '8') )

To do complex queries as you asked, you will need to use a condition group and to query the delta.

$query = \Drupal::entityQuery('node');
$query->condition('status', NODE_PUBLISHED)
  ->condition('type', 'custom_type')
  ->condition('custom_taxonomy', [2, 8], 'IN')
  ->condition('custom_taxonomy.%delta', 2, '=')
  ->sort('field_last_name', DESC);
$or = $query->orConditionGroup();
$or->condition('custom_taxonomy.0.target_id', 2);
$or->condition('custom_taxonomy.0.target_id', 8);
$query->condition($or);

See QueryInterface::condition documentation.