Add Custom attributes to Custom attribute set programmatically

Yes, this is possible.

First, set these keys in your $data array to following values to avoid adding attribute to all attribute sets:

'user_defined'         => true,
'group'                => ''

Then add attribute to your attribute set:

$attributeSetId = $this->getAttributeSetId($entityTypeId, 'New Attr Set');
$this->addAttributeToSet($entityTypeId, $attributeSetId, 'General', 'new_attr', 10);