Magento 2 : How To Create UNIQUE KEY Field in Custom Tabel

You can use Index Type as \Magento\Framework\DB\Adapter\AdapterInterface::INDEX_TYPE_UNIQUE
To add unique key use code,

->addIndex(
    $installer->getIdxName(
        'YOUR_TABLE_NAME',
        ['YOUR_UNIQUE_FIELD_NAME'],
        \Magento\Framework\DB\Adapter\AdapterInterface::INDEX_TYPE_UNIQUE
    ),
    ['YOUR_UNIQUE_FIELD_NAME'],
    ['type' => \Magento\Framework\DB\Adapter\AdapterInterface::INDEX_TYPE_UNIQUE]
)

You can add multiple column combination as UNIQUE with,

->addIndex(
    $installer->getIdxName(
        'catalog_product_entity_datetime',
        ['entity_id', 'attribute_id', 'store_id'],
        \Magento\Framework\DB\Adapter\AdapterInterface::INDEX_TYPE_UNIQUE
    ),
    ['entity_id', 'attribute_id', 'store_id'],
    ['type' => \Magento\Framework\DB\Adapter\AdapterInterface::INDEX_TYPE_UNIQUE]
)

Refer vendor\magento\module-catalog\Setup\InstallSchema.php