Not able to convert directly to decimal value?

Convert to a bigint first and then convert to a decimal:

SELECT CONVERT(DECIMAL(38, 0), CONVERT(BIGINT, CONVERT(binary(2), 'a321', 2)))

The binary representation of decimals is quite different from integers, and not all binary representations can be converted to a decimal.