Is booth multiplication algorithm for multiplying 2 positive numbers?

You're not giving enough room for your sign handling. 5 is not 101, but 0101: it has to start with a 0, because values starting with 1 are negative. 101 is actually -3: it's the two's complement of 011, which is 3. Similarly, 4 is not 100, but 0100; 100 is -4. So when you multiply 101 by 100, you're actually multiplying -3 by -4; that's why you get 12.