how long is a memory address typically in bits

The processor uses 32 bits to store an address. With 32 bits, you can store 2^32 distinct numbers, ranging from 0 to 2^32 - 1. "Byte addressing" means that each byte in memory is individually addressable, i.e. there is an address x which points to that specific byte. Since there are 2^32 different numbers you can put into a 32-bit address, we can address up to 2^32 bytes, or 4 GB.

It sounds like the key misconception is the meaning of "byte addressing." That only means that each individual byte has its own address. Addresses themselves are still composed of multiple bytes (4, in this case, since four 8-bit bytes are taken together and interpreted as a single 32-bit number).

I was under the impression that for a 32-bit processor, it can address upto 2^32 bits, which is 4.29 X 10^9 bits (NOT BYTES).

This is typically not the case -- bit-level addressing is quite rare. Byte addressing is far more common. You could design a CPU that worked this way, though. In that case as you said, you would be able to address up to 2^32 bits = 2^29 bytes (512 MiB).