What does "cost per bit" mean?

The cost per bit is the monetary cost (i.e. in dollars and cents) per unit of memory, thats all, ie a way of saying "assuming the amount of memory required is the same". (You could equally say the cost per gig of memory - but mentioning the size implies a usage case. A bit is very arguably the smallest amount that can be stored)

The cost per bit is just the price you have to pay for specific amount of memory.

If some amount of brand X memory costs $20 and the same amount of brand Y memory costs $25, then brand X has better cost per bit. The cost per bit differences are much more pronounced when comparing different kinds of memory (SRAM, DRAM, flash, SSD...)

In addition to these other answers that rightly mention that cost per bit is the amount of money per bit:

If you have memory that stores 1024 bytes, that's 8192 bits. If that memory cost you $10 then it would be $10(cost)/8192 bits, or $0.001220703125 per bit. If you had memory with a faster access time, maybe it would cost $20 instead, in which case you'd have $0.00244140625 per bit.

Faster access time: $0.00244140625  cost per bit 
Greater capacity:   $0.001220703125 cost per bit