Why is SPI Flash memory so limited in max size, and cost way more (per MB) than SD flash memory?

Welcome to the world of consumer electronics and manufacturing in volume! Nobody ever said it made sense!

The difference in price has nothing to do with anything technical. It is purely the economics of the market. The SPI Flash is being sold in relatively low quantities and somewhat high profit margins. The SD card is being sold in huge quantities and a very low profit margin.

While on the surface it might seem that the SD card would be more expensive since it has a smaller capacity and less "middlemen", that obviously isn't the case.

Another complication is that you could buy one make/model of SD card today, and then buy the same make/model in 3 month, and you would not be guaranteed to get the exact same thing. In those 3 months the internal design of the SD card could change. For most consumers this would not matter, but for some embedded users this could kill your application. Also, the SD card maker is not going to tell you of these changes. The same is not true of the SPI Flash, where you will most likely get the same thing for years.

You can get SD cards from manufacturers that will guarantee that they sell the same part for years, but it will be much more expensive.

These things are true of many products, not just SPI Flash and SD Cards. Memory (Flash and RAM) is the most obvious one. Another one is the iPad. In many cases it would be cheaper to buy iPads in bulk than to try and manufacture your own-- even in 100,000 unit quantities. You can't underestimate the purchasing power of a large company building millions of units at a time.

There are other factors that I didn't cover. Differences in part types, packages, purchasing channels, etc. But the problem you raise is more complicated than any one single factor can account for. My market/economic explanation is the biggest factor, but not the only one.

The first part is NOR flash. The second is NAND flash, it is cheaper to produce. Learn more at Wikipedia here.

It isn't the interface that costs more, it is the type of flash that is the main driver.

Plus there isn't much competition in large SPI flash. I suspect that's the main driver at this density.

Additionally your are comparing a proper semiconductor distributor to random web sellers. The web sellers may have varying quality, reliability, etc.

To @Olins question, NOR flash reads quickly (~100 ns) a word or byte at a time. NAND flash reads slowly (~us) an a block at a time.