Protect sensitive data in embedded devices?

I have worked with the development of credit card machines, the most common attacks were:

1. Human resource leak, when an engineer leaked sensitive data or inserted backdoors to the product. This is not related to your question, I wont expand.
2. Software hacking, where the attacker will try to download your firmware and or try to grant "root" access exploiting JTAG, UART, ethernet, USB and any other port your product may have. It doesn't matter how secure your key is, if they get a JTAG to run on it the key shall be easily recovered. Same with a root console via serial, or if they are able to download, hack and reprogram your flash.
3. Bus sniffing, where the attacker will plug spies into your buses (mmc, spi, uart, etc... even on memory such as DDR3 bus) to spy on data. If you transfer your sensitive data unencrypted into a bus that can be spyed, it is a security breach. To avoid that you should bury your buses into the PCB and use BGAs only, do net let it accessible in outer layers.
4. Encrypt everything. There's been attacks where memory chips were hatched (physically!) and bit states were read visually by a microscope!

This is the order of the most common attacks I have known of.

You could always use secure chips, secure bootloaders and tamper traps to erase the whole thing in case of abuse.

This is completely depended on the functionality of your device (or your secure information). Generally AES-128 bit encryption is enough for reasonable grade of security in embedded devices. But storing keys in non-protected memory(processor's flash memory) cause a serious security matters,There are many Physical/Logical ways for extracting data from non-volatile memory.

You can Store your keys in general SRAM memory and protect system with tamper detection keys,or use MCU with built-in crypto co-processor and secure key storing(e.g. lpc18s,lpc43s and etc.)or use a external cryptographer chip like TPM or SAM for boosting your security level. But as said it before at first you must decide how & when you need to en/decrypt data , Is this needed to transmit plain data(decrypted information) in unsecured data lines? You can receive more information in below link which I asked a while ago: https://electronics.stackexchange.com/questions/198274/storing-a-secure-key-in-an-embedded-devices-memory