How effective is LSI CacheCade SSD storage tiering?

Since CacheCade SSDs can be used for write and read cache, what purpose does the RAID controller's onboard NVRAM play?

If you leave the write caching feature of the controller enabled, the NVRAM will still be used primarily. The SSD write cache will typically only be used for larger quantities of write data, where the NVRAM alone is not enough to keep up.

When used as a write cache, what danger is there to the CacheCade SSDs in terms of write endurance? Using consumer SSDs seems to be encouraged.

This depends on how often your writes are actually causing the SSD write cache to become necessary... whether or not your drives are able to handle the write load quickly enough that the NVRAM doesn't fill up. In most scenarios I've seen, the write cache gets little to no action most of the time, so I wouldn't expect this to have a big impact on write endurance - most writes to the SSDs are likely to be part of your read caching.

Do writes go straight to SSD or do they hit the controller's cache first?

Answered above... Controller cache is hit first, SSD cache is more of a 2nd line of defense.

How intelligent is the read caching algorithm? I understand how the ZFS ARC and L2ARC functions. Is there any insight into the CacheCade tiering process?

Sorry... no knowledge to contribute on that - hopefully someone else will have some insight?

What metrics exist to monitor the effectiveness of the CacheCade setup? Is there a method to observe a cache hit ratio or percentage? How can you tell if it's working?

It doesn't look like any monitoring tools are available for this as there are with other SAN implementations of this feature set... And since the CacheCade virtual disk doesn't get presented to the OS, you may not have any way to manually monitor activity either. This may just require further testing to verify effectiveness...

Opinion/observation: In a lot of cases (when used correctly, read cache appropriately sized for the working data set) this feature makes things FLY. But in the end, it can be hit-and-miss.