Since upgrading to Solaris 11, my ARC size has consistently targeted 119MB, despite having 30GB RAM. What? Why?

Unfortunately I cannot solve your problem, but here's some background information:

• The ARC target size does not seem to be a fix value. I experience the same problem on a Solaris 11 machine and after each reboot, at some point the target size seems to lock in at a value between ~100 and ~500MB.

• At least 3 other people are facing the same issue, as discussed in http://mail.opensolaris.org/pipermail/zfs-discuss/2012-January/050655.html

• There is also an open bug report (7111576) on "My Oracle Support" (https://support.oracle.com). If your server is under a valid support contract, you should file a service request and refer to that bug. As of now, any bugfix still seems to be work in progress...

Other than that, there's not much you can do. If you've yet to upgrade your zpool/zfs versions, you might try booting into your old Solaris 11 Express boot environment and run that until Oracle finally decides to release a SRU that fixes the issue.

Edit: Since the question of performance degradation has been discussed above: It all depends on what you're doing. I've seen horrible latencies on my Solaris 11 NFS share ever since upgrading to Solaris 11 11/11. Compared to your system, however, I have relatively few spindles and rely heavily on ARC and L2ARC caching working as expected (please be aware that the problem also causes L2ARC not to grow to any reasonable size). This is certainly not an issue of misinterpreted statistics.

Even though you might not rely too heavily on ARC/L2ARC, you will probably be able to reproduce it by reading a large file (that would normally fit into your RAM) multiple times with dd. You will probably notice that the first time reading the file will actually be faster than any consecutive reads of the same file (due to the ridiculous ARC size and countless cache evictions).

Edit: I have now managed to receive an IDR patch from Oracle that resolves this issue. If your system is under support, you should ask for the IDR patch for CR 7111576. The patch applies to Solaris 11 11/11 with SRU3.