Is tuning the innodb_buffer_pool_size important on Solaris ZFS?
No, he's not right. If a block is not in the InnoDB cache, then it has to be fetched, which means it will come either from disk or from the ZFS cache, at which point two copies of it exist in main memory. If you use that block, it will come out of the InnoDB cache. If you write that block, it will go from the InnoDB cache to the disk. The ZFS cache is just a helpless spectator in this scenario.
However, if your issue is INSERT performance, it's unlikely to be related to this unless your system is under a very heavy load... is it? From the docs:
If an index record should be inserted into a nonunique secondary index, InnoDB checks whether the secondary index page is in the buffer pool. If that is the case, InnoDB does the insertion directly to the index page. If the index page is not found in the buffer pool, InnoDB inserts the record to a special insert buffer structure. The insert buffer is kept so small that it fits entirely in the buffer pool, and insertions can be done very fast.
Periodically, the insert buffer is merged into the secondary index trees in the database. Often it is possible to merge several insertions into the same page of the index tree, saving disk I/O operations. It has been measured that the insert buffer can speed up insertions into a table up to 15 times.
The insert buffer merging may continue to happen after the inserting transaction has been committed. In fact, it may continue to happen after a server shutdown and restart (see Section 13.2.6.2, “Forcing InnoDB Recovery”).
Insert buffer merging may take many hours when many secondary indexes must be updated and many rows have been inserted
SHOW ENGINE INNODB STATUS should show you exactly what the DB is waiting on. Also see if strace can help.
He's not correct.
In a nutshell, what he is arguing is that you do not need such a large InnoDB buffer pool because ZFS has some fancy caches like a L2 ARC.
While cache misses might be less expensive, each page will still needed to be loaded into InnoDB for modifications to be made, and a smaller buffer pool will mean more churning of what pages are loaded in memory.
The internal lock (mutex) held during page replacement (buf0buf) in versions of MySQL prior to 5.5 is a real hot spot, and I'm sure it would be very trivial to write a benchmark to prove him wrong.
(Other steps have to happen during page replacement too, such as verifying/updating page checksums).