How to balance SQL Server memory with SQL Server Analysis Services memory

I think it is feasible to force both services' hands by having a scheduled task or service that:

(a) before SQL Server's nightly processing, shuts down the SSAS service, and increases the memory allotted to SQL Server

(b) after SQL Server has done its processing, reduces max server memory and restarts SQL Server

(c) starts SSAS

This assumes that SSAS isn't required to be online while SQL Server is processing. Otherwise change (a) from "shuts down" to "restarts"... as long as SSAS isn't actively doing anything while SQL Server is processing, there shouldn't be any concern that it will steal memory back.

This also assumes that you want each service to overlap memory consumption, and use the same memory at different times - I'm sure there are ways with affinity / NUMA etc. to completely segregate their operations space or, in the worst case, to host them on different machines.

Do SQL and SSAS ever need to run at the same time? If no, are you 100% sure?

I would look at setting the memory setting for SQL and SSAS to see if they can cooperate with one another.

We'll need to leave some memory for the OS, other processes, and multi-page allocations. Perhaps 4G?

Try something like this...

On the SQL side, set Max Server Memory to 12G and Min Server Memory to 6G.

On the SSAS side, set TotalMemoryLimit to 12G and the LowMemoryLimit to 6G.

Also, do not enable Lock Pages In Memory.

Try a few iterations, adjusting these numbers up or down.

If this just doesn't work, and you know for a fact SQL and SSAS do not need to run at the same time, then I would try Aaron's suggestion of scheduling jobs to turn services off and on.