How can I index a 3-D matrix with a 2-D mask in MATLAB?

You can do this by replicating your logical mask M across the third dimension using REPMAT so that it is the same size as D. Then, index away:

D_masked = D;
D_masked(repmat(~M,[1 1 size(D,3)])) = NaN;

If replicating the mask matrix is undesirable, there is another alternative. You can first find a set of linear indices for where M equals 0, then replicate that set size(D,3) times, then shift each set of indices by a multiple of numel(M) so it indexes a different part of D in the third dimension. I'll illustrate this here using BSXFUN:

D_masked = D;
index = bsxfun(@plus,find(~M),(0:(size(D,3)-1)).*numel(M));
D_masked(index) = NaN;