Software for symbolic matrix calculus?

Indeed, I was having the same problem. Hence, I implemented a matrix calculus toolbox myself. You can find it at www.matrixcalculus.org. It can compute vector and matrix derivatives and will return the result in terms of the original vectors and matrices involved.

I have not used it, but the NCAlgebra (NC for noncommutative) extension for Mathematica does do directional derivatives at least. It is aimed mostly at algebra involving noncommutative variables and their adjoints, however, so that may or may not be sufficient for your needs.