How to differentiate an operator in QM?

Operators and derivatives

Let $X$ and $P$ be individual operators satisfying $$ [X,P]=i. \tag{1} $$ The operator $X$ is just one operator, so there is nothing to differentiate here.

Now consider a one-parameter family of operators, with one operator $X(t)$ for each time $t$. This is often called an operator-valued function. That's just a fancy name for having a separate operator $X(t)$ for each value of $t$, just like an ordinary function $f$ consists of a separate number $f(t)$ for each value of $t$. Operators can be added and subtracted, so the difference $X(t+\epsilon)-X(t)$ is a perfectly well-defined operator. The difference is generally non-zero for $\epsilon\neq 0$, so we can consider the derivative $$ \frac{d}{dt} X(t) := \lim_{\epsilon\to 0}\frac{X(t+\epsilon)-X(t)}{\epsilon} \tag{2} $$ as usual. Just like for an ordinary function, the derivative may or may not be well-defined (the limit may or may not exist or be unique), but let's only consider cases where it is well-defined.

Sometimes we write $X$ as an abbreviation for $X(t)$, which is apparently what your lecturer did. It's a very common abbreviation. However, for the rest of this post, I'll always write the $t$-dependence explicitly, whenever there is any $t$-dependence at all.

The Heisenberg picture

For any observable $A$ defined at time $t=0$, the Heisenberg picture defines a whole family of observables $A(t)$, one for each time $t$, like this: $$ A(t) := U(-t)A U(t) \tag{4} $$ with $$ U(t)=e^{-iHt} \tag{5} $$ where $H$ is the Hamiltonian. Equation (5) implies $$ \frac{d}{dt} U(t) = -iH\,U(t). \tag{6} $$ This works for an operator $H$ just like it would if $H$ were a number, because everything in equations (5) and (6) commutes with $H$. I actually prefer to reverse the logic, using equation (6) together with the condition $$ U(0)=1 \tag{7} $$ to define the family of operators $U(t)$, and I think of (5) as a mere abbreviation for the conditions (6) and (7). Equations (4) and (6) imply the key equation $$ \frac{d}{dt} A(t) = i\big[H,A(t)\big]. \tag{8} $$ By the way, this is not specific to QFT. Equations (4)-(8) are how the Heisenberg picture works in any quantum theory, and none of this requires referring to the Schrödinger picture at all.

Application to the harmonic oscillator

Now suppose that $X$ and $P$ are individual operators satisfying equation (1), namely the position and momentum observables at time $t=0$. Use equation (4) to define a family of operators $X(t)$. Then use equation (8) to get \begin{align} \frac{d^2}{dt^2}X(t) &= \frac{d}{dt}i\big[H,X(t)\big] \\ &= i\left[H,\frac{d}{dt}X(t)\right] \\ &= -\Big[H,\big[H,X(t)\big]\Big]. \\ &= -\Big[H,\big[H,U(-t)XU(t)\big]\Big] \\ &= -U(-t)\Big[H,\big[H,X\big]\Big]U(t). \tag{9} \end{align} To get the last line, I used the fact that $H$ commutes with $U(\pm t)$, which is clear from equation (5). With $H$ defined as in the OP, straightforward application of (1) gives $$ \Big[H,\big[H,X\big]\Big] = w^2 X. \tag{10} $$ Substitute this back into the last line of (9) to get $$ \frac{d^2}{dt^2}X(t) = -w^2 U(-t)XU(t) = -w^2 X(t), \tag{11} $$ which is the desired result.

To define an operator it is sufficient to define its action on an arbitrary state $\Psi$. The definition of $d \hat{x} / d t$ is as follows:

$$ \left( \frac{d \hat{x}}{dt} \right) \left| \Psi \right> = \frac{d}{dt} \left( \hat{x} \left| \Psi \right> \right). $$

Equivalently, the matrix elements of the derivative of an operator are derivatives of the matrix elements of that operator.

In linear algebra an operator is something which acts on a vector and returns another vector. A linear operator can be represented by a matrix. So in this sense you should think of differentiating an operator like differentiating a matrix. Take for example this time dependent rotation matrix: $$\partial_tR=\partial_t\begin{pmatrix} \cos\omega t & \sin\omega t\\ -\sin\omega t & \cos\omega t \end{pmatrix}= \omega\begin{pmatrix} -\sin\omega t & \cos\omega t\\ -\cos\omega t & -\sin\omega t \end{pmatrix}.$$ In quantum mechanics these operators are more generalized to act on elements that live in Hilbert spaces, which means these operators can also act on functions in addition to vectors.

You should also note that how an operator acts looks different in a different basis. Let's consider the Schrodinger position operator first. Here $\hat x$ just multiplies by $x$ in the position basis, while it is a differential operator in the momentum basis: $$\hat x\psi(x)=x\psi(x)\quad\leftrightarrow\quad\hat x\phi(p)=i\hbar\frac{d}{dp}\phi(p)$$

In both cases this is time independent, so $\partial_t\hat x=0$. This is because the Schrodinger operators are defined to be that way. In the Heisenberg picture you can calculate the time derivative as follows: $$\frac{d}{dt}\hat x(t)=\frac{i}{\hbar}[H,\hat x]=\frac{p(t)}m$$ You can calculate the same for the momentum operator using the commutation relations between $\hat x$ and $\hat p$ to calculate the second derivative. I will leave this as an exercise. Usually Heisenberg operators have a subscript $x_H$ when first introduced but I assume your teacher uses them so much he leaves them off. For more info see https://en.wikipedia.org/wiki/Heisenberg_picture