Notation regarding differential form and vector field

First, let me make some recalls.

Let $X$ be a vector field on $M$ and $f\colon M\rightarrow\mathbb{R}$ be a smooth function, then by definition, one has: $$X(f)\colon M\rightarrow\mathbb{R},x\mapsto T_xf(X(x)).$$ On a side note, this construction is used along with Hadamard's lemma to prove that one has a bijective correspondence between vector fields on $M$ and derivations $C^\infty(M,\mathbb{R})$ i.e. endomorphism $\delta$ of the real vector space $C^\infty(M,\mathbb{R})$ such that for all $f,g\in C^{\infty}(M,\mathbb{R})$, one has: $$\delta(fg)=\delta(f)g+f\delta(g).$$ This bijective correspondence is of central interest when defining the Lie bracket of two vector fields.

Coming back to your question.

If $\omega$ is differential form of degree $1$ on $M$ and $Y$ is a vector field on M, then $\omega(Y)\colon M\rightarrow\mathbb{R}$ is defined as: $$\omega(Y)(x)=\omega_x(Y(x)).$$ Therefore, $X(\omega(Y))$ is the function obtained from $\omega(Y)$ applying the derivation associated to $X$.

I cannot help providing a proof for the identity you stated, as it is a frequently used one.

Proposition. Let $M$ be a smooth manifold, let $X,Y\in\Gamma(TM)$ and let $\omega\in\Lambda^1T^*M$, then one has: $$\mathrm{d}\omega(X,Y)=X(\omega(Y))-Y(\omega(X))-\omega([X,Y]).$$

Proof. First, let define $\alpha\colon\Gamma(TM)\times\Gamma(TM)\rightarrow C^\infty(M,\mathbb{R})$ by: $$\alpha(X,Y):=X(\omega(Y))-Y(\omega(X))-\omega([X,Y]).$$ Notice that $\alpha$ is a $C^\infty(M,R)$-bilinear map using the $C^\infty(M,\mathbb{R})$-linearity of $\omega$ and the fact that $X(\cdot)$, $Y(\cdot)$ are derivations, along with the following identity: $$[fX,gY]=fg[X,Y]+fX(g)Y-gY(f)X.$$ Besides, since $\mathrm{d}\omega$ is a differential form of degree $2$, it is also a $C^\infty(M,\mathbb{R})$-bilinear map.

To conclude, notice that the desired equality is local and it therefore suffices to work in a chart of $M$: $$(U,(x_1,\ldots,x_n)).$$ This chart induces $(TU,(\partial_{x_1},\ldots,\partial_{x_n}))$ a chart of $TM$ and $(T^*U,(\mathrm{d}x_1,\ldots,\mathrm{d}x_n))$ a chart of $T^*M$. Hence, we can write: $$\omega_{\vert U}=\sum_{k=1}^n\omega_k\mathrm{d}x_k.\tag{1}$$ In particular, one gets the following expression: $$\mathrm{d}\omega_{\vert U}=\sum_{k,\ell=1}^n\partial_{x_\ell}\omega_k\mathrm{d}x_{\ell}\wedge\mathrm{d}x_k.\tag{2}$$ Recall that any element of $\Gamma(TU)$ is a $C^\infty(M,\mathbb{R})$-linear combination of $\partial_{x_i}$ and the problem boils down to: $$\mathrm{d}\omega(\partial_{x_i},\partial_{x_j})=\alpha(\partial_{x_i},\partial_{x_j}).$$ Which is almost immediate, since: $$\mathrm{d}x_{\ell}\wedge\mathrm{d}x_k(\partial_i,\partial_j)=\begin{vmatrix}\mathrm{d}x_\ell(\partial_i)&{d}x_\ell(\partial_j)\\\mathrm{d}x_k(\partial_i)&\mathrm{d}x_k(\partial_j)\end{vmatrix}=\begin{vmatrix}\delta_{i,\ell}&\delta_{j,\ell}\\\delta_{i,k}&\delta_{j,k}\end{vmatrix}=\delta_{i,\ell}\delta_{j,k}-\delta_{i,k}\delta_{j,\ell},$$ so that pluging in $(2)$, one gets: $$\mathrm{d}\omega(\partial_i,\partial_j)=\partial_{x_i}\omega_j-\partial_{x_j}\omega_i.$$ Besides, from $(1)$, one has $\omega(\partial_i)=\omega_i$, $\omega(\partial_j)=\omega_j$ and using Schwarz's lemma, one gets: $$\alpha(\partial_i,\partial_j)=\partial_{x_i}\omega_j-\partial_{x_j}\omega_i.$$ Whence the result. $\Box$

Remark. Let $V$ be a real vector space of finite dimension and let $\xi_V$ be the trivial vector bundle over $M$ with fiber $V$, then one actually defines the exterior derivative of differential forms with values in $\xi_V$ i.e. sections of $\Lambda^rT^*M\otimes\xi_V$ by this precise formula.

The proposition then tells us that if $V=\mathbb{R}$, the two definitions coincide.

We only did the work for $r=1$, but the formula extends in a natural fashion to greater $r$.