Gradient of $A \mapsto \sigma_i (A)$

Let $\{e_i\}$ denote the standard basis vectors. Then $q_i=Qe_i$ is the $i^{th}$ column of $Q$.
The definition of semi-orthogonality says that the columns of $Q$ are orthonormal, i.e. $$\eqalign{ I &= Q^TQ \\ e_i^T(I)e_j &= e_i^T(Q^TQ)e_j \\ \delta_{ij} &= q_i^Tq_j \\ }$$ Multiply the SVD by the $i^{th}$ columns of $(U,V)$ to isolate the $i^{th}$ singular value. $$\eqalign{ A &= \sum_{j=1}^k \sigma_j u_j v_j^T \\ u_i^TAv_i &= \sum_{j=1}^k \sigma_j (u_i^Tu_j)(v_j^Tv_i) = \sum_{j=1}^k \sigma_j\,\delta_{ij}^2 \;=\; \sigma_i \\ }$$ Rearrange this result with the help of the trace/Frobenius product $\Big(A\!:\!B={\rm Tr}\!\left(A^TB\right)\Big)$
Then calculate the differential and gradient. $$\eqalign{ \sigma_i &= u_iv_i^T:A \\ d\sigma_i &= u_iv_i^T:dA \\ \frac{\partial\sigma_i}{\partial A} &= u_iv_i^T \\ }$$ Similarly, the singular vectors also vary with $A$. $$\eqalign{ \sigma_i u_i &= Av_i \\ \sigma_i u_i &= \left(v_i^T\otimes I_m\right){\rm vec}(A) \\ \sigma_i\,du_i &= \left(v_i^T\otimes I_m\right){\rm vec}(dA) \\ \frac{\partial u_i}{\partial{\rm vec}(A)} &= \frac{v_i^T\otimes I_m}{\sigma_i} \\ \\ \\ \sigma_i v_i^T &= u_i^TA \\ \sigma_i v_i &= \left(I_n\otimes u_i^T\right){\rm vec}(A) \\ \sigma_i\,dv_i &= \left(I_n\otimes u_i^T\right){\rm vec}(dA) \\ \frac{\partial v_i}{\partial{\rm vec}(A)} &= \frac{I_n\otimes u_i^T}{\sigma_i} \\ \\ }$$