Can we define a derivative on the $p$-adic numbers?

Plenty of this has been done, and part of it is still actively researched. A very good textbook resource on the state of the art of the theory in the 1980s are chapters 2 to 4 of W. H. Schikhof's Ultrametric Calculus.

In particular, this contains many examples where that naive notion behaves weirdly (like in the other answer), even for someone who is already used to ultrametric spaces a bit. It turns out that to imitate $C^1$-functions in the $p$-adic setting, it is more useful to look at those $f$ where the limit

$$f'(a) := \lim_{(x,y) \to (a,a)} \frac{f(x)-f(y)}{x-y}.$$

exists. This is called strict differentiability e.g. in the article Robert Israel refers to in the comments. This theory and appropriate generalisations, as well as relation to appropriate notions of (locally) analytic functions, possible integrals and antiderivatives, Mahler bases etc., is expanded quite a bit in this book.

This is well-defined but does not always behave as expected. For example, see page 5 of these nice notes for a $p$-adic function whose derivative is zero but which isn't even locally constant. https://www2.math.ethz.ch/education/bachelor/seminars/hs2011/p-adic/report8.pdf

There is a short part on derivatives in $\mathbb{Q}_p$ in Fernando Q.Gouvêa's "$p$-adic numbers".

There is also no reason to restrict to finite-dimensional $V,W$. You might be interested in the notion of Fréchet derivatives.