git tries to delete a directory on checkout

I just had to quit out of VS Code to get the error to go away

This error message comes from this patch (in compat/mingw.c), and this one (also in compat/mingw.c).
It usually means your directory is "busy", or isn't empty (but contains only private files, ie non-versioned files).

So make sure that, when checking out a branch, you don't have a running process making your specific directory "busy".

Ted M Young helpfully comments:

Another definition of "busy" is that it's your current directory!
It seems safest to do a cd to the root directory and then change directories to avoid this error.