Delete matching files in all subdirectories

Remove all *.swp files underneath the current directory, use the find command in one of the following forms:

• find . -name \*.swp -type f -delete
  
  The -delete option means find will directly delete the matching files. This is the best match to OP's actual question.
  
  Using -type f means find will only process files.
  
  

• find . -name \*.swp -type f -exec rm -f {} \;
find . -name \*.swp -type f -exec rm -f {} +
  
  Option -exec allows find to execute an arbitrary command per file. The first variant will run the command once per file, and the second will run as few commands as possible by replacing {} with as many parameters as possible.
  
  

• find . -name \*.swp -type f -print0 | xargs -0 rm -f
  
  Piping the output to xargs is used form more complex per-file commands than is possible with -exec. The option -print0 tells find to separate matches with ASCII NULL instead of a newline, and -0 tells xargs to expect NULL-separated input. This makes the pipe construct safe for filenames containing whitespace.
  
  

See man find for more details and examples.

find . -name '*.swp' -delete

Having find do the delete itself remove any risk for space embedded in filename, ... For extra security also consider adding -type f for files only.

find /path -type f -name "*.swp" -delete
find /path -type f -name "*.swp" -exec rm -f "{}" +;

bash 4.0

shopt -s globstar
rm -f /path/**/*.swp