Could a temp directory with a lot of files slow down the Windows PC?

Yes, a huge number of files in the temp directory can indeed slow down your Windows system. A couple of examples can be found here and here and here

Some of the reasons for this are that anytime an application creates or reads a temporary file from this folder, Windows has to process the folder's contents (i.e. to find the file or to see if a duplicate file is being created, etc). Another reason is that whenever Windows goes looking for an application to execute where the exact file path is not known, it first iterates through all the files in all the directories listed in the system's PATH environment variable. In many systems, the temp folder is included on this list of folders.

In general it is good practice to clear out the TEMP folder(s) on a regular basis. Some people say weekly, some monthly.