How to umount non-sudo sshfs created directory

sshfs uses FUSE (File system in USErspace) instead of the regular mount with elevated permissions.

That also means you can not use umount (the counterpart of mount) to unmount the file system though, but you have fusermount -u, the FUSE unmount command:

fusermount -u /temp/user/harddrive

For more info, see e.g. man sshfs and man fusermount.