sshfs uses FUSE (File system in USErspace) instead of the regular mount with elevated permissions.
That also means at least on old releases up to 16.04 you can not use umount (the counterpart of mount) to unmount the file system, although this should work as well on 18.04 and newer releases.
Generally you can unmount the file system using the corresponding command that is part of the used FUSE package. Up to Ubuntu 20.04, this is fuse version 2 which contains the fusermount command, whereas since around Ubuntu 22.04, it has been replaced by fuse3 containing fusermount3 instead.
Therefore, on a modern Ubuntu release (22.04 or newer), the preferred method is to use:
fusermount3 -u <MOUNTPOINT>
Or in your specific example:
fusermount3 -u /temp/user/harddrive
For more info, see e.g. man sshfs and man fusermount or man fusermount3 for your respective Ubuntu release.