Disable PowerShell beep on backspace

The beep is provided by the PSReadline module, which shipped with Windows 10. You need to change the PSReadline option to disable the bell:

Set-PSReadlineOption -BellStyle None

If you want this change for all future PowerShell sessions, then you need to add this command to your PowerShell profile. For example, to set the option for "Current User, Current Host" ($Profile):

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser
if (!(Test-Path -Path "$Profile")) {New-Item -ItemType File -Path "$Profile" -Force}
Add-Content -Value "Set-PSReadlineOption -BellStyle None" -Path "$Profile"

The first line allows your profile run a startup script when PowerShell opens (About Execution Policies). The second line tests to see if you already have a startup script defined for "Current User, Current Host". The third line adds the bell option to your startup script.

A more permanent solution that extends @PetSerAl's answer:

1. Run PowerShell as administrator.
2. Execute: set-executionpolicy remotesigned. This will allow PowerShell scripts to run on your computer. If you are not sure on what that is do not continue.
3. Go to C:\Windows\System32\WindowsPowerShell\v1.0 if you are on Windows 10.
4. Create a file named Microsoft.PowerShell_profile.ps1.
5. Edit that file and add Set-PSReadlineOption -BellStyle None.
6. Open a new PowerShell console and you should not hear the beep any more.

All (new) PowerShell consoles for all users on your computer are now silent.

This answer is for Windows 10.  For other versions of Windows, the folder might differ.