How to backspace or delete?

The temporary solution is set the tty erase command to whatever your backspace key sends in the terminal, when connected via ssh.

stty erase ^H

The ^H sequence above is not the literal text but a control character entered by pressing ctrl-v and then backspace.

You can add this command to your .bashrc file on the docker VM to have it set automatically each time you connect. When editing the file in a terminal you have to enter the string with the same escape sequence as above.

Doing this in .bashrc will set erase for all logins to the VM so you may negatively impact the way erase works for other terminal types that connect.

You would normally fix this on the client side rather than the server. For example, PuTTY has a specific setting for this. I'm not sure why your powershell/ssh combo doesn't have erase mapped correctly as Docker normally works out of the box. Check what your Docker shortcuts do when launching the docker/ssh terminal and do the same when you launch your terminal to manually connect.

The accepted answer works, but I thought I'd point out that if you launch bash rather than stay in sh then it all works so just type bash at the # prompt and you will get the functionality you require.