In GIMP, how do I make part of image transparent manually?

In GIMP layers can either have an alpha channel or have no alpha channel. When you try erasing parts of an image when there's no alpha channel you get backround color.

To add an alpha channel you can bring up the Layers dialog by pressing Ctrl+L, right-click on the layer you want to edit and select Add Alpha Channel there.

From there everything is very much like in Photoshop - i.e. Eraser tool erases to transparency, you can make a selection (using, for example, Magic Wand tool) and erase everything in it by pressing Del, etc.

If you use an eraser, that will erase to transparence. You also can select an area and press the delete key (not backspace)