Is it possible to use pip to install a package over ssh in a self-hosted gitlab?

I don't know why, but by running the following command it worked (slash instead of : after <my_domain>):

pip install git+ssh://git@<my_domain>/se7entyse7en/<project_name>.git
#                                    ^
#                             slash instead of :

Yes. This is the default use:

pip install git+ssh://git@<my_domain>:22/<project_group>/<project_name>.git

The use of the colon by itself implies the default ssh port number 22. Because you can control the port number of your server, the port number could be different. Git enables customisation by not providing :22/ or / only.